What Did This Castle Look like before? Exploring Referential Relations in Naturally Occurring Multimodal Texts (2021)

Abstract: Multi-modal texts are abundant and diverse in structure, yet Language & Vision research of these naturally occurring texts has mostly focused on genres that are comparatively light on text, like tweets. In this paper, we discuss the challenges and potential benefits of a L&V framework that explicitly models referential relations, taking Wikipedia articles about buildings as an example. We briefly survey existing related tasks in L&V and propose multi-modal information extraction as a general direction for future research.

## introduction many types of naturally occuring texts are inherently multi-modal: articles, social media posts, recipes, encyclopedias, manuals, advertisement, comics, etc. research on semiotics has long noted that the relationship between the linguistic and visual elements of such texts is extremely complex @xcite and varies widely across genres @xcite . to date, research in vision & language, however, has mostly focussed on crowdsourced data that simply aligns relatively short snippets of text to images (e.g. @xcite ), sequences of images (e.g. @xcite ) or video (e.g. @xcite ). here, the text-image relationship is simplified to a substantial, if not artificial, degree. in this paper, we take a qualitative look at some examples of real-world multi-modal texts, i.e. wikipedia articles on entities of the type "building". we find that many phenomena occuring jointly in these texts are currenlty tackled as separated tasks in v&l or text processing. we argue that a promising direction for future research in v&l is to aim for a joint framework that combines these different phenomena and levels of analysis. we believe that such a framework would be useful in a range of typical nlp applications (such as information extraction) where, currently, state-of-the-art models usually only process the text of a multimodal document. arnold and tilton (2020) discuss the motivation for such projects in the context digital humanities. the example documents discussed in this paper differ from typical objects of v&l research in many respects, but most importantly in terms of their (i) structure and (ii) semantics or topic. thus, our building articles are relatively long (i.e. much longer than image paragraphs in @xcite ), contain multiple images and text segments that do not directly relate to any of the images. concerning their semantics, the documents discuss buildings which constitute a type of named entity. this entity can be depicted visually in very diverse ways (see section 2) and that can be associated with a rich body of knowledge (e.g. historical events) described in the text. we will show how these two aspects call for a v&l framework that accounts for diverse referential relations whereas in most v&l tasks assume a single, fixed text-image relation. ## qualitative case study this section discusses observations we made by manually exploring a range of building articles from wikipedia. we leave the empirical consolidation of our findings for future work. ## related work in vision & language fixed text-image relation probably the most widely used image-text data in vision & language are so-called image captions that provide a general, neutral description of an image's content, cf. mscoco @xcite or @xcite captions . typical captions consist of a single sentence that directly refers to the image. other work has looked at more fine-grained referential relations such as referring expressions in the form of noun phrases that identify specific objects in an image @xcite . work on even more fine-grained resolution that captures object parts is relatively rare, but see @xcite . a complementary trend is to use texts that are (slightly) longer than image captions, such as image paragraphs that describe the image content in a sequence of sentences @xcite or dialogues that center on identifying an object in a sequence of turns @xcite or an image from a set of images @xcite . all of these datasets are crowdsourced and target a referential task on a specific, fixed level, i.e. image-sentence, object-phrase, object-dialogue, image-dialogue. it is worth noting that, internally, many recent largescale models in v&l process object-phrase relations while encoding image-sentence pairs @xcite @xcite @xcite , combining referential relations on two different levels. none of these tasks and models, however, deal with image-text pairs where significant parts of the text have no relation to the visual content, thereby circumventing the need to identify fragments that do indeed stand in a referential relation to a given image. diverse referential relations there is some initial work on datasets and tasks that capture more varied semantic or discursive relations between image and text: @xcite tag the image intent in multi-modal twitter posts, distinguishing between intents like 'provocative', 'expressive' or 'promotive'. their annotations assign a global label to the image which captures the relation to the text as a whole. this goes beyond literal image descriptions, but still does not capture structurally diverse referential relations. @xcite investigate text-image coherence in recipe texts that describe sequences of consecutive actions in a cooking context. structurally, the recipe's text is already segmented, with an image aligned to each step. @xcite distinguish image-text relations with respect to which part of the action is shown and whether all entities affected by an action are visible/mentioned in the text. both papers work on naturally occuring text, though these are still relatively short (tweets and 1-2 sentences per step respectively). neither task faces the segmentation problem to a degree that is similar to the complex structure we encountered in multi-modal wikipedia articles. by contrast, in our building example, the rhetorical purpose and authorial intent of each picture seems to be more or less uniform. that is, images are included to illustrate (as opposed to being provocative or expressive). likewise, the semiotics of these images are overwhelmingly parallel to the content of the text. @xcite work with a more coarsegrained and somewhat simplified version of the problem discussed in this paper. their task is to correctly predict the physical alignment of images and sections in wikipedia articles. this approach utilizes the inherent document structure foot_1 , however our observations (see section 2) call into question the presupposition that alignment in layout entails alignment in content. a similar text-image matching task is discussed in @xcite , where the authors seek to match the images in a document to the most relevant sentences in it (leaving out the captions). their model is trained on collections of sentences and images from the same documents; or different documents, for instances of non-relatedness. this information is used at test time to estimate the individual links between the sentences and images of a given document. @xcite is highly relevant to the concerns discussed in this paper because it has some success in grappling with the comparatively large amounts of text in the wikipedia article genre. ## towards multi-modal information extraction many of the phenomena we have discussed in section 2 have long been researched in nlp models that extract information from text only. prominently, text-oriented nlp has long been interested in detecting and processing entities, i.e. named entity recognition (ner) is a very well-known nlp task that is useful in a range of applications @xcite . the most standard named entity categories are person, location and organisation, however there is a number of ner tools with varying tag sets. one way to model texts of the type illustrated in fig. 1 would be to move towards multi-modal ner models that identify mentions of entities in a text and link them to corresponding images or image regions, cf. asgari-chenaghlu et al. (2020) for a similar proposal. event detection is another text-based nlp task that has been approached with the use of cnns (nguyen and grishman, 2015) and, more recently, attention mechanisms @xcite . multimodal event detection could be useful to capture referential relations as shown in fig. 2c . finally, models that represent or encode relations between entities @xcite in a multi-modal text would be an extremely useful tool in our setting. as a step towards processing comparatively large chunks of text, discourse segmentation @xcite al. (2017), iruskieta et al. (2019) ) splits documents into elementary discourse units. parsing these texts as a discourse is also a topic of ongoing research @xcite . to the best of our knowledge, research in vision & language has hardly been inspired by these classical, entity-centric task in text processing. this general impression is corroborated by the very comprehensive v&l survey of @xcite . ## conclusion in this paper, we have discussed some complexities of referential relations that arise in natually occuring multi-modal texts. solving at least some of these requires the use of far more involved text processing techniques than is common for widespread v&l tasks such as image captioning or visual dialogue. our domain -architectural sites -narrows this potentially sprawling problem somewhat. while every building is an entity unto itself, there are common features that are shared by large subsets. we argue that wikipedia articles are a valuable source of raw data for multi-modal document analysis, since they constitute a genre of document that is freely available in large quantities and across languages. 4 however, it is questionable whether models that identify the type of image-text relations discussed in this paper can be developed without hand-annotated data. in terms of uitlity and intended audience, it may be worth considering work like arnold and tilton (2020), whose aim is to add robust, searchable annotations to existing collections of historical images. this leads the authors to develop a model that automatically labels images using image segmentation and a pre-defined ontology. we believe that moving towards such more realistic texts in v&l is interesting both from a linguistic, and from an application-oriented perspective, i.e. for multi-modal information extraction.