W iki S cenes with Descriptions: Aligning Paragraphs and Sentences with Images in W ikipedia Articles (2024)

Abstract: Research in Language & Vision rarely uses naturally occurring multimodal documents as Wikipedia articles, since they feature complex image-text relations and implicit image-text alignments. In this paper, we provide one of the first datasets that provides ground-truth annotations of image-text alignments in multi-paragraph multi-image articles. The dataset can be used to study phenomena of visual language grounding in longer documents and assess retrieval capabilities of language models trained on, e.g., captioning data. Our analyses show that there are systematic linguistic differences between the image captions and descriptive sentences from the article’s text and that intra-document retrieval is a challenging task for state-of-the-art models in L&V (CLIP, VILT, MCSE).

## introduction research in language & vision (l&v) aims at building models that ground language in the visual modality and therefore requires datasets that align text and images. to date, most work in l&v uses datasets that have been obtained via annotation of images in a way that image and text are aligned by construction as in, e.g., image captioning or vqa datasets @xcite @xcite . multimodal imagetext data that occurs "in the wild", as in, e.g., articles, recipes, comics, etc., is less commonly used since their image-text relations are much more complex @xcite and the alignment of images and text is often left implicit. existing work on processing image-text alignment in multi-modal documents has usually been unsupervised, facing the challenge of missing evaluation and training data @xcite . for this reason, it is unclear to what extent state-of-the-art (multi-modal) language models can discover text-image alignments in complex multi-image multi-paragraph documents and to what extent grounding capabilities in these models are biased by specific linguistic properties of annotated captions. with this work, we contribute to closing this gap and provide one of the first datasets that provide ground-truth annotations of image-text alignment in complex multimodal documents. 1 figure 1 shows a paragraph from the wikipedia article on the reims cathedral foot_1 , illustrating some of the complexities that can arise in text-image alignment in real multimodal documents. the paragraph contains highly descriptive sentences that refer to visual elements of the building shown in corresponding images. thus, in this example, three sentences from the same paragraph match three different images, but there is no explicit alignment between sentences and images (e.g. through references). the paragraph also contains sentences that are not descriptive and do not match any of the images. at the same time, the images are accompanied by captions that briefly describe the image content and make it easier for the reader to establish its relation to the main text. furthermore, this paragraph is embedded in a much longer document which contains many more, possibly matching images of this building. these alignment patterns between images and sentences in a longer text as well as captions of these images and corresponding sentences have, to date, not been extensively studied in l&v research and there is currently no available dataset that provides annotations for textimage alignments in wikipedia articles. in this paper, we conduct an annotation study on an existing dataset of multimodal wikipedia articles on buildings, wikiscenes @xcite , and enrich the dataset with annotations of alignments between textual elements (sentences, paragraphs) and images. since the articles in wikiscenes are about visual entities from the domain of historical buildings, they feature text that is at times highly descriptive and, thereby, comparable to caption-like descriptions (see, e.g., the mention of the facade of the reims cathedral in figure 1 ). we restrict our annotation study to descriptive relations between text and images, i.e. textual elements that describe visual content shown in an image within the article, refraining from including more complex discourse relations involving complementary relations and others @xcite . to deal with the fact that the articles are rather long and contain many images, we introduce a two-step annotation procedure, where we first ask annotators to skim the article for relations between paragraphs and images, and then annotate sentence-image alignments in a second step. the dataset we obtain from our annotation setup, wikiscenes with descriptions, can enhance research on visual language grounding in longer documents and assess grounding capabilities in language models. our initial analyses in this paper focus on understanding how the descriptive sentences that occur within the main text and that match a particular image differ from captions of that image. we also experiment with baseline intra-document retrieval to evaluate l&v models on image-text alignment in our dataset. these analyses address the following research questions: our analyses reveal systematic linguistic differences between the image captions on the one and descriptive sentences from the article's text at both linguistic and conceptual levels. we show that our dataset can serve as a challenging benchmark for image-text alignment in long documents. ## background our data collection is related to other efforts focused on multi-modal articles, e.g., wikicaps @xcite and @xcite , or datasets for news image captioning @xcite @xcite . in comparison to these, our extension of @xcite 'swikiscenes features more detailed annotations of grounded text spans within sentences of the main text. annotation of relations between spans or entities in longer text is generally challenging, as discussed in, e.g., work on coreference @xcite ). annotation of multi-modal documents further comes with the significant complication that the number of possible combinations of text spans and images increases quadratically with the length of the text and the number of images. there is some work on l&v datasets and tasks that capture more varied semantic or discursive relations between image and text: @xcite al. (2019 @xcite propose to combine frameworks from the area of semiotics with computational analysis of image-text relations, suggesting a framework for multimodal news analysis. in contrast to these accounts, our dataset features more or less uniform relations between texts centered on buildings and images, i.e. the texts stand in a descriptive relation to the content of images. @xcite work with a more coarsegrained and somewhat simplified version of the problem discussed in this paper. their task is to correctly predict the physical alignment of images and sections in wikipedia articles. this approach utilizes the inherent document structure and consequently saves on expensive manual annotation. however, our observations call into question the presupposition that alignment in layout entails alignment in content. a similar text-image matching task is discussed in @xcite , where the authors seek to match the images in a document to the most relevant sentences in it (leaving out the captions). their model is trained on collections of sentences and images from the same documents or different documents, for instances of non-relatedness. this information is used at test time to estimate the individual links between the sentences and images of a given document. @xcite is highly relevant to the concerns discussed in this paper because it shows some success in handling comparatively large amounts of text in the genre of wikipedia articles. very recently, @xcite presented the documentclip model designed to capture the interaction of text and images in longer multimodal documents. importantly, they assume that images are, by default, aligned to the paragraph they co-occur with in the spatial document layout. this is a strong assumption and our dataset of ground-truth alignments between sentences, paragraphs, and images can be used to further test and benchmark such models. ## methods this section introduces the methods we use to analyze our dataset and to test l&v models on it. in our experiments, we look at two ways of aligning text and images: first, we study sentence-caption alignment, i.e. we investigate whether captions of images in an article are similar to sentences in the article's text that annotators marked as matching this image. second, we study sentence-image alignment using multimodal l&v models. ## results and analysis in this section, we analyze the relationship between images, captions, and sentences from a linguistic and application perspective. section 5.1 compares linguistic properties between captions and descriptions. then we conduct experiments on intra-document retrieval using the methods for sentence-caption and sentence-image alignment in section 4. , comparing the performance of unimodal and multimodal embedding models. ## discussion we introduced a dataset for text-image alignment in multi-paragraph, multi-image documents, connecting captioned images with text spans from the main text which are depicted in the image. our experiments show that these annotations provide a valuable benchmark dataset to evaluate the capabilities of zero-shot unimodal and multimodal pretrained models, that are challenged by image-text alignment in long and domain-specific documents. based on the results, we revisit our research questions and possible implications of our experiments for future research on multimodal documents. do descriptions of images in articles show different linguistic properties than captions of the corresponding images? yes. the analysis in section 5.1 shows that descriptive, matched sentences from the main text exhibit different pos and ner distributions compared to the original captions written by wikipedia authors. this highlights the importance of moving beyond the strong focus on captions in l&v research and indicates that different types of descriptions occurring within (and across) documents may exhibit different linguistic phenomena for visual language grounding. do the original captions in wikipedia differ systematically from captions generated by captioning models? partially. the analysis in section 5.1 indicates that original captions written by wikipedia authors differ in some aspects from the generated captions, which we expect to reflect the style of crowdsourced captions that many l&v models are currently trained on. this is not surprising but showcases that the style of captions collected in annotation and crowdsourcing experiments differs from naturally occurring captions found in real documents. this may bias or limit l&v models in a way that they do not encounter descriptive, visually grounded language in its full breadth in their pretraining data. are similarities between descriptive sentences within a text and captions robust enough to serve as a baseline for intra-document retrieval? partially. the results in section 5.2 show that intra-document retrieval for sentences and images via their captions works when the set of images/captions is restricted to the paragraph level, but drastically decreases at the article level. this holds for different types of captions. the retrieval score analysis shows inconclusive results in terms of the effect of captioning on different models. how do image-sentence retrieval baselines compare to caption-sentence retrieval? the results in section 5.2 show that sentence embeddings can distinguish more accurately between matched and unmatched sentences than multimodal models when looking at retrieval within an entire article. we believe that this may be because existing l&v models are typically trained on short texts that prioritize visually grounded language, but rarely on datasets of longer texts that include nondescriptive sentences. generally, it appears that the multimodal models we tested lack awareness of depictability (i.e. detecting language that is visually grounded). uni-modal sentence embedding models, on the other hand, seem to be less accurate in distinguishing grounded from non-grounded sentences at the more fine-grained paragraph level. for applications like intra-document retrieval in text-dominated documents, unimodal sentence embeddings still provide a better solution, but multimodal models have complementary strengths at the more fine-grained paragraph level distinctions. it