Concadia: Towards Image-Based Text Generation with a Purpose (2022)

Abstract: Current deep learning models often achieve excellent results on benchmark image-to-text datasets but fail to generate texts that are useful in practice. We argue that to close this gap, it is vital to distinguish descriptions from captions based on their distinct communicative roles. Descriptions focus on visual features and are meant to replace an image (often to increase accessibility), whereas captions appear alongside an image to supply additional information. To motivate this distinction and help people put it into practice, we introduce the publicly available Wikipedia-based dataset Concadia consisting of 96,918 images with corresponding English-language descriptions, captions, and surrounding context. Using insights from Concadia, models trained on it, and a preregistered human-subjects experiment with human- and model-generated texts, we characterize the commonalities and differences between descriptions and captions. In addition, we show that, for generating both descriptions and captions, it is useful to augment image-to-text models with representations of the textual context in which the image appeared.

## introduction image-based natural language generation (nlg) has great potential for productive applications in accessibility, image search, creative writing, and navigational instruction tasks, among other areas. however, current systems fall short of realizing this potential even though they can generally produce fluent, truthful texts in a wide range of scenarios (e.g., @xcite @xcite . in this paper, we identify and address one of the fundamental limitations holding these systems back: insufficient attention to the communicative purpose of the text being generated. we focus on the high-level distinction between descriptions and captions. descriptions are created for the purpose of replacing an image in context, figure 1 : an example from our wikipedia-based corpus concadia of an image from the article on bananas with two associated texts: the description as provided in the image's alt text, and the caption as displayed below the image in the article. for captions, the image content is presupposed whereas descriptions aim to stand in for the image. while grounded in the same image, description and caption convey vastly different information. to enable in-depth exploration of the description/caption distinction, we introduce concadia, a wikipedia-based corpus of 96,918 images with associated english-language captions, alt text descriptions, and accompanying context from the respective wikipedia article. concadia provides the opportunity to directly compare descriptions and captions for shared images, and it can be used to develop image-based nlg systems that serve these distinct communicative purposes as well. to further substantiate the description/caption distinction, we report on a linguistic analysis of the commonalities and differences between these two kinds of text in wikipedia (section 3). we then develop and assess image-based nlg systems using transformer-and lstm-based architectures. two clear lessons emerge from these modeling efforts. first, description generation is easier than caption generation, as one might expect given the tighter relationship between descriptions and images. second, it is highly beneficial to include representations of the textual context in which an image appeared. this finding aligns with much recent work showing that blind and low-vision (blv) users value image descriptions that tie the image to its surrounding context @xcite @xcite . @xcite identify benefits of using context in the caption domain, for instance, for inferring names of people visible in an image. in the example in figure 1 , both the caption and description reflect the context (commercial aspects of bananas), and the caption additionally connects to the type of banana discussed in the article. most datasets for imagebased nlg provide only image-text pairs, with no context, which may fundamentally limit their usefulness. a strength of concadia is that all the image-text pairs are given within their wikipedia contexts. finally, we provide evidence from a preregistered human-subjects experiment (section 5). our results further reinforce the observation that descriptions and captions serve distinct communicative purposes. in addition, the results hold not only for the descriptions and captions from concadia but extend to our model-generated texts as well. this serves to further validate that systems trained to generate captions are inadequate for the task of generating descriptions, and vice versa, and that model evaluations need to be sensitive to communicative purpose. together with the modeling results, this speaks to the importance of both communicative purpose and context when it comes to developing effective image-based nlg systems. ## captions vs. descriptions in prior work in ai, the task of generating prose text from images is often referred to as "image captioning", which glosses over distinctions of communicative purpose. the current section offers a synopsis of research, within and outside of ai, that is relevant to our proposed description/caption distinction and supports a more nuanced view of image-based nlg tasks. the conceptual description/caption distinction we propose poses a terminological challenge. we use "caption" to mean "image-supporting information expressed in text", which we contrast with "(alt) description" to mean "text intended to replace an image". this is aligned with general popular usage and with the terminology in (photo-)journalism and graph accessibility, fields that conceptually already make a description/caption distinction. however, the term "caption" is now a loaded term in ai, and future work should be careful in considering the conceptual distinctions despite the challenges in terminology. ## the concadia dataset we introduce concadia, contextualized captions and alt descriptions from wikipedia, a corpus extracted from wikipedia consisting of images with their naturally occurring alt descriptions, captions, and surrounding context. concadia allows for a direct comparison of descriptions and captions for a specific image in a specific context. in this section, we characterize the distribution of captions and descriptions on all of wikipedia, describe the concadia dataset generation process, and provide a qualitative comparison on the similarities and dif-ferences of the captions and descriptions in concadia. appendix a provides implementation details and further illustrations. ## model experiments our investigation of the similarity between captions and descriptions in section 3.3 indicates that they differ in principled ways but that they are not unrelated. this suggests that, while captions and descriptions can't stand in for each other, they can potentially inform each other's content when we automatically generate them. we further motivated that there is evidence that both description and caption generation would benefit from integrating the larger context where the image occurred. we now pursue these hypotheses in the context of largescale caption and description generation. ## human evaluation in section 3.3 and section 4, we provided initial evidence that descriptions and captions fulfill separate communicative purposes. we now present a human-subject experiment in which we directly test whether captions are more suitable than descriptions for providing information that can't be obtained from the image alone, and whether descriptions are more useful than captions for replacing an image. furthermore, we investigate whether models might capture these differences. our experiment focuses on the resnet-lstm model, the weakest of our models, which should then provide a conservative assessment of how closely automatic generations align with human texts. our quantitative and qualitative hypotheses, along with our exclusion criteria and the analyses, were preregistered. 5 details on the recruitment and data exclusions are provided in appendix c. ## conclusion we argue for a distinction between descriptions (text intended to replace an image) and captions (text intended to contextualize an image), and we provide evidence for their similarities and dif-ferences through linguistic analyses and humansubject evaluations. we introduce a corpus, concadia, consisting of images and their corresponding alt descriptions, captions, and context, and we show that concadia is a valuable resource for training nlg systems for both description and caption generation. the structure of concadia additionally allows us to show that both generation tasks benefit from contextual information. beyond the conceptual relevance for advancing image-based nlg systems, this has practical implications specifically for image accessibility. captions which often co-occur with images that lack an accessibility description could be a valuable resource for building more powerful description generation models.