Exploiting Image–Text Synergy for Contextual Image Captioning (2021)

Abstract: Modern web content - news articles, blog posts, educational resources, marketing brochures - is predominantly multimodal. A notable trait is the inclusion of media such as images placed at meaningful locations within a textual narrative. Most often, such images are accompanied by captions - either factual or stylistic (humorous, metaphorical, etc.) - making the narrative more engaging to the reader. While standalone image captioning has been extensively studied, captioning an image based on external knowledge such as its surrounding text remains under-explored. In this paper, we study this new task: given an image and an associated unstructured knowledge snippet, the goal is to generate a contextual caption for the image.

## introduction in multimodal (image-text) documents, images are typically accompanied by captions. these may contain specific knowledge about the narrative -location, names of persons etc. -or may exhibit thematic knowledge grounding the sentimental value of the image in the narrative. the image captions explicitly or implicitly refer to the image and its surrounding text, and play a major role in engaging the reader. we call this type of captions contextual captions, and figure 1 illustrates the corresponding contextual image captioning problem. generating captions for standalone images @xcite or summarizing a piece of text @xcite are well-studied problems. however, generating image captions accounting for contextual knowledge is a largely unexplored task and poses many challenges. related tasks include multimodal summarization @xcite and title generation @xcite . multimodal summarization usually involves segmentation and i recently moved to buffalo, ny and every day i am discovering how beautiful this town is. i took this pic...and i was thrilled about it! i wanted to share the pallet of colors the sunset had that evening. ## related work image captioning. prior work on captioning conditioned only on images @xcite @xcite has been successful for descriptive captions with explicit grounding to image objects. recently, captions with sentimental and abstract concepts have been explored @xcite @xcite @xcite . although external knowledge bases like dbpedia (factual knowledge) @xcite and conceptnet (commonsense knowledge) @xcite have been leveraged, all prior work ignores the knowledge present in the text surrounding images in social media and other domains. contextual image captioning leverages the latter kind of knowledge. multimodal summarization. research on multimodal embeddings @xcite @xcite has facilitated studying image-text data. summarization of multimodal documents @xcite @xcite proceeds by aligning a subset of images with extracted (chen and zhuge, 2019a), or generated (chen and zhuge, 2018) text segments from the original document. in contrast, image captions in our dataset do not explicitly summarize the associated text and rather act as a short commentary that captures knowledge from both modalities. news image captioning. a task similar to our problem is captioning images within news articles @xcite . a key challenge here is to correctly identify and generate named entities @xcite . however, news image captions tend to be descriptive compared to the subjective nature of captions in our dataset representing common social media content. ## datasets to the best of our knowledge, the only existing image-text caption datasets are from the news domain (e.g., daily mail corpus) containing nonaffective descriptive captions with mentions of named entities. instead, we consider reddit, which offers a rich source of multimodal data. out of the image-related subreddits, /r/pics is particularly suitable for our problem because of the nature of posts. firstly, the posts do not contain expert jargon, unlike other subreddits like /r/photographs. secondly, the image captions are mostly affective and not drab descriptions. lastly, post frequency is high, presenting a big dataset. data scraping. we scrape the subreddit /r/pics to collect 250,000 posts over the span of a year. for each post, we grab the image, the post title, and 1-10 comments. we consider the post title as ground truth caption since it is written by the image poster, ensuring a consistent and coherent intent. the comments are concatenated, preserving their tree structure, to serve as the unstructured knowledge associated with the image. inappropriate posts that do not adhere to community standards and were flagged by moderators are removed. data characteristics. the collected images do not adhere to any particular subject or theme. the paragraphs are ~59.2 words long, and the captions are ~10.6 words long on an average. in some posts, captions and comments may contain different named entities (ne), making prediction of the ground truth ne difficult. for example, the caption "my friend and i are en route to the grand canyon" may be accompanied with the comment "try to hike down to the colorado. also visit zion national park!" the nes in the paragraph (colorado, zion) do not match that in the caption (grand canyon). owing to this characteristic, we study two distinct variants of the dataset -one containing nes and the other without nes. we denote these variants as +ne and ne, respectively. the comments sometimes exhibit topic drift, e.g., a comment on the grand canyon post may be "i remember my last trip to india...we had spicy food!". hence, we also study variants with ensured context overlap -one common word (ignoring stop words) between caption and comments. these variants are suffixed overlap -e.g. +ne-overlap. we report experimental results on all of these variants, adopting a 30,000/8,000/8,000 train/val/test split for each of them. ## contextual captioning model we refer to the text context associated with each image as 'paragraph'. this offers external knowledge which may be absent in the image modality alone. figure 2 shows our proposed model architecture. given an input image i and an associated input paragraph @xmath0, . . . , w p m } of length m , our model (an encoder-decoder architecture) generates a caption @xmath1, . . . , w c n } of length n . for image encoding, we use features extracted from a pre-trained resnet152 @xcite model. to encode the input paragraph, we deploy a bidirectional lstm (bilstm). the outputs of the bilstm, denoted as @xmath2, . . . , g m }, where at each decoder time step @xmath3, . . . n }, the attention weights α t over the encoder states depend on the current decoder state h t and the encoder states g. formally, finally, we pass the concatenated output through two dense layers with a non-linear activation layer (e.g, relu) placed in between. the output logits are then passed through a softmax function to obtain the output distribution p(.) over the vocabulary. we train our model end-to-end by minimizing the negative log-likelihood, i.e., θ * = argmin θ -log p(c | i, p ; θ). note that we obtain the input embeddings, x i , and y t , of the encoder and decoder, respectively, from the embedding layer of a pretrained bert base model. the model's objective is to learn the optimal parameters θ * to maximize the log-likelihood log p(c|i, p ; θ). therefore, we train our model end-to-end by minimizing the negative loglikelihood defined as: ## conclusion we propose the novel task of contextual image captioning that exploits complementary knowledge from different modalities in multimodal contents. to facilitate a thorough study of this task, we curate a new dataset comprising ∼250,000 multimodal reddit posts. we provide an analysis of the dataset along with experimental results to identify interesting factors that determine the quality of generated contextual captions. we hope that our work will kindle and support follow-up research on this under-explored task, with downstream applications such as content authoring tools and multimodal conversational agents.