Visually-Aware Context Modeling for News Image Captioning (2024)

Abstract: News Image Captioning aims to create captions from news articles and images, emphasizing the connection between textual context and visual elements. Recognizing the significance of human faces in news images and the face-name co-occurrence pattern in existing datasets, we propose a face-naming module for learning better name embeddings. Apart from names, which can be directly linked to an image area (faces), news image captions mostly contain context information that can only be found in the article. We design a retrieval strategy using CLIP to retrieve sentences that are semantically close to the image, mimicking human thought process of linking articles to images. Furthermore, to tackle the problem of the imbalanced proportion of article context and image context in captions, we introduce a simple yet effective method Contrasting with Language Model backbone (CoLaM) to the training pipeline. We conduct extensive experiments to demonstrate the efficacy of our framework. We out-perform the previous state-of-the-art (without external data) by 7.97/5.80 CIDEr scores on GoodNews/NYTimes800k. Our code is available at https://github.com/tingyu215/VACNIC.

## introduction online news consumption heavily relies on news images as a key source of supplementary information alongside articles. these images, paired with engaging and informative captions, play a crucial role in capturing readers' attention. typically, a news image illustrates a portion of the article, with the caption linking the image content to the article. ideally, readers should be able to grasp the essence of the news article by browsing through its images and their corresponding captions. news image captioning, the task of generating a caption for an image using the contextual information derived from the corresponding article, contrasts with generic image captioning, where the news image caption: elizabeth warren, a democrat running for the senate, at a diner in shrewsbury, mass. ## related work in contrast to generic image captioning methods @xcite @xcite @xcite , which rely solely on images as input, news image captioning takes both images and news articles as input. it dictates that models should prioritize captions that not only depict image content, but also summarise the corresponding article segments. early approaches to the task focus on learning the representation of a news article and its connection with an image, including utilizing n-gram language models for extracting phrases seen in the article @xcite , or building an encoder-decoder based architecture with vgg (simonyan and zisserman, 2015) for encoding the image, word2vec @xcite for encoding the article, and an lstm as caption decoder bart @xmath0: generation: : mask if no face feed forward chain of person names pa self attention (b)face naming module name embedding hf hn face feat. k,v feed forward "dayan @xmath1,v (a) integrating features into bart he (b) face naming module hv : frozen image clip image encoder mlp retrieved segment: cubs' wood ends career with strikeout and hug…the adrenaline was the same, the nerves were the same…i can't give enough credit to the fans, just a tremendous feeling." ## conclusion in this paper, we introduce a new framework for utilizing visual inputs in news image captioning. inspired by human attention mechanisms, we developed a face naming module for aligning names with faces in images, based on face-name cooccurrence patterns. for context that cannot be visually grounded in the images, we utilize clip for sentence retrieval from articles, aiding comprehension. to address the imbalance between article and image context in captions, we introduce colam, guiding the model to focus more on article content. our extensive experiments demonstrate the effectiveness of our method, which achieves more than 6-point improvement in cider scores over the previous state-of-the-art on two commonly-used news image captioning datasets.