Focus! Relevant and Sufficient Context Selection for News Image Captioning (2022)

Abstract: News Image Captioning requires describing an image by leveraging additional context derived from a news article. Previous works only coarsely leverage the article to extract the necessary context, which makes it challenging for models to identify relevant events and named entities. In our paper, we first demonstrate that by combining more fine-grained context that captures the key named entities (obtained via an oracle) and the global context that summarizes the news, we can dramatically improve the model’s ability to generate accurate news captions. This begs the question, how to automatically extract such key entities from an image? We propose to use pre-trained vision and language retrieval model CLIP to localize the visually grounded entities in the news article, and then capture the non-visual entities via a open relation extraction model. Our experiments demonstrate that by simply selecting better context from the article, we can significantly improve the performance of existing models and achieve the new state-of-the-art performance on multiple benchmarks.

## introduction news image captioning is an extension of the standard image captioning task, where, besides the image, one needs to leverage longer context in the form of the news article. this is important in order to capture the global news story context to properly discuss the image. at the same time, the specific entities (e.g. people) in the image are also often referenced in the article. coincidentally, generating the named entities @xcite , is one of the most critical challenges in news image captioning. we analyze the captions and the corresponding articles on three popular benchmarks, goodnews @xcite , ny800ktimes @xcite , visualnews @xcite , and find that 58.7% -74.5% of the named entities present in captions also appear in the articles. this shows that the key information of a news caption, namely the named entities, can often be directly derived from the associated news article. however, due to the length and abundance of other information in the full articles, it is challenging to uncover the key entities. inspired by this, we explore how to select relevant yet sufficient context from the news article to assist with the news image caption generation. to analyze what the right type of context is, we start with an oracle-based study where we demonstrate that combining key local context and global context leads to the best result. next, we move to designing a multimodal retrieval method to handle automatic extraction of key local context (key named entities). our method consists of two stages, first we retrieve the visually grounded entities, then we discover the non-visual ones via open relation extraction. figure 1 illustrates our approach. our full method outperforms the vanilla baseline on three benchmarks, and achieves the new state-ofthe-art results on two of them. ## related work automatic image captioning has achieved tremendous success over the past few years, where many methods @xcite @xcite are developed to accurately describe the visual objects and their relationships in the image. recently, news image captioning has started to get increased attention with the focus on producing narration of news images with richer human-like information derived from the news article. the research on news image captioning is inspired by several recently collected benchmark datasets including breakingnews @xcite , goodnews @xcite , ny800ktimes @xcite , and visual-news @xcite . earlier works such as @xcite and @xcite propose a two-stage pipeline, where they first generate template captions and then insert named entities into corresponding positions. @xcite propose a hierarchical article encoding mechanism, where the model first retrieves the most relevant sentences and then attends to the appropriate words in the retrieved context with images involved in both steps. later works, like @xcite and @xcite introduce end-to-end models that directly encode articles and images with pre-trained transformer architectures (bert @xcite and roberta @xcite ), then generate the caption directly. the end-to-end method is better than the template-based approach as it models the rich contextual information in named entities using byte-pair encoding (bpe) @xcite . a recent work by @xcite proposes to follow journalistic guidelines when generating news image captions, i.e. each caption must contain the who, when, where, and how entities. there is rarely any effort spent on context selection to help news caption generation in previous work. the lstm-based method such as @xcite and @xcite coarsely encode the full article either as a general embedding vector or a set of sentence embedding vectors. later, the transformer-based approach such as @xcite and @xcite directly encode the article at the token-level and achieved much better performance than the lstm-based approach. however, they usually need to truncate the article to meet the encoding length limitation of the trans-former architecture, which inevitably leads to the loss of useful information. @xcite is one of few works that addresses context selection for news image caption, where they train a cross-modal retrieval model vse++ @xcite to find the sentences from the article that are most relevant to the image. however, the retrieved relevant sentences from vse++ are limited to only capture visually grounded information. we argue that other valuable information, i.e. non-visual named entities, should be included as the relevant context to help news image caption generation. ## oracle study: optimal news context selection we posit that there are two critical types of context in the news article that we need to extract to help the model generate accurate news image captions: ## automatic key local context detection in the previous section, we identified an optimal strategy for selecting news context, which relied on the named entities found in the ground truth captions. in practice, automatically determining the key named entities remains an open challenge. to address it, we propose a simple multi-modal retrieval pipeline to extract the key named entities from the news article. the pipeline consists of two stages: 1. we identify visually grounded named entities, such as people's names and geographic locations, via cross-modal retrieval; 2. we discover the key named entities that are not visually grounded, e.g. time, by exploring factual relations between the entities in the article and the detected visually grounded entities. visually grounded named entities retrieval according to @xcite , the named entities in the news captions can be generally divided into four groups: who, when, where, and misc. the mapping between the spacy named entity types and each category can be found in to localize the visually grounded named entities in the news article, we first find the relevant sentences from the entire news article that are se-mantically close to the image. our approach is to leverage a large pre-trained cross-modal retrieval model clip @xcite trained on over 400 million image-text pairs from the internet. the sentence relevance is measured by the cosine similarity with the image in the learned embedding space of clip. as there is a large domain discrepancy between news sentences and the training data of clip, we first need to fine-tune clip on the news image captioning dataset. the clip model is fine-tuned with a contrastive loss to distinguish the positive image-sentence pairs from the negative pairs. for the positive sentences we use the news image captions since there are no labels for which news article sentences are relevant to each image, and the writing style of the captions is close to that of the article sentences. besides using captions describing other images as negative samples, we also define hard negatives as news article sentences that do not cover any non-stop words in the news caption. during image captioning, we use the trained clip model to rank news sentences against the image and pick the top 2 as the most relevant sentences. the visually grounded named entities are then retrieved as the who and where entities contained in the retrieved relevant sentences. if the top 2 retrieved sentences do not contain any visually grounded entity, we keep searching the rest of the retrieved sentences descending by similarity score to the news image until at least a visually grounded entity is captured. non-visually grounded named entities retrieval we define the named entities belonging to the rest of the categories: when and misc as non-visually grounded name entities. such entities are not semantically related to the image and are more challenging to localize via cross-modal retrieval. however, they are often related to the visually grounded named entities in the news text. for example, in the sentence ali kashani-rafye, started selling bastani (persian for ice cream) in 1980 at his grocery., the time entity 1980 is associated with the person's name ali kashani-rafye with the action selling bastani. the task of extracting such relations between named entities is known as open relation extraction. we leverage an open relation extraction method opennre @xcite to detect all the relations between each pair of named entities in every sentence of the news. we filter out the relations with a detection confidence score lower than 0.7. after filtering, if there is a relation extracted between a non-visually grounded named entity and a detected visually grounded named entity from the cross-modal retrieval, we use such non-visually grounded named entities during final context extraction. news context selection once we get the full list of the detected named entities, we follow the optimal strategy derived in sec 3 to select the news context. we combine the local-focused context guided by the detected named entities and the global context that is consisted of the title and the first paragraph to format the relevant news context. ## experiment in this section, we demonstrate the impact of context section for news image captioning in the automatic setting. we first introduce the baselines and the evaluation datasets. then we discuss our findings from the experimental results. ## conclusion our paper explores news context selection to improve the performance of existing models on news caption generation. our proposed strategy to select relevant context is two-fold: (1) include the key local context that focuses on the key information in the news caption, namely the named entities. (2) include the global context, such as titles and the first paragraph, that summarizes the news story. both parts are validated in an oracle setting with a strong news captioning model, transform-and-tell (tell) @xcite , across multiple benchmarks. we further study key local context selection in a practical (automatic) setting, where we first detect a set of visual named entities with a multi-modal retrieval pipeline, find related non-visual named entities via relation extraction, then use these named entities to pick relevant news article sentences. the performance of tell is consistently improved using our proposed context selection, and it achieves the new state-of-the-art on two benchmarks. we hope that our findings will inspire future work to develop models that even more effectively leverage the relevant context in the article to improve caption generation quality. meanwhile, our proposed method can also be potentially extended to select a relevant context from external sources to cover the key named entities missing from the original news article.