Visual News: Benchmark and Challenges in News Image Captioning (2021)

Abstract: We propose Visual News Captioner, an entity-aware model for the task of news image captioning. We also introduce Visual News, a large-scale benchmark consisting of more than one million news images along with associated news articles, image captions, author information, and other metadata. Unlike the standard image captioning task, news images depict situations where people, locations, and events are of paramount importance. Our proposed method can effectively combine visual and textual features to generate captions with richer information such as events and entities. More specifically, built upon the Transformer architecture, our model is further equipped with novel multi-modal feature fusion techniques and attention mechanisms, which are designed to generate named entities more accurately. Our method utilizes much fewer parameters while achieving slightly better prediction results than competing methods. Our larger and more diverse Visual News dataset further highlights the remaining challenges in captioning news images.

## introduction image captioning is a language and vision task that has received considerable attention and where important progress has been made in recent years @xcite @xcite @xcite . this field has been fueled by recent advances in both visual representation learning and text generation, and also by the availability of image-text parallel corpora such as the common objects in context (coco) captions dataset @xcite . while coco contains enough images to train reasonably good captioning models, it was collected so that objects depicted in the images are biased toward a limited set of everyday objects. moreover, while it provides high-quality human * work completed before joining amazon. president obama and mitt romney debate in hempstead ny on tuesday. virginia cavaliers fans celebrate on the court after the cavaliers game against the duke blue devils at john paul jones arena. a baseball player hitting the ball during the game. a bunch of people who are holding red umbrellas. figure 1 : examples from our visual news dataset (left) and coco @xcite . visual news provides more informative captions with name entities, whereas coco contains more generic captions. in this paper, we introduce visual news, a dataset and benchmark containing more than one million publicly available news images paired with both captions and news article text collected from a diverse set of topics and news sources in english (the guardian, bbc, usa today, and the washington post). by leveraging this dataset, we focus on the task of news image captioning, which aims at generating captions from both input images and corresponding news articles. we further propose visual news captioner, a model that generates captions by attending to both individual word tokens and named entities in an input news article text, and localized visual features. news image captions are typically more complex than pure image captions and thus make them harder to generate. news captions describe the contents of images at a higher degree of specificity and as such contain many named entities referring to specific people, places, and organizations. such named entities convey key information regarding the events presented in the images, and conversely events are often used to predict what types of entities are involved. e.g. if the news article mentions a baseball game then a picture might involve a baseball player or a coach, conversely if the image contains someone wearing baseball gear, it might imply that a game of baseball is taking place. as such, our visual news captioner model jointly uses spatial-level visual feature attention and word-level textual feature attention. more specifically, we adapt the existing tranformer @xcite to news image datasets by integrating several critical components. to effectively attend to important named entities in news articles, we apply the attention on attention technique on attention layers and introduce a new position encoding method to model the relative position relationships of words. we also propose a novel visual selective layer to learn joint multimodal embeddings. to avoid missing rare named entities, we build our decoder upon the pointergenerator model. news captions also contain a significant amount of words falling either in the long tail of the distribution or resulting in out-ofvocabulary words at test time. in order to alleviate this, we introduce a tag cleaning post-processing step to further improve our model. previous works @xcite have attempted news image captioning by adopting a two-stage pipeline. they first replace all specific named entities with entity type tags to create templates and train a model to generate template captions with fillable placeholders. then, these methods search in the input news articles for entities to fill placeholders. such approach reduces the vocabulary size and eases the burden on the template generator network. however, our pope emeritus benedict xvi left and pope francis greet each other in st peter's basilica vatican city pope francis installed 19 new cardinals saturday in a ceremony that unexpectedly included pope emeritus benedict xvi marking the first time the two appeared together in public. this batch of new cardinals the first appointed by francis is significant because the group includes prelates from developing countries such as burkina faso and haiti in line with the pope's belief that the church should do more to help the world s poor saturday's ceremony also helped move the spotlight away from more controversial topics ... ## related work image captioning has gained increased attention, with remarkable results in recent benchmarks. a popular paradigm @xcite @xcite uses a convolutional neural network as the image encoder and generates captions using a recurrent neural network (rnn) as the decoder. the seminal work of @xcite proposed to attend to different image patches at different time steps and @xcite improved this attention mechanism by adding an option to sometimes not to attend to any image regions. other extensions include at news image captioning is a challenging task because the captions often contain named entities. prior work has attempted this task by drawing contextual information from the accompanying articles. @xcite select the most representative sentence from the article; @xcite encode news articles using pre-trained word embeddings and concatenate them with cnn visual features to feed into an lstm @xcite ; @xcite propose a template-based method in order to reduce the vocabulary size and then later retrieves named entities from auxiliary data; @xcite also adopt a template-based method but extract named entities by attending to sentences from the associated arti-guardian bbc usa wash. 3 our visual news dataset visual news comprises news articles, images, captions, and other metadata from four news agencies: the guardian, bbc, usa today, and the washington post. to maintain quality, we first filter out images whose height or width is smaller than 180 pixels. we then keep examples with a caption length between 5 and 31 words. figure 2 shows some examples from visual news. although only images, captions, and articles are used in our experiments, visual news provides other metadata, such as article title, author, and geo-location. we summarize the difference between visual news and other popular news image datasets in figure 3 shows the average count of named entity types in captions from each agency. for instance, usa today has on average 0.84 "person" entities per caption while bbc has only 0.46. the washington post has 0.29 "date" entities whereas usa today has 0.47. we also randomly select 50, 000 captions from each agency and calculate their unique named entities to see how many they have in common with each other (as summarized in to further demonstrate the diversity in visual news, we train a show and tell @xcite captioning model on 100, 000 examples from a certain agency and test it on 10, 000 examples from the other agencies. we report cider scores in ## method figure 4 presents an overview of visual news captioner. we first introduce the image encoder and the text encoder. we then explain the decoder in section 4.3. to solve the out-of-vocabulary issue, we propose tag-cleaning in section 4.4. ## experiments in this section, we first introduce details of implementation. then baselines and competing methods will be discussed. lastly, we present comprehensive experiment results on both the good-news dataset and our visual news dataset. ## conclusion and future work in this paper, we study the task of news image captioning. first, we construct visual news, the largest news image captioning dataset consisting of over one million images with accompanying articles, captions, and other metadata. furthermore, we propose visual news captioner, an entity-aware captioning method leveraging both visual and textual information. we validate the effectiveness of our method on three datasets through exten-