Journalistic Guidelines Aware News Image Captioning (2021)

Abstract: The task of news article image captioning aims to generate descriptive and informative captions for news article images. Unlike conventional image captions that simply describe the content of the image in general terms, news image captions follow journalistic guidelines and rely heavily on named entities to describe the image content, often drawing context from the whole article they are associated with. In this work, we propose a new approach to this task, motivated by caption guidelines that journalists follow. Our approach, Journalistic Guidelines Aware News Image Captioning (JoGANIC), leverages the structure of captions to improve the generation quality and guide our representation design. Experimental results, including detailed ablation studies, on two large-scale publicly available datasets show that JoGANIC substantially outperforms state-of-the-art methods both on caption generation and named entity related metrics.

## introduction research on generating textual descriptions of images has made great progress in recent years with the introduction of encoder-decoder architectures @xcite @xcite @xcite @xcite . those models are generally trained and evaluated on image captioning datasets like coco @xcite and flickr @xcite that only contain generic object categories but no details such as names, locations, or dates. the captions generated by these methods are thus generic descriptions of the images. the news image captioning problem @xcite @xcite ) can be seen as a multimodal extension of the image captioning task with additional context provided in the form of a news article. specifically, given image-article pairs as input, the news captioning task aims to generate an informative caption that describes the image with proper named entities and context extracted from the article. the development of automatic news image caption generation methods can ease the process of adding images to articles and produce more engaging content. according to the news manual foot_0 and international journalists' network foot_1 , a caption should help news readers understand six main components @xcite related to the image and article. as shown in fig. 1 , different journalists can write captions to cover different components for the same image and article pair. previous news image captioning work @xcite has not directly addressed the challenge of generating a caption that follows those journalistic principles. in this work, we tackle the news image captioning problem by introducing these guidelines in our modeling through a new concept called a 'caption template', which is composed of 5 key components, detailed in section 3. we propose a journalistic guidelines aware news image captioning (joganic) model that, given an image-article pair, aims to predict the most likely active template components and, using component-specific decoding block, produces a caption following the provided template guidance. joganic thus models the underlying structure of the captions, which helps to improve the generation quality. captions for images that accompany news articles often include named entities and rely heavily on context found throughout the article (making the text encoding process especially challenging). we propose two techniques to address these issues: (i) integration of features specifically to extract relevant named entities, and (ii) a multi-span text reading (mstr) method, which first splits long articles into multiple text spans and then merges the extracted features of all spans together. our work has two main contributions: (i) the definition of the template components of a news caption based on journalistic guidelines, and their explicit integration in the caption generation process of our joganic model; (ii) the design of encoding mechanisms to extract relevant information for the news image captioning task throughout the article, specifically a dedicated named entity representation and the ability to process longer article. experimental results show better performance than state of the art on news image caption generation. we release the source code of our method at https://github.com/dataminr-ai/joganic . ## defining caption templates the objective of news image captioning is to give the reader a clear understanding of the main components who, when, where, what, why and how depicted in the image given the context of the article. we propose to exploit the idea of components in the caption generation process, but we first need to define components that can be automatically detected in the ground truth caption for training. the who, when and where components can be retrieved via named entity recognition (ner). as shown in the right column of tab. 1, we define named entities with type 'person', 'norp' and 'org' as who, those with type 'date' and 'time' as when, and ones with type 'fac', 'gpe' and 'loc' as where. we define the component misc as the rest of the named entities. the what, why and how components are hard to define and can correspond to a wide range of elements, we propose to merge them into a context component, which we assume is present if a verb is detected by a part-ofspeech (pos) tagger foot_2 . in fig. 1 , captions 1 and 2 have an context component, but caption 3 does not contain a verb and thus has no context. in summary, our proposed news caption template consists of at most five components: who, when, where, context and misc. we report the percentage of each component in the captions of the good-news and nytimes800k datasets in tab. 2. the who is present in almost all the captions, and all components appear commonly in both datasets. ## template-guided news image captioning in this section, we formally define the news captioning task and introduce the idea of template guidance and our journalistic guidelines aware news image captioning (joganic) approach. we then propose two strategies to address the specific challenges of named entities and long articles. ## experiments we evaluate joganic on two large-scale publicly available news captioning datasets: goodnews @xcite and nytimes800k @xcite both collected using the new york times public api 5 , with the latter being larger and con- ## conclusion news image captioning is a challenging task as it requires exploiting both image and text content to produce rich and well structured captions including