Image Caption Generation for News Articles (2020)

Abstract: In this paper, we address the task of news-image captioning, which generates a description of an image given the image and its article body as input. This task is more challenging than the conventional image captioning, because it requires a joint understanding of image and text. We present a Transformer model that integrates text and image modalities and attends to textual features from visual features in generating a caption. Experiments based on automatic evaluation metrics and human evaluation show that an article text provides primary information to reproduce news-image captions written by journalists. The results also demonstrate that the proposed model outperforms the state-of-the-art model. In addition, we also confirm that visual features contribute to improving the quality of news-image captions.

## introduction image captioning, i.e., automatic generation of a natural language description from an image, has received much attention from both fields of computer vision (cv) and natural language processing (nlp) @xcite @xcite . image captioning is important not only in practical application (e.g., automatic indexing of images) but also as a challenge of image understanding (e.g., recognition of objects, object-object relationships, and scenes). in this paper, we address a more advanced task, news-image captioning: this task generates a description of an image, given the image and its article body as input. the news-image captioning task is different from the conventional image captioning, which receives only an image as input. in other words, news-image captioning requires a mutual understanding of image and text. early work proposed a two-stage approach for news-image captioning @xcite . the first stage annotates keywords to a given image and text, and the second stage realizes a description based on the extracted keywords. recent work presented an end-to-end approach that integrates image and text features in deep neural networks @xcite @xcite . however, the previous studies did not focus on the usefulness of text in the news-image captioning task, extending the conventional models for image captioning to incorporate text features. figure 1 shows an example of a news-image caption. it may be difficult to recognize which is the central object in the image, for example, people, violins, and stick (bow). also, the caption includes much information (e.g., juilliard orchestra, vladimir jurowsky, and alice tully hall) that may be hard to tell only from the image. this kind of example is rather common in news articles, where a text is the major medium of information, and an image and its caption provide additional explanations that support the text. however, no previous work explored a method that integrated an article text seamlessly with an image in the task of news-image captioning. in this paper, we present a method for news-image captioning based on transformer @xcite , a successful architecture for various nlp tasks, including machine translation, abstractive summarization, contextualized word embeddings. we propose a transformer model that integrates text and image modalities and attends to textual features from visual features in generating a caption. the experimental results show that an article text provides primary information to reproduce news-image captions written by journalists. the results also demonstrate that the proposed model outperforms the state-ofthe-art model (biten et al., 2019). we report the results of human evaluation and discuss challenges in news-image captioning. the dataset and code used in this work are publicly available foot_0 . ## multimodal transformer model given an image i and a text as a sequence of n tokens (x 1 , x 2 , . . . x n ), the task of news-image captioning generates a caption as a sequence of m tokens (y 1 , y 2 , . . . y m ). as the base architecture for realizing the task, we use the transformer model @xcite , which has been a popular architecture for headline generation for news articles @xcite @xcite . because headline generation is a sequence-to-sequence task from (x 1 , . . . x n ) to (y 1 , . . . y m ), we consider incorporating features from an image i to the architecture. figure 2 illustrates the proposed model. the model consists of image encoder, image-article encoder, and decoder. the image encoder converts an image i into a feature vector p @xmath0, here, d presents the number of dimensions of hidden feature vectors, and cnn(.) is a convolutional neural network to convert an image into a feature vector. in this study, we compare two cnn models trained on the datasets of object recognition and scene recognition as cnn(.). the image-article encoder computes joint representations for an input image and text. the input to the first layer of the image-article encoder is a d × n matrix, whose column vectors present an addition of token embedding and positional encoding. the image-article encoder stacks n layers of two submodules: encoder module and image-attending module. the encoder module is identical to the encoder component of the original transformer model: it is a mapping from r d×n to r d×n by using a multi-head self-attention followed by a feed-forward layer. the image-attending module is similar to the decoder component of the original transformer model: it is a mapping from r d×n to r d×n by using a multi-head target-source attention followed by a feedforward layer. here, we provide the image vector p i as keys and values for the scaled dot-product attention so that this module can build intermediate representations by attending to the input image. the image-attending module adds the outputs from the encoder module and the target-source attention via the residual connection. thus, we expect that target-source attention can focus on implicit contextual correlations between visual and textual features, modifying input representations based on the input image. the decoder has m layers, each of which is exactly the same as the original transformer decoder. a decoder layer consists of multi-head masked self-attention over output tokens, followed by the targetsource attention that receives the encoder output as keys and values of the scaled dot-product attention. in this way, the overall architecture can generate a caption by selectively drawing attention to the important information of the image and text from the news article. ## experiments the questions we want to explore in this section are two-folds: to answer these questions, we select the news-image captioning dataset: goodnews, and present fixes for its issues in section 3.1. after explaining the experimental settings (in sections 3.2 to 3.5), we report evaluation results by using the automatic evaluation (in section 3.6) followed by the human evaluation (in section 3.7). ## related work image captioning recent advances in image captioning followed the encoder-decoder architecture, where the encoder extracts visual features from images, and the decoder generates the caption @xcite @xcite . notably, the attention mechanism further improved the quality of generated captions by finding implicit alignments between visual and textual features. these models achieved good performance in generating captions at a descriptive level with a consistent writing style. different from generic image captions, news-image captions are often influenced by contextual information from the news article. news-image captioning news-image captioning has been paid little attention in the literature. early work @xcite presented models with two stages: the first stage is an annotation model that suggests keywords for an image; and the second stage realizes sentences based on the extraction result of the first stage. recent studies departed from these approaches by utilizing the deep neural network to find the implicit image-text correlations @xcite . @xcite is the most recent method yielding the state-of-art performance on the goodnews dataset. the method consists of two stages: in the first stage, long short-term memory (lstm) combines textual features at sentence-level and visual features at object-level to generate a template sentence. the second stage then inserts named entities into the placeholder of the template sentence to realize captions with named entities. in contrast, our method required no template sentence and included named entities directly from article text based on the transformer architecture. ## conclusion in this paper, we presented a method for news-image captioning based on the transformer model, which integrates text and image modalities and attends to textual features from visual features in generating a caption. the experimental results demonstrated that the proposed model could integrate visual and textual information in generating captions. meanwhile, we also found that news-image captioning as a context-oriented image captioning task, text from news article fundamentally contributes our model in generating context-coherent captions, with the engagement of image further improves qualities. in the future, we would like to explore a better approach for recognizing visual contents from images to improve the quality of generated captions. besides, we are also interested in whether a model of news-image captioning is transferrable to other multimodal tasks such as multimodal translation and visual question answering.