Exploring the Impact of Vision Features in News Image Captioning (2023)

Abstract: The task of news image captioning aims to generate a detailed caption which describes the specific information of an image in a news article. However, we find that recent state-of-art models can achieve competitive performance even without vision features. To resolve the impact of vision features in the news image captioning task, we conduct extensive experiments with mainstream models based on encoder-decoder framework. From our exploration, we find 1) vision features do contribute to the generation of news image captions; 2) vision features can assist models to better generate entities of captions when the entity information is sufficient in the input textual context of the given article; 3) Regions of specific objects in images contribute to the generation of related entities in captions.

## introduction image captioning is a multi-modal task which has developed a lot in recent years @xcite @xcite . image captioning models can generate captions which accurately describe the generic object categories and object relations in images on image captioning datasets such as coco @xcite or flickr @xcite . however, generic image captioning datasets above often contain less details in captions such as names, places or other specific entity information which are common in captions of news images. news image captioning @xcite @xcite aims to generate more specific descriptions of images by providing rich contextual information in associated news articles. specifically, with a news image and the corresponding article given, models need to generate a caption which not only describes the whole image generally but also contains the specific information such as names or places of objects in the image. there has made significant progress with the introduction of transformer-based end-to-end captioning methods @xcite @xcite . as a multi-modality task, news image captioning models usually generate captions with both textual and vision features. note that a series of explorations @xcite @xcite have been made for other multi-modality tasks like multimodal machine translation(mmt) to resolve the impact of vision features. it is natural and important to explore the role of vision features in the news image captioning task where all specific textual information is in news articles. we come up with these questions: rq1: do vision features help or not? and rq2: how do vision features help? particularly, rq2 can be decomposed into two subquestions: rq2-1: which part in captions do vision features help to generate? and rq2-2: which part in images helps the generation? in order to answer the above questions, we conduct a series of experiments using the most successful news image captioning models in recent studies on two main news image caption datasets good-news @xcite and nytimes800k @xcite . we first evaluate models under incongruent vision features to preliminarily determine whether the model is sensitive to the vision features. then we modify the vision features and textual features respectively to explore the specific contribution of vision features to the captioning process. we cover the specific type of image regions to explore the relationship between the generation of entities and their related regions of images. following @xcite 's work, we also conduct probing tasks to find how vision features affect caption generation under insufficient input textual features. our main conclusions are: ## related work image captioning has improved significantly in recent years, and early models @xcite @xcite exploited convolutional neural network and recurrent neural network to encode images and decode captions, respectively. @xcite attended to different image patches when generating different tokens and @xcite applied the attention mechanism to regions of the corresponding object. news image captioning takes the news articleimage pairs as input and generates image captions rich in named entity information, making it a challenging task. news image captioning models generate specific entity information by applying the textual information from the associated articles. there are two main categories of news image captioning models: (1) template-based methods @xcite which first generate the intermediate templates with placeholders for entities and then extract the specific entity information from associated articles. (2) end-to-end methods @xcite @xcite which generate the whole caption directly in one step. @xcite applied pre-trained model roberta as text encoder and transformer decoder to generate the captions with byte-pairencoding(bpe) @xcite . @xcite utilized transformer architecture with extra methods like entity-guide and visual-selective layer to obtain better textual and vision features. the early template-based method ended up with relatively low performance due to generating captions in the linguistically limited context in the second step. @xcite generated captions directly with predicted template guide vectors based on transformer architecture as well and achieved competitive performance. @xcite proposed a method to select more relevant and sufficient context with multi-modal pre-trained model clip @xcite and relation extraction model to obtain better performance. news image caption task is a multi-modal task with textual and vision features as input. some of the previous models can obtain competitive performance even with single-textual input only, which also happened in other multimodal tasks like mmt(multimodal machine translation). @xcite found that models with random unrelated images can also obtain competitive results, @xcite ; @xcite conducted probing mask experiments and pointed out that vision features contribute more when the input text is insufficient. @xcite found that vision features assist mmt models through regularization training. we refer to some of these methods to analyze the contribution of vision features in news image captioning task as well. ## conclusions in this work, we design and conduct extensive experiments to explore the impact of vision features on news image captioning models. first, we determine that vision features do contribute to generating news captions. second, from our degradation experiment, we find that vision features can help models obtain a better generation of entities that appear both in textual context and captions. the low inclusion of entities in textual context will limit the improvement obtained from applying vision features. at last, we show that specific regions of images help models with the better generation of related entities in captions. however, images with regions covered can also generate the corresponding entities better. we believe it is important for future research to improve the ability of models to generate entities out of textual context and make better use of the specific information of different regions in images.