Improving Image Captioning via Predicting Structured Concepts (2023)

Abstract: Having the difficulty of solving the semantic gap between images and texts for the image captioning task, conventional studies in this area paid some attention to treating semantic concepts as a bridge between the two modalities and improved captioning performance accordingly. Although promising results on concept prediction were obtained, the aforementioned studies normally ignore the relationship among concepts, which relies on not only objects in the image, but also word dependencies in the text, so that offers a considerable potential for improving the process of generating good descriptions. In this paper, we propose a structured concept predictor (SCP) to predict concepts and their structures, then we integrate them into captioning, so that enhance the contribution of visual signals in this task via concepts and further use their relations to distinguish cross-modal semantics for better description generation. Particularly, we design weighted graph convolutional networks (W-GCN) to depict concept relations driven by word dependencies, and then learns differentiated contributions from these concepts for following decoding process. Therefore, our approach captures potential relations among concepts and discriminatively learns different concepts, so that effectively facilitates image captioning with inherited information across modalities. Extensive experiments and their results demonstrate the effectiveness of our approach as well as each proposed module in this work.

## introduction the image captioning task aims at generating a human-like description for a given image, normally requiring recognition and understanding of the content in the image, including objects, attributes, and their relationships, etc. the task is regarded as an interdisciplinary research of computer vision and natural language processing and has become a popular topic in recent years. current methods @xcite @xcite @xcite @xcite @xcite usually follow an encoder-decoder framework, using a pre-trained object detector/classifier as an encoder to mine visual information in an image, and then feeding it into an rnn- @xcite or transformer- @xcite based decoder for description prediction along with partially generated words. however, in most cases, the extracted visual information is insufficient, even with the use of powerful visual feature extractors. this shortcoming makes the decoder rely too much on partially generated words to predict the remaining words to ensure the fluency of the generated description, that is, the model relies too much on linguistic priors during decoding, and sometimes the resulted words does not related to the image at all. in short, the major challenge that the image captioning facing now is that description generation relies too much on linguistic priors and has little to do with images. figure 2 : the overall framework of our approach, including concept prediction, weighted graph convolutional networks (w-gcn) and language decoder. the last weight matrix refers to the relations between concepts. our model can be trained in an end-to-end manner. to this end, some studies @xcite add additional visual information to obtain strong visual features and increase the contribution of vision to generate captions. the other studies @xcite @xcite have noticed the importance of semantic concepts based on visual content, which can provide rich and accurate semantic understanding, accordingly helping semantic alignment and generating reliable text descriptions. thus the conventional pipeline of their studies is to firstly predict the semantic concepts, and then send the predicted semantic concepts along with the visual features into the decoder to predict description. although promising results are obtained by these studies, they still ignore the relations among predicted concepts, which would not alleviate the overfitting on linguistic priors. for example, in one possible scenario that a model predicts the semantic words "baby" and "drink" from an image, the model is much more likely to predict the sentence as "@xmath0. on the contrary, "drink" is more likely to be followed by "water" based on the linguistic prior if the model ignores the relations. another example is shown in figure 1 , the predicted concept itself makes it difficult for language generation to escape linguis-tic priors. meanwhile, it is seen that concepts have relations, which are not only shown among objects in the image, but also in word dependencies in the text. structured semantic concepts improve the performance of the model from the perspective of language generation. in this paper, we propose a structured concept predictor (scp) to improve image captioning, which not only integrates concept prediction into the end-to-end image captioning, but also predicts the structures of obtained concepts based on the word dependencies. specifically, we propose weighted graph convolutional networks (w-gcn), with its input graph built based on mutual information priors of all descriptions in an unsupervised manner. the mutual information priors are the probability of co-occurrence of two words within a certain distance in the same description, making the language generation no longer limited to local contextual information and measuring the relationship between two concept words well. meanwhile, applying attention over the graph makes the concept features discriminatively learned, thereby reducing linguistic priors in captioning. thus, we structuralize the semantic concepts and integrate them into the end-to-end image captioning, which keep the description generation associated with image at all generation steps and reduce over-reliance on lin-guistic priors. experiments on the widely used ms coco benchmark demonstrate the superiority of our approach over strong baselines, with qualitative analyses confirming its ability in better capturing structural relationship among semantic concepts, so that offers good interpretability for our method. ## the approach in this section, we describe our proposed scp, which extracts concepts from a given image and utilizes graph convolutional networks to build topological structures between concepts, so that the structured concepts can help generate descriptive text. figure 2 gives an overview of scp, which consists of three main components. ## related work image captioning is a practical task that has been widely concerned and studied. there have been many studies before @xcite @xcite @xcite @xcite . inspired by the transformer structure in nlp, the transformer-based encoder-decoder structure has recently become mainstream, where the interaction between multi-modal information can be enhanced. @xcite integrates the spatial relationships between objects through geometric attention based on the transformer structure. @xcite use a set of learnable object queries to extract semantics from multi-scale features, and feed them into a transformer to generate captions. the previous researches have shown that exploring semantic concepts and their relations make con-tributions to high-quality descriptions. @xcite encode image into abstract scene graphs, which represent semantics and structure at a finegrained level, thus generating detailed descriptions. @xcite not only construct a multimodal relational graph for images, but also encodes relational graphs for all sentences in the dataset to fully capture language features. then a cascaded gan is developed to achieve cross-domain alignment of image and text pairs, which are fed into the decoder. @xcite explore the semantics available in captions and construct a captionguided visual relationship graph, and leverage it to enhance image representation and caption generation. the concept token network is introduced to predict concept words based on visual information. these concepts contain rich semantic information, which benefits the image captioning task @xcite . @xcite try to capture the hierarchical semantic structure in the text space, helps visual features carry semantics and generates finer-grained and more reasonable phrases and collocations. to effectively extract contextual information, constructing a graph is proposed to model the relations between words @xcite . ## conclusion in this paper, we explore the significance of concepts and their structures for accurately describing images in the image captioning task. to predict concepts and their structures, we propose a structured concept predictor (scp), so that enhance the contribution of visual signals and further use their relations to distinguish cross-modal semantics for better description generation. particularly, we design weighted graph convolutional networks (w-gcn) to depict concept relations driven by word dependencies, and then learns differentiated contributions from these concepts for following decoding process. therefore, our approach captures potential relations among concepts and discriminatively learns different concepts, so that effectively facilitates image captioning with inherited information across modalities. extensive experiments indicate that our approach achieves competing performances on ms coco benchmark. qualitative analyses confirm its ability in better capturing structural relationship among semantic concepts, so that offers good interpretability for the proposed model.