Improving Image Captioning with Better Use of Caption (2020)

Abstract: Image captioning is a multimodal problem that has drawn extensive attention in both the natural language processing and computer vision community. In this paper, we present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. The representation is then enhanced with neighbouring and contextual nodes with their textual and visual features. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences. We perform extensive experiments on the MSCOCO dataset, showing that the proposed framework significantly outperforms the baselines, resulting in the state-of-the-art performance under a wide range of evaluation metrics. The code of our paper has been made publicly available.

## introduction automatically generating a short description for a given image, a problem known as image captioning @xcite , has drawn extensive attention in both the natural language processing and computer vision community. inspired by the success of encoder-decoder frameworks with the attention mechanism, previous efforts on image captioning adopt variants of pre-trained convolution neural networks (cnn) as the image encoder and recurrent neural networks (rnn) with visual attention as the decoder @xcite @xcite . many previous methods translate image representation into natural language sentences without 1 https://github.com/gitsamshi/weakvrd-captioning figure 1 : visual relationship graphs from a pre-trained detection model @xcite and from the ground-truth caption (bottom). as in many other joint text-image modeling problems, it is crucial to obtain a good semantic representation in image captioning that bridges semantics in language and images. the existing approaches, however, have not yet adequately leveraged the semantics available in captions to con-struct image representation and generate captions. as shown in figure 1 , although vrd detection models present a strong capacity in predicting salient objects and the most common predicates, they often ignore predicates vital for captioning (e.g., "grab" in this example). exploring better models would still be highly desirable. a major challenge for establishing a structural connection between captions and images is that the links between predicates and the corresponding object regions are often ambiguous: within the "image-level" label (obj 1 , pred, obj 2 ) extracted from captions, there may exist multiple object regions corresponding to obj 1 and obj 2 . in this paper, we propose to use weakly supervised multi-instance learning to detect if a bag of object (region) pairs in an image contain certain predicates, e.g., predicates appearing in ground-truth captions here (or in other applications, they can be any given predicates under concerns). based on that we can construct caption-guided visual relationship graphs. once the visual relationship graphs (vrg) are built, we propose to adapt graph convolution operations @xcite to obtain representation for object nodes and predicate nodes. these nodes can be viewed as image representation units used for generation. during generation, we further incorporate visual relationships-we propose multi-task learning for jointly predicting word and tag sequences, where each word in a caption could be assigned with a tag, i.e., object, predicate, or none, which takes as input the graph node features from the above visual relationship graphs. the motivation for predicting a tag in each step is to regularize which types of information should be taken into more consideration for generating words: predicate nodes features, object nodes features, or the current state of language decoder. we study different types of multi-task blocks in our models. as a result, our models consist of three major components: constructing caption-guided visual relationship graphs (cgvrg) with weaklysupervised multi-instance learning, building context-aware cgvrg, and performing multi-task generation to regularize the network to take into account explicit predicate object/predicate constraints. we perform extensive experiments on the mscoco @xcite image captioning dataset with both supervised and reinforcement learning strategy @xcite . the experiment results show that the proposed models significantly outperform the baselines and achieve the state-of-the-art performance under a wide range of evaluation metrics. the main contributions of our work are summarized as follows: image captioning a prevalent paradigm of existing image captioning methods is based on the encoder-decoder framework which often utilizes a cnn-plus-rnn architecture for image encoding and text generation @xcite @xcite . soft or hard visual attention mechanism @xcite has been incorporated to focus on the most relevant regions in each generation step. furthermore, adaptive attention @xcite has been developed to decide whether to rely on visual features or language model states in each decoding step. recently, bottom-up attention techniques @xcite have also been proposed to find the most relevant regions based on bounding boxes. there has been increasing work focusing on filling the gap between image representation and caption generation. semantic concepts and attributes detected from images have been demonstrated to be effective in boosting image captioning when used in the encoder-decoder frameworks @xcite @xcite . visual relationship @xcite and scene graphs @xcite have been further employed for image encoder in a unimodal @xcite or multi-modal @xcite manner to improve the overall performance via the graph convolutional mechanism @xcite . besides, @xcite proposes a relationship-based captioning task to lead better understanding of images based on relationship. as discussed in introduction, we will further explore the relational semantics available in captions for both constructing image representation and generating caption. visual relationship detection visual relations between objects in an image have attracted more studies recently. conventional visual relation detection have dealt with subject-predicate-object triples, including spatial relation and other semantic relation. @xcite detect the triples by performing subject, object, and predicate classification separately. @xcite attempt to encode more distinguishable visual features for visual relationships detection. probabilistic output of object detection @xcite is also considered to reason about the visual relationships. ## the models given an image i, the goal of image captioning is to generate a visually grounded natural language sentence. we learn our model by minimizing the cross-entropy loss with regard to the ground truth caption s * = {w * 1 , w * 2 , ..., w * t }: the model is further tuned with a reinforcement learning (rl) objective @xcite to maximize the reward of the generated sentence s: an overview of our image captioning framework is depicted in figure 2 , with the detail of the components described in the following sections. ## conclusions this paper presents a novel image captioning architecture that constructs caption-guided visual relationship graphs to introduce beneficial inductive bias to better utilize captions. the representation is further enhanced with text and visual features of neighbouring nodes. during generation, the network is regularized to take into account explicit object/predicate constraints with multi-task learning. extensive experiments are performed on the mscoco dataset, showing that the proposed framework significantly outperforms the baselines, resulting in the state-of-the-art performance under various evaluation metrics. in the near future we plan to extend the proposed approach to several other language-vision modeling tasks.