Bridge the Gap: High-level Semantic Planning for Image Captioning (2020)

Abstract: Recent image captioning models have made much progress for exploring the multi-modal interaction, such as attention mechanisms. Though these mechanisms can boost the interaction, there are still two gaps between the visual and language domains: (1) the gap between the visual features and textual semantics, (2) the gap between the disordering of visual features and the ordering of texts. To bridge the gaps we propose a high-level semantic planning (HSP) mechanism that incorporates both a semantic reconstruction and an explicit order planning. We integrate the planning mechanism to the attention based caption model and propose the High-level Semantic PLanning based Attention Network (HS-PLAN). First, an attention based reconstruction module is designed to reconstruct the visual features with high-level semantic information. Then we apply a pointer network to serialize the features and obtain the explicit order plan to guide the generation. Experiments conducted on MS COCO show that our model outperforms previous methods and achieves the state-of-the-art performance of 133.4% CIDEr-D score.

## introduction image captioning which aims to generate textual descriptions of images, is a significant task in both computer vision and natural language process. it not only requires recognizing and understanding the objects and attributes from the given image but also needs to verbalize them with natural language in proper order. previous works with neural models follow the encoder-decoder paradigm that uses convolutional neural network (cnn) to encode the input image and apply recurrent neural network (rnn) as decoder to generate the textual descriptions (as shown in figure 1(a) ) @xcite @xcite @xcite . to explore the multi-modal interaction between the visual content and textual description, some recent methods @xcite apply visual attention mechanism to model the interaction. the visual attention works by learning to selectively attend to image features extracted by the encoder when generating each word. for better interaction, a large number of works focus on boost the performance of neural models with improved attention mechanisms @xcite @xcite . however, there are still two gaps between the visual and language domains that visual attention does not address: (1) the gap between the visual features and textual semantics, (2) the gap between the disordering of visual features and the ordering of texts. for one thing, it is hard for the decoder to associate each word in the caption with the features without a high-level semantic understanding. for another thing, with the visual attention these neural models implicitly select which features to focus on at each decoding step without any explicit guidance or exterior supervision, which makes the generation process uncontrollable and inexplicable. there have been some researches focusing on alleviating both of the two problems. some of them apply semantic attention to leverage the high-level semantic information to narrow the first gap (fang et al., 2015; you et al., 2016). wu et al.(2016) explicitly represent high-level semantic concepts and incorporate them into the cnn-rnn approach. li et al. (2019) propose the entangled attention based on the transformer architecture @xcite to explore visual and semantic information simultaneously. these methods focus more on leveraging the high-level semantic information to enhance neural models but pay little attention to the relation between the detected semantic information and the extracted visual features. for the second gap, @xcite propose a controllable framework that can generate captions grounded on a sequence of image regions which are sorted by a sorting network. though achieving the controllability to some extent, the method struggles to avoid the problems of inflexibility and error propagation between the sorting and generation. to address these issues mentioned above, we propose a high-level semantic planning based attention network (hs-plan) that incorporates both a high-level semantic reconstruction and an explicit order planning as shown in figure 1 @xcite . (1) to narrow the gap between the visual features and textual semantics, an attention based reconstruction module is designed to re-represent the visual feature of each image region with the corresponding high-level concepts predicted by the object detector and attribute classifier. (2) to bridge the gap between the disordering of visual features and the ordering of textual sentences, we implement an attention based pointer network to make explicit order-plan to guide the caption generation. after the planning stage, the planned features are fed to an attention based caption model. the caption model first applies an order-sensitive encoder to encode the planned features further and learn the absolute and relative order information of features with position encoding. then a visual attention based decoder is employed to generate the textual description of the input image guided by the determined plan. we conduct experiments on a large benchmark dataset named ms coco @xcite to evaluate our proposed model. the results show that our model outperforms all the baselines and achieves the state-of-the-art performance: achieving 133.4% cider-d score with a single model and 134.8% with an ensemble of four models on "karpathy" test split. the qualitative human evaluation also demonstrates that our model can generate more fluent, faithful and coherent captions. ## related work begin with show and tell @xcite , numbers of neural-based encoder-decoder models are proposed for image captioning. they utilize cnn-rnn based frameworks by encoding images into features and then translating image features into sentences, and achieve significant improvements on captioning. recently attention mechanisms are widely used in image captioning, which provide guidance for choosing the most relevant image region when generating words of sentences @xcite @xcite @xcite . specifically, huang et al. @xcite propose an enhanced attention mechanism to determine the relevance of attention results for better multi-modal interaction. moreover, @xcite applies reinforcement learning with a self-critical reward to models for a more efficient training process. however, these methods are limited to the generation of the word in sentences from image features. it is still hard for these methods to bridge the gaps between the visual and language domains. previous captioning approaches focus on two different dimensions to alleviate the problems. some focus on a better understanding of images or presentation of image features with high-level semantic information @xcite . specifically, @xcite explicitly represent high-level semantic concepts and incorporate them into the cnn-rnn approach. @xcite leverage scene graph for more meaningful semantic representation to transfer the inductive bias from the pure language domain to the vision-language domain. @xcite propose the entangled attention based on the transformer architecture @xcite to explore visual and semantic information simultaneously. and others concentrate more on the controllability of the generation stage. @xcite propose a controllable framework that can generate captions grounded on a sequence of image regions which are sorted by a sorting network. different from it, our method is flexible that just makes an explicit order plan to guide the generation instead of generates words step-by-step depending on the control signal in a fixed order. ## methodology in this section, we devise hs-plan to model explicit high-level planning to guide the image captioning. the target of captioning model is to generate a textual sentence @xmath0, y 2 , ..., y t } of the given image i. traditional encoder-decoder models formulate the problem as a two-stage process: feature extraction and caption generation. but our model further decompose the problem into a three-stage process: the architecture of hs-plan is shown in figure 2 . after extracting visual features from a given image, our model first applies a semantic reconstruction module to integrate textual semantic information into visual features to re-represent them. then an attention-based pointer network is applied to make explicit order-plan to guide the caption generation. after the planning stage, an order-sensitive encoder is employed to further encode the features which are then used for generating textual descriptions with the decoder guided by the determined plan. ## conclusion in this paper we integrate the planning strategy to attention based neural models and propose a novel high-level semantic planning method to bridge the gap between the visual features and textual semantics. we design a high-level semantic planning based attention network (hs-plan) that incorporates both a semantic reconstruction and an explicit order planning to guide the caption generation. experiments are conducted on a large benchmark dataset mscoco and show that our model outperforms the baselines on both automatic and human evaluation. the experimental results also demonstrate that our model can generate more fluent, faithful and coherent captions.