Interactive Key-Value Memory-augmented Attention for Image Paragraph Captioning (2020)

Abstract: Image paragraph captioning (IPC) aims to generate a fine-grained paragraph to describe the visual content of an image. Significant progress has been made by deep neural networks, in which the attention mechanism plays an essential role. However, conventional attention mechanisms tend to ignore the past alignment information, which often results in problems of repetitive captioning and incomplete captioning. In this paper, we propose an Interactive key-value Memory- augmented Attention model for image Paragraph captioning (IMAP) to keep track of the attention history (salient objects coverage information) along with the update-chain of the decoder state and therefore avoid generating repetitive or incomplete image descriptions. In addition, we employ an adaptive attention mechanism to realize adaptive alignment from image regions to caption words, where an image region can be mapped to an arbitrary number of caption words while a caption word can also attend to an arbitrary number of image regions. Extensive experiments on a benchmark dataset (i.e., Stanford) demonstrate the effectiveness of our IMAP model.

## introduction image captioning has received a significant amount of attention in recent years and is applicable in various scenarios such as virtual assistants, image indexing, and support of the disabled. significant progress has been made to generate a single sentence to describe an image @xcite . however, a single sentence has limited descriptive capacity and fails to recapitulate every detail of an image, which largely undermines applications of image captioning in real-world scenarios. one recent alternative to sentence-level captioning is image paragraph captioning with the aim of generating a coherent and fine-grained paragraph (usually 4-6 sentences) to describe an image. inspired by the successful use of the encoder-decoder framework employed in neural machine translation (nmt) @xcite , most works on image paragraph captioning employ a convolutional neural network (cnn) as an encoder to obtain fixed-length image representations, and then generates image descriptions with a long short-term memory (lstm) @xcite decoder and attention mechanisms. one representative method is to use region-based visual attention to produce a topic vector and then employ language attention to generate caption @xcite . although conventional attention-based methods have greatly enhanced the performance of image paragraph captioning, there is no mechanism to effectively keep track of the attention history in learning the dynamic alignment between the neural representations of images and that of natural languages. we argue that lacking coverage (history) information might result in two problems in conventional image paragraph captioning: (i) repetitive captioning that some image regions are unnecessarily accessed for multiple times and (ii) incomplete captioning that some image regions are mistakenly unexplored. concretely, the attention at each time step shows which image regions the model should focus on to predict the next target word in the paragraph. however, generating a target word heavily depends on imap(w/o memory) imap the image is of a large pool . there are people standing on the sidewalk in front of the fence . there are many trees in the background behind the pool . there are a few people walking around the pool . there is a large green tree behind the pool . the large pool is white and blue in color . there are several umbrellas in front of the pool . there is a large body of water behind the pool . there are people standing on the side of the fence watching the pool . there are many trees in the background . the relevant parts of the whole image, and an image region is involved in the generation of the whole paragraph. as a result, repetitive captioning and incomplete captioning inevitably happen because of ignoring the coverage of image regions of interest. figure 1 shows an example from stanford dataset. the model (i.e., imap w/o memory) which is unaware of the coverage information generates repeated sentences to describe the same region of the image ("there is a large green tree behind the pool"), while the "several umbrellas" and "a large body of water" are unexplored. in this paper, we propose an interactive key-value memory-augmented attention for image paragraph captioning (imap) to alleviate the repetitive captioning and incomplete captioning problems. our model exploits the recent success of the hierarchical lstm to generate image paragraph captions @xcite . a sentence lstm recursively generates sentence topic vectors conditioned on the image features learned by a cnn encoder, and a word rnn is subsequently adopted to decode each topic into output sentence word by word with an attention mechanism to learn the context image representation at each decoding step. different from conventional attention methods, the imap model generates the image context representation that is appropriate for predicting the next target word with iterative memory access operations conducted on a key-memory and a value-memory, inspired by the memory-augmented attention @xcite . this mechanism allows the model to track the coverage information typically for each salient object within the image, and therefore avoid repetitive and incomplete captioning. specifically, we leverage the key-value paired memories to interactively maintain the visual and language features of the input image. the key-memory keeps updated to track the interaction history between the image representation and the decoder by a writing operation, while the value-memory keeps fixed to store the original semantic image features throughout the whole decoding process. in each decoding step, the model learns to address relevant memories based on the key-memory using the "query" state learned from the previous decoder state and previous prediction, and the corresponding values in value-memory are subsequently returned as the image context representation. in addition, we employ an adaptive attention mechanism to realize adaptive alignment from image regions to caption words, where an image region can be mapped to an arbitrary number of caption words while a caption word can also attend to an arbitrary number of image regions. we summarize our main contributions as follows. (1) we propose an interactive key-value memoryaugmented attention to better keep track of attention history and image coverage information, helping the decoder to overcome the repetitive and incomplete captioning problems by automatically distinguishing which parts of the image have been described and which parts are unexplored. (2) we leverage language phrases features with interactive key-value memory-augmented attention to help the model learn better alignment between visual and language features. (3) experiments on an image paragraph captioning benchmark demonstrate that the proposed method outperforms previous state-of-the-art approaches by a substantial margin, across multiple evaluation metrics. ## related work single sentence image captioning automatic image captioning involves analyzing the visual content of an input image, and generating a textual sentence that verbalizes its most salient aspects @xcite . inspired by the success of the encoder-decoder framework in neural machine translation, most recent image captioning methods employ the sequence-to-sequence (seq2seq) model to generate image captions @xcite @xcite @xcite . for instance, an attention-based encoder-decoder neural network was proposed in @xcite , which learned to dynamically attend to different locations of the images during decoding different words in the captions. @xcite proposed a bottom-up and top-down attention mechanism to enable attention to be computed at the level of objects and salient regions. there were also several recent image captioning studies employing reinforcement learning techniques in the encoder-decoder neural networks. for instance, @xcite presented a self-critical sequence training (scst) method by considering the optimization of image captioning as a reinforcement learning problem. image paragraph captioning since a single sentence has limited descriptive capacity and fails to recapitulate every detail of an image, the task of image paragraph generation has received increasing attention recently, which describes an image with a long, descriptive, and coherent paragraph. @xcite was one of the early image paragraph captioning studies, which employed a sentence-level recurrent neural network (rnn) to generate sentence topic vectors, and then applied a word-level rnn to decode each topic vector into a sentence. subsequently, @xcite introduced a recurrent topic-transition generative adversarial network (rtt-gan) to extend the hierarchical rnn by proposing an adversarial framework between a paragraph generator and two multi-level discriminators (sentence discriminator and topic-transition discriminator). @xcite augmented the hierarchical rnn by leveraging coherence vectors to ensure cross-sentence topic smoothness and global topic vectors to summarize the overall information of the image. @xcite proposed a convolutional auto-encoding model for image paragraph captioning, which incorporated a convolutional and deconvolutional autoencoding framework for topic modeling on region-level features of an image. different from the aforementioned methods, the imap model focuses on alleviating the repetitive and incomplete captioning problems in conventional image paragraph captioning by designing an interactive key-value memory-augmented attention mechanism to track the coverage information typically for each salient object within the image. ## our methodology given an image i, image paragraph captioning aims to generate a long paragraph descriptions @xmath0, y 2 , ..., y n }, where n is the number of sentences in the paragraph y . each sentence y @xmath1, w y i 2 , ..., w y i t } consists of t words. as illustrated in figure 2 , the proposed imap model consists of three parts: (i) an image encoder, which encodes a query image and outputs a set of visual feature vectors and corresponding dense phrases (language features); (ii) a hierarchical decoder, which leverages a sentence-level lstm to generate sentence topic vectors, and then employs a word-level rnn to decode each topic vector into a sentence word by word; (iii) an interactive key-value memory-augmented attention module, which is used to keep track of the attention history and encourage the decoder to consider the unexplored salient image regions. next, we will introduce each part of our imap model in detail. ## experimental setup dataset we conduct the experiments and evaluate our imap model on the widely used stanford image paragraph dataset @xcite , which is the only open source benchmark dataset available for image paragraph captioning. stanford dataset is collected from ms coco @xcite and visual genome @xcite . this dataset consists of 14,575 images for training, 2,487 images for validation, and 2,489 images for testing, each of which has one human-annotated paragraph. implementation details for each image, faster r-cnn @xcite initialized with the vgg-16 network @xcite is applied to detect objects of the image, and top @xmath2. the size of each feature vector is 4,096, and the word embedding size to 512. we set the maximum number of sentences in each paragraph to @xmath3, the maximum length of each sentence to 30 via padding operation, and the maximum length of dense phrases to 8. the hidden sizes of both sent-lstm and two stacked word-lstms are set to 512. the number of hidden units in the attention layer is 512. we set λ w , λ s and λ a to 1.0, 5.0 and 10 -4 respectively. the maximum number of attention steps is set to 4. the vocabulary used in the experiment is the same as @xcite . we first pre-train our model with the cross-entropy loss function, and use adam optimizer @xcite with an initial learning rate 5 × 10 -4 to learn the model. after that, the self-critical training method with cider as the reward is used to further optimize the model. during this stage, the initial learning rate of adam optimizer is set to 5 × 10 -5 . ## conclusion in this paper, we proposed an effective interactive key-value memory-augmented attention to alleviate repetitive and incomplete captioning problems in image paragraph captioning, which maintains a timely updated key-memory to track attention history and a fixed value-memory to store the image features during the whole decoding process. to verify the effectiveness of the proposed model, we conducted extensive experiments on the widely used stanford dataset. the experimental results demonstrated that our model achieved impressive results compared to state-of-the-art image paragraph generation techniques.