Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning (2021)

Abstract: Unsupervised image captioning is a challenging task that aims at generating captions without the supervision of image-sentence pairs, but only with images and sentences drawn from different sources and object labels detected from the images. In previous work, pseudo-captions, i.e., sentences that contain the detected object labels, were assigned to a given image. The focus of the previous work was on the alignment of input images and pseudo-captions at the sentence level. However, pseudo-captions contain many words that are irrelevant to a given image. In this work, we investigate the effect of removing mismatched words from image-sentence alignment to determine how they make this task difficult. We propose a simple gating mechanism that is trained to align image features with only the most reliable words in pseudo-captions: the detected object labels. The experimental results show that our proposed method outperforms the previous methods without introducing complex sentence-level learning objectives. Combined with the sentence-level alignment method of previous work, our method further improves its performance. These results confirm the importance of careful alignment in word-level details.

## introduction image captioning is a task to describe images in natural languages. this is a fundamental challenge with regard to automatically retrieving and summarizing the visual information in a human-readable form. recently, considerable progress has been made @xcite @xcite owing to the development of neural networks and a large number of annotated image-sentence pairs @xcite @xcite . however, these pairs are limited in their coverage of scenes foot_1 , and scaling them is difficult owing to the cost of manual annotation. unsupervised image captioning @xcite aims to describe scenes that have no corresponding image-sentence pairs, without requiring additional annotation of the pairs. the only available resources are images and sentences drawn from different sources and object labels detected from the images. although it is highly challenging, unsupervised image captioning has the potential to cover a broad range of scenes by exploiting a large number of images and sentences that are not paired by expensive manual annotation. to train a captioning model in this setting, previous work @xcite employed sentences that contained the object labels detected from given images. we refer to these sentences as pseudo-captions. however, pseudocaptions are problematic in that they are likely to contain words that are irrelevant to the given images. assume that an image contains two objects cat and girl (figure 1 ). this situation could give rise to various possible pseudo-captions, e.g., "a girl is holding a cat," "a cat is sleeping with a girl," "a girl is running with a cat." when the first sentence is the correct caption of the image, the words sleeping and running of the other sentences are irrelevant to the image. as the detected object labels provide insufficient information to judge which sentence corresponds to the image, many pseudocaptions containing such mismatched words can be produced. rnn gate a girl is holding a cat a cat is sleeping with a girl a girl is running with a cat ... regardless of the problem in pseudo-captions, previous work @xcite did not explicitly remove word-level mismatches. they tried aligning the features of images and their pseudo-captions at the sentence level. although this line of approach can potentially align the images and sentences correctly if there are sentences that exactly describe each image, it is not likely to hold for the images and sentences retrieved from different sources. to shed light on the problem of word-level spurious alignment in the previous work, we focus on removing mismatched words from image-sentence alignment. to this end, we introduce a simple gating mechanism that is trained to exclude image features when generating words other than the most reliable words in pseudo-captions: the detected objects. the experimental results show that the proposed method outperforms previous methods without introducing complex sentence-level learning objectives. combined with the sentence-level alignment method of previous work, our method further improves its performance. these results confirm the importance of careful alignment in word-level details. ## method our model comprises a sequential encoder-decoder model, a gating mechanism on the encoderdecoder model, a pseudo-label on the gating mechanism, and a decoding rule to avoid the repetition of object labels, as presented in figure 1 . ## experiments we ran the experiments under two different settings, @xcite and @xcite , for a fair comparison with each. for brevity, we refer to the settings in @xcite and @xcite as setting a and b, respectively. the difference of the settings is clarified in ## related work there has been considerable research with different settings and approaches to describe scenes that have no image-sentence pairs. novel object captioning @xcite @xcite attempted describing unseen objects in captions. they incorporated an image classifier or object detector trained on objects not included in imagesentence pairs. @xcite tested captioning models on the generation of unseen combinations of objects, and @xcite extended this to the unseen combinations of objects, attributes, and relations. in both settings, only the combinations were unseen, but each word in the combinations appeared in the training data. semisupervised approaches utilized caption retrieval models to automatically collect the corresponding captions for unannotated images to augment imagesentence pairs @xcite . the above work was evaluated on the scenes where correct descriptions partially overlapped with those in the training image-sentence pairs. however, there can be scenes with no such overlap due to the limited coverage of the currently available image-sentence pairs. taking a step further, unsupervised image captioning @xcite aims to describe scenes that have no overlap with the image-sentence pairs, without the annotation of the pairs. to test in that situation, the task does not allow to use any image-sentence pairs. the only available resources are images and sentences drawn from different sources and object labels detected from the images. @xcite first trained an encoderdecoder model that takes object labels in a sentence as its input and outputs the original sentence. after training, this model took the object labels detected from each image and outputted a sentence to pair with the image as its pseudo-caption. these pairs were then used to initialize a caption generator for the subsequent image-sentence alignment: bi-directional (image-to-sentence and sentence-toimage) feature reconstruction and gan training @xcite to ensure fluency in generated captions. in the work of @xcite , pseudo-captions were sentences that contained object labels detected from a given image. they employed metric learning and gan training to minimize the difference between images and pseudocaptions in their latent space, as well as to maximize the difference between images and sentences wherein no detected object label was included. our approach is different from them in that it focuses on removing the mismatched words of pseudo-captions to take reliable supervision only, rather than forcing the use of the entire pseudocaptions for image-sentence alignment. although the previous work additionally ensured to align detected object labels to images, they did not prevent the spurious alignment between images and words. as an eased setting of unsupervised image captioning, unpaired image captioning has also been explored @xcite @xcite . the major difference from unsupervised image captioning is that images and sentences are drawn from image-sentence pairs, rather than from different sources. that is, every image has completely matched captions in pseudo-captions, which is not the case in unsupervised image captioning. as correct captions exist for each image, previous approaches focused on matching images and sentences at the sentence level. contrary to these approaches, we focus on employing unsupervised image captioning and devising a method to remove word-level spurious alignment in the much noisier pseudo-captions. another variation of unpaired image captioning is the generation of captions in one language that has no image-sentence pairs, using paired images and captions in another language @xcite . however, this line of research is beyond the scope of our work, as it requires imagesentence pairs to be at least in one language. our gating mechanism borrowed the idea of adaptive attention @xcite . adaptive attention serves to control when generating words from image representations. although these methods assume that the control is automatically learned from image-sentence pairs, this is not the case in an unsupervised setting. our method is different from theirs in that we add heuristic pseudo-labels to train the gate when using image representations. ## conclusion we investigated the importance of removing wordlevel spurious alignment between images and pseudo-captions in the task of unsupervised image captioning. for this purpose, we introduced a simple gating mechanism trained to align image features with only the most reliable words in pseudo-captions. the experimental results showed that our proposed method outperformed the previous methods without the sentence-level learning objectives used in the previous methods. moreover, our method improved the performance further by combining with the previous methods. these results confirmed the importance of careful alignment in word-level details.