Retrieval, Analogy, and Composition: A framework for Compositional Generalization in Image Captioning (2021)

Abstract: Image captioning systems are expected to have the ability to combine individual concepts when describing scenes with concept combinations that are not observed during training. In spite of significant progress in image captioning with the help of the autoregressive generation framework, current approaches fail to generalize well to novel concept combinations. We propose a new framework that revolves around probing several similar image caption training instances (retrieval), performing analogical reasoning over relevant entities in retrieved prototypes (analogy), and enhancing the generation process with reasoning outcomes (composition). Our method augments the generation model by referring to the neighboring instances in the training set to produce novel concept combinations in generated captions. We perform experiments on the widely used image captioning benchmarks. The proposed models achieve substantial improvement over the compared baselines on both composition-related evaluation metrics and conventional image captioning metrics.

## introduction generating a textual description for a given image, a problem known as image captioning @xcite , requires a conditional generation model to recognize salient visual regions, e.g., object @xcite or scene graph detection @xcite , align visual features with textual tokens @xcite @xcite , and verbalize them in a natural language sentence @xcite . current state-of-theart image captioning models benefit from powerful neural autoregressive generation models, attention mechanisms, and progress in object or scene graph detection. they have achieved significant progress in obtaining visual representations for images as well as modelling alignment between visual features and textual tokens, resulting in superior per-figure 1 : comparison of compositional generalization in generated descriptions between human and machine @xcite . however, when verbalising the visual semantic concepts into natural language sentences, these models still fall short of compositional generalization for images with novel concept combinations @xcite . note that making systematic generalizations @xcite ) from limited data is an essential property of human language. as shown in figure 1 , the visual instances of "horse" and "cow" as well as the scene containing concept combinations of "cow eat" have been observed during training. while the existing models can often generate "horse on" for the picture, it would be effortless for humans to generate a caption containing "horse eat" even this combination has not been observed during training. it is partly due to the fact that current language generation models rely heavily on the surface distributional characteristics of the captions and hence are discouraged from generating unseen concept combinations @xcite . to remedy the problem, we propose to leverage prototype-based generation approaches @xcite which can explicitly expose concepts of other training examples by asking the model to decide what prototypes to retrieve in either a heuristic or learned way. in other words, these approaches have a chance to peek into retrieved prototypes for concepts without relying on the generation component. in addition, to combine the concepts from the prototypes, we enhance the conditional generation model by incorporating analogical reasoning @xcite @xcite , based on the idea that, if two things are similar on the visual side, they are probably also similar on the text side. specifically, in each generation step, we compare the visual and textual representation between the current state in the language model decoder and analogy entity pairs (a visual entity and its text form a pair) extracted from retrieved prototypes to produce sentences with improved generalization of semantic compositions. as a result, our model consists of two major components: (1) a multi-prototype retriever (c.f. section 3.3) for obtaining multiple prototypes, which aims to cover the basic concepts in the described image, and (2) an analogical reasoning editor (c.f. section 3.4) to perform analogical reasoning over extracted analogy entity pairs, in order to compose these concepts for generation. we perform extensive experiments on the widely used benchmark mscoco @xcite with both maximum likelihood estimation and reinforcement learning strategies @xcite . the experiment results show that the proposed models significantly outperform the baselines under both text-similarity based metrics and composition related metrics. the main contributions of our work are summarized as follows: ## related work image caption generation image captioning aims at generating visually grounded descriptions for images. current models often leverage a cnn or variants as the image encoder and an rnn or transformer as the decoder to generate sentences @xcite @xcite @xcite . previous work has used a visual attention mechanism @xcite @xcite @xcite @xcite , explicit high-level attributes detection @xcite @xcite to align visual and textual features. for the learning method, people use reinforcement learning methods @xcite @xcite , or contrastive or adversarial learning @xcite to generate descriptive captions @xcite ) with improved quality. the distribution shift between training and test stages also has received a lot of attention, such as generating captions with novel concepts @xcite @xcite . more recently, @xcite proposes 24 concept pairs to explicitly investigate the composition generation ability of current neural image captioning models. compositional generalization systematic compositionality, a method to capture underlying rules from limited data and generalize them to novel situations, is a key feature in human intelligence @xcite . the topic is closely related to cognitive science @xcite and connectionist literature @xcite . while the topic is widely studied in the semantic parsing literature @xcite , it is less investigated in natural language generation. @xcite introduces a resample and recombine network to improve generalization in two nlp problems, i.e., instruction following and morphological analysis. ## method our framework is designed to enhance text generation with compositional generalization through analogical reasoning from retrieved prototypes. the framework is built on the classical two-layer lstm network, i.e., updown @xcite , but this method is orthogonal to more re- figure 2 : the model framework consists of a prototype retriever and analogical reasoning editor where the former attempts to obtain multiple prototypes to the described image and the latter uses analogical reasoning to leverage the analogy entity pairs for generation. therefore, even if "white refrigerator" is a novel combination, we can generate a captioning containing it from entities in prototypes. ## conclusion we explore a prototype-based generation approach to encourage image captioning models to produce sentences with improved compositional generalization. we design a multi-prototype retriever and an analogical reasoning editor to merge the analogy entity pairs into the generation process. we demonstrate the effectiveness of the model on both composition related and quality related evaluation metrics over both common concept and rare concept splits. we perform detailed analyses on the results. in the future, we will explore this framework on the transformer based decoders.