Combine to Describe: Evaluating Compositional Generalization in Image Captioning (2022)

Abstract: Compositionality – the ability to combine simpler concepts to understand & generate arbitrarily more complex conceptual structures – has long been thought to be the cornerstone of human language capacity. With the recent, notable success of neural models in various NLP tasks, attention has now naturally turned to the compositional capacity of these models. In this paper, we study the compositional generalization properties of image captioning models. We perform a set experiments under controlled conditions using model and data ablations, each designed to benchmark a particular facet of compositional generalization: systematicity is the ability of a model to create novel combinations of concepts out of those observed during training, productivity is here operationalised as the capacity of a model to extend its predictions beyond the length distribution it has observed during training, and substitutivity is concerned with the robustness of the model against synonym substitutions. While previous work has focused primarily on systematicity, here we provide a more in-depth analysis of the strengths and weaknesses of state of the art captioning models. Our findings demonstrate that the models we study here do not compositionally generalize in terms of systematicity and productivity, however, they are robust to some degree to synonym substitutions

## introduction deep neural networks have undoubtedly become the standard option for many natural language processing (nlp) tasks with tangible results across a variety of tasks @xcite @xcite @xcite . despite their success, neural networks are regularly criticized from a growing body of research for their limited capacity to generalize beyond the distribution of the data on which they 1 code & data are available here. previous work studying compositionality has primarily focused on artificially created datasets, where compositional rules can be isolated from other natural language phenomena @xcite . in this setting, the majority of prior work has largely focused on systematicity under the prism of a downstream task. while these approaches have provided valuable insights, compositionality is multifaceted and a single test can yield misleading findings regarding compositional generalization. in this paper, we explore compositionality from the perspective of image captioning as a grounded natural language task and propose an evaluation framework with multiple dimensions of compositionality for captioning models. in particular, we adapt the independent and task-agnostic compositionality tests from @xcite to the task of image captioning using data and model ablations. each test is designed to quantify the behavior of a model along a specific dimension of compositionality. in particular, we evaluate three facets of compositionality: (1) systematicity @xcite : the ability to generalize to unseen combinations of concepts learned in isolation during training; (2) productivity: the capacity to extend predictions beyond the observations; and (3) substitutivity: the robustness of predictions under synonym substitution. previous approaches investigating compositionality in image captioning have focused primarily on systematicity @xcite @xcite . our work thus constitutes a more in-depth analysis on the compositional capabilities of captioning models. our findings regarding systematicity indicate that the standard fine-tuning approach using reinforcement learning provides gains in word-overlap metrics but hinders systematic generalization. in productivity, we demonstrate that models struggle to extend the length of their prediction beyond the training distribution. finally, with substitutivity, we demonstrate that state of the art captioning models we study here are robust against substitutions of fine-grained with more high-level synonyms. ## related work in mathematical logic, the principle of compositionality declares that the meaning of an expression can be derived from the meanings of its constituent expressions @xcite . from the perspective of natural language, if all lexical/word meaning is abstracted out from a sentence, then what remains are the rules of composition. implications of the principle influence research to this day with longstanding debates regarding compositional properties of vector space and neural models. compositionality in neural language processing initial approaches on distributional, vectorspace semantics use tensors as word and phrase meaning representations, and has studied various tensor operations for composition @xcite @xcite @xcite . in all this work, the compositional operations are fixed in advance based on some linguistic theory. in contrast, neural models learn to encode meaning: compositional operations are neither fixed during processing, nor given in advance. to encourage compositionality of neural models, prior work clusters around data augmentation @xcite , loss functions that encode different inductive biases @xcite , as well as meta-learning @xcite . benchmarking compositionality compositionality is often measured as systematic generalization in different tasks including: in navigation environments @xcite , where the objective is to translate commands into sequences of actions; or in question-answering, @xcite @xcite , where to answer a question the model needs to infer underlying relationships between entities. additional benchmarks include evaluating arithmetic expressions @xcite , and logical entailment @xcite . compositionality in visually grounded natural language compositionality has also been studied from the perspective of visually grounded natural language. previous work on visual question answering (vqa) measures generalization to novel question-answer pairs on natural @xcite , and synthetic datasets @xcite . similarly, @xcite , proposes an evaluation framework that accounts for a model's systematic generalization capacity coupled with task performance in the context of visual guessing games. more closely related to this paper, prior work has examined compositionality for image captioning @xcite @xcite . however, the aforementioned works mainly focus on systematicity alone and thus provide valuable, but limited insights on compositional properties. some prior work has studied compositionality along different prisms. @xcite examines compositionality under multiple dimensions extending the work of @xcite by grounding language to grid world environments. @xcite provides a multifaceted view on compositional properties of neural models under a set of task-agnostic tests instantiated with an artificial translation task. the distilled conclusion of this work is that the performance on a single downstream task is not a representative indicator of compositional awareness, even if this task is designed to be highly compositional. this paper can be viewed as an extension of the latter line of work, where we adapt the more fine-grained compositionality tests to the visually grounded image captioning task. ## testing compositionality in image captioning in this section, we describe the proposed tests for evaluating compositionality in image captioning. figure 1 illustrates examples from each test. we adopt a subset of task-agnostic tests proposed by @xcite . our suite consists of three tests: systematicity, productivity, and substitutivity. training data a white cat laying on top of a car. a black dog inside a brightly lit room. a black puppy laying on a couch. systematicity a white dog is next to a yellow fire hydrant. productivity a dog wearing goggles and a red collar in the backseat of a car. substitutivity a puppy dog is standing on the seat of a motorcycle. in the systematicity test, we evaluate the ability to combine concepts (white cat, black dog) to form novel output (white dog). with productivity we focus on the conditions where a model can produce output extending beyond the observable samples. in the substitutivity test, we investigate if models are robust to synonym substitutions. note that the training data is different across each compositional test. for each test we define custom training and evaluation splits of the mscoco dataset @xcite . appendix a contains additional details concerning dataset splits, and we will release this data in the public domain. ## experiments and results model we use m 2 -transformer @xcite , and adopt the configuration with the best reported results. following standard protocol @xcite @xcite , the training scheme in all experiments consists of two phases: cross-entropy (xe) and cider optimization. for xe, we apply teacher forcing where the model is trained to predict the next token given the previous ground truth tokens. we adopt self critical sequence training (scst) @xcite , as the reinforcement learning paradigm for cider optimization. the reward function is the cider score obtained with sampled sentences using beam search. for both phases, we applied the same training hyperparameters as in @xcite . the training phases are performed sequentially. we start by optimizing xe and then fine-tune the model using scst. we used early stopping to terminate a training phase, whenever the cider score on the validation set did not improve for 5 consecutive epochs. evaluation we evaluate compositional generalization using standard metrics in image captioning: bleu (b1, b4, @xcite , me-teor (m, denkowski and lavie, 2014), rouge (r, lin, 2004), and cider (c, @xcite we also quantify semantic similarity using the multi-reference bertscore @xcite . for the case of systematicity, we follow previous approaches @xcite , and additionally report recall@k of the pair of interest over the k generated captions using beam search (@xmath0. . . 5). ## qualitative analysis for each proposed test we randomly sampled 100 examples from the derived splits and inspected the generated captions. in this section, we report the main findings based on that pool. additional material is provided in the appendix c. ## conclusion we presented a series of tests for compositionality in image captioning. this work contributes towards what it means for a captioning model to be 'compositional', and what properties we would like them to have. we performed data and model ablations to identify limitations of state of the art models across three dimensions of compositionality. our findings in the systematicity align with the findings from previous works. we find that transformer-based captioning models rarely make systematic generalizations. however, as shown by the experiments in productivity, this is also partially due to the model not producing adjectives and nouns. we demonstrated that the well-established cider fine-tuning coupled with beam search decoding actually exacerbates the already poor performance on systematicity. in productivity, we found out that models struggle to extend their predictions to match the length of the ground truth captions. both models trained using xe and scst generated less number of adjectives, nouns, and verbs compared to the ground truth captions. on average we observed that models after xe optimization provide captions with more adjectives and verbs, while models incorporating scst generate descriptions with more nouns. we further included a set of experiments concerning the visual density of the image with similar results. the substitutivity experiment showcased that it is easy for the models to substitute a fine-grained with abstract descriptions of concepts. in most cases, the part of the caption following the synonym word was identical to the part of the caption after the substitution with its synonym. a natural extension to the substitutivity experiment would include the performance after substituting more abstract with fine-grained descriptions. with our framework we provided insights regarding the evaluation and training of image captioning models. word overlap metrics favor models that generate sequences closer to the target rather than more 'grounded' models that focus on actual properties of the objects in the image. this calls for a training regime that mitigates this issue by introducing multimodal metrics that take both text and vision into account (e.g., clipscore @xcite ). additionally, different training strategies should be adopted to allow the model to explore the search space, and learn to generate sequences that go beyond the average sequence length.