The Role of Syntactic Planning in Compositional Image Captioning (2021)

Abstract: Image captioning has focused on generalizing to images drawn from the same distribution as the training set, and not to the more challenging problem of generalizing to different distributions of images. Recently, Nikolaus et al. (2019) introduced a dataset to assess compositional generalization in image captioning, where models are evaluated on their ability to describe images with unseen adjective–noun and noun–verb compositions. In this work, we investigate different methods to improve compositional generalization by planning the syntactic structure of a caption. Our experiments show that jointly modeling tokens and syntactic tags enhances generalization in both RNN- and Transformer-based models, while also improving performance on standard metrics.

## introduction image captioning is a core task in multimodal nlp, where the aim is to automatically describe the content of an image in natural language. to succeed in this task, a model first needs to recognize and understand the properties of the image. then, it needs to generate well-formed sentences, requiring both a syntactic and a semantic knowledge of the language @xcite . deep learning techniques are the standard approach to tackling this problem: images are represented by visual features extracted from convolutional neural networks (e.g. @xcite , and sentences are generated by conditioning recurrent neural networks (e.g. @xcite , or transformers @xcite on the extracted visual features. while deep neural networks achieve impressive performance in a variety of applications, including image captioning, their ability to demonstrate compositionality, defined as the algebraic potential to understand and produce novel combinations from known components @xcite , has been questioned. semantic compositionality of language in neural networks has attracted interest in the community @xcite @xcite as compositionality is conjectured to be a core feature not only of language but also of human thought @xcite . in image captioning, improving compositional generalization is a fundamental step towards generalizable systems that can be employed in daily life. to this end, @xcite recently introduced a compositional generalization dataset where models need to describe images that depict unseen compositions of primitive concepts. for example, models are trained to describe images with "white" entities and all types of "dog" concepts but never the adjective-noun composition of "white dog." in their dataset, models are evaluated on their ability to caption images depicting the unseen composition of held out concepts. their study suggests that rnn-based captioning models do not compositionally generalize, and that this is primarily attributable to the language generation component. in this paper, we study the potential for syntax to improve compositional generalization in image captioning by combining syntactic planning and language generation in a single model. our study is inspired by the traditional natural language generation (nlg) framework @xcite , where nlg is split into three distinct steps: text planning, sentence planning, and linguistic realization. while state-of-the-art captioning models typically proceed directly from visual features to sentence generation, we hypothesize that a model that plans the structure of a sentence as an intermediate step will improve compositional generalization. a model with a planning step can learn the high-level structure of sentences, making it less prone to overfitting the training data. specifically, we explore three methods for integrating syntactic planning into captioning in our experiments: (a) pre-generation of syntactic tags from the image, (b) interleaved generation of syntactic tags and words @xcite , and (c) multi-task learning with a shared encoder that predicts syntactic tags or words @xcite . we do so while also empirically investigating four different levels of syntactic granularity. the main findings of our experiments are that: ## planning image captions natural language generation has traditionally been framed in terms of six basic sub-tasks: content determination, discourse planning, sentence aggregation, lexicalization, referring expression generation and linguistic realization @xcite . within this framework, a three-stage pipeline has emerged @xcite : ## experimental setup data we use training and evaluation sets such that paradigmatic gaps exist in the training set. that is, for a concept pair {c i , c j }, the validation d val and test d test sets only contain images in which at least one of the captions contains the pair of concepts, while the complementary set -where concepts c i and c j can only be observed independently -is used for training d train . following nikolaus et al. (2019) , we select the same 24 adjective-noun and verb-noun concept pairs, and split the english coco dataset @xcite into four sets, each containing six held out concept pairs. pre-processing we first lower-case and strip away punctuation from the captions. we then use stanfordnlp @xcite to tokenize and lemmatize the captions, and to extract universal pos tags and syntactic dependency relations. for iobbased chunking, we train a classifier-based tagger on conll2000 data @xcite using nltk @xcite . finally, we use the a* ccg parsing model by @xcite with elmo embeddings @xcite to extract ccg tags. visual features are extracted from 36 regions of interest in each image using bottom-up attention @xcite trained on visual genome @xcite . ## syntax awareness in ## results and discussion we now report the final performance of three image captioning models that integrate the syntactic planning (interleave+pos) with word generation. ## conclusion we investigated a variety of approaches along with the use of syntactic tag sets to achieve compositional generalization in image captioning via sentence planning. our results support the claim that combining syntactic planning and language generation consistently improves the generalization capability of rnn-and transformer-based image captioning models, especially for inanimate colornoun combinations. while this approach penalizes image-sentence ranking models, we showed that this can be overcome with an adaptive mechanism, resulting in state-of-the-art performance on the compositional generalization task. we believe our results will lead to further exploration of syntaxaware captioning models given their potential to better generalize, both in terms of under-researched syntactic granularity (e.g. ccg) and more expressive alternatives to modeling syntactic structure. another direction for future work is to focus on size-noun compositions, which rely on the successful integration of real-world knowledge.