Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering (2021)

Abstract: Answering questions that involve multi-step reasoning requires decomposing them and using the answers of intermediate steps to reach the final answer. However, state-of-the-art models in grounded question answering often do not explicitly perform decomposition, leading to difficulties in generalization to out-of-distribution examples. In this work, we propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser. Our model induces latent trees, driven by end-to-end (the answer) supervision only. We show that this inductive bias towards tree structures dramatically improves systematic generalization to out-of- distribution examples, compared to strong baselines on an arithmetic expressions benchmark as well as on C losure, a dataset that focuses on systematic generalization for grounded question answering. On this challenging dataset, our model reaches an accuracy of 96.1%, significantly higher than prior models that almost perfectly solve the task on a random, in-distribution split.

## introduction humans can effortlessly interpret new language utterances, if they are composed of previously observed primitives and structure @xcite . neural networks, conversely, do not exhibit this systematicity: although they generalize well to examples sampled from the distribution of the training set, they have been shown to struggle in generalizing to out-of-distribution (ood) examples that contain new compositions in grounded question answering @xcite and semantic parsing @xcite . consider the question in figure 1 . this question requires querying object sizes, comparing colors, identifying spatial relations, and computing intersections between sets of objects. neural models succeed when these concepts are combined in ways that were seen at training time. however, they commonly fail when concepts are combined in new ways at test time. a possible reason for this phenomenon is the expressivity of architectures such as lstms @xcite and transformers @xcite , where ''global'' representations that depend on the entire input are computed. contextualizing token representations using the entire utterance potentially lets the model avoid step-by-step reasoning, ''collapse'' reasoning steps, and rely on shortcuts @xcite @xcite . such failures are revealed when evaluating models for systematic generalization on ood examples. this stands in contrast to pre-neural log-linear models, where hierarchical representations were explicitly constructed over the input @xcite . in this work, we propose a model for visual question answering (qa) that, analogous to these classical pre-neural models, computes for every span in the input question a representation and a denotation, that is, the set of objects in the image that the span refers to (see figure 1 ). denotations for long spans are recursively computed from shorter spans using a bottom-up cky-style parser without access to the entire input, leading to an inductive bias that encourages compositional computation. because training is done from the final answer only, the model must effectively learn to induce latent trees that describe the compositional structure of the problem. we hypothesize that this explicit grounding of the meaning of sub-spans through hierarchical computation should result in better generalization to new compositions. we evaluate our approach in two setups: (a) a synthetic arithmetic expressions dataset, and figure 1 : an example from closure illustrating how our model learns a latent structure over the input, where a representation and denotation is computed for every span (for denotation we show the set of objects with @xmath0.5). for brevity, some phrases were merged to a single node of the tree. for each phrase, we show the split point and module with the highest probability, although all possible split points and module outputs are softly computed. skip(l) and skip(r) refer to taking the denotation of the left or right sub-span, respectively. to conclude, we propose a model with an inherent inductive bias for compositional computation, which leads to gains in systematic generalization, and induces latent structures that are useful for understanding its inner workings. this suggests that despite the success of general-purpose architectures built on top of contextualized representations, restricting information flow inside the network can benefit compositional generalization. our code and data can be found at https://github.com/benbogin/glt -grounded-latent-trees-qa . ## compositional generalization language is mostly compositional; humans can understand and produce a potentially infinite number of novel combinations from a finite set of components @xcite . for example, a person would know what a ''winged giraffe'' is even if she's never seen one, assuming she knows the meaning of ''winged'' and ''giraffe''. this ability, termed compositional generalization, is fundamental for building robust models that learn from limited data @xcite . neural networks have been shown to generalize well in many language understanding tasks in i.i.d splits @xcite . however, when evaluated on splits that require compositional generalization, a significant drop in performance is observed. for example, in scan (lake and baroni, 2018) and gscan @xcite , synthetically generated commands are mapped into a sequence of actions. when tested on unseen command combinations, models perform poorly. a similar trend was shown in text-to-sql parsing @xcite , where splitting examples by the template of the target sql query resulted in a dramatic performance drop. sqoop @xcite shows the same phenomena on a synthetic visual task, which tests for generalization over unseen combinations of object properties and relations. this also led to methods that construct compositional splits automatically @xcite . in this work, we focus on answering complex grounded questions over images. the clevr benchmark @xcite ) contains synthetic images and questions that require multi-step reasoning, for example, ''are there any cyan spheres made of the same material as the green sphere?''. while performance on this task is high, with an accuracy of 97%-99% @xcite , recent work @xcite introduced closure: a new set of questions with identical vocabulary but different structure than clevr, asked on the same set of images. they evaluated generalization of different models and showed that all fail on a large fraction. the most common approach for grounded qa is based on end-to-end differentiable models such as film @xcite , mac (hudson and manning, 2018), lxmert @xcite , and uniter @xcite . these models do not explicitly decompose the problem into smaller sub-tasks, and are thus prone to fail on compositional generalization. a different approach @xcite is to parse the image into a symbolic or distributed knowledge graph with objects, attributes, and relations, and then parse the question into an executable logical form, which is deterministically executed. last, neural module networks (nmns; @xcite parse the question into an executable program, where execution is learned: each program module is a neural network designed to perform an atomic task, and modules are composed to perform complex reasoning. the latter two model families construct compositional programs and have been shown to generalize better on compositional splits @xcite compared with end-to-end models. however, programs are not explicitly tied to question spans, and search over the space of programs is not differentiable, leading to difficulties in training. in this work, we learn a latent structure for the question and tie each question span to an executable module in a differentiable manner. our model balances distributed and symbolic approaches: we learn from downstream supervision only and output an inferred tree, describing how the answer was computed. we base our work on latent tree parsers @xcite @xcite that produce representations for all spans, and softly weight all possible trees. we extend these parsers to answer grounded questions, grounding sub-trees in image objects. closest to our work is gupta and lewis (2018) (gl), who have proposed to compute denotations for each span by constructing a cky chart. our model differs in several important aspects: first, while both models compute the denotations of all spans, we additionally compute dense representations for each span using a learned composition function, increasing the expressiveness of our model. second, we compute the denotations using simpler and more generic composition operators, while in gl the operators are fine-grained and type-driven. last, we work with images rather than a knowledge graph, and propose several modifications to the chart construction mechanism to overcome scalability challenges. ## model we first give a high-level overview of our proposed glt model ( §3.1), and then explain our model in detail. ## experiments in this section, we evaluate our model on both in-distribution and ood splits. ## conclusion developing models that generalize to compositions that are unobserved at training time has recently sparked substantial research interest. in this work we propose a model with a strong inductive bias towards compositional computation, which leads to large gains in systematic generalization and provides an interpretable structure that can be inspected by humans. moreover, our model also obtains high performance on real language questions (clevr-humans). in future work, we will investigate the structures revealed by our model in other grounded qa setups, and will allow the model freedom to incorporate noncompositional signals, which go hand in hand with compositional computation in natural language.