Improving Compositional Generalization with Latent Structure and Data Augmentation (2022)

Abstract: Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization. Compositional data augmentation via example recombination has transferred some prior knowledge about compositionality to such black-box neural models for several semantic parsing tasks, but this often required task-specific engineering or provided limited gains. We present a more powerful data recombination method using a model called Compositional Structure Learner (CSL). CSL is a generative model with a quasi-synchronous context-free grammar backbone, which we induce from the training data. We sample recombined examples from CSL and add them to the fine-tuning data of a pre-trained sequence-to-sequence model (T5). This procedure effectively transfers most of CSL’s compositional bias to T5 for diagnostic tasks, and results in a model even stronger than a T5-CSL ensemble on two real world compositional generalization tasks. This results in new state-of-the-art performance for these challenging semantic parsing tasks requiring generalization to both natural language variation and novel compositions of elements.

## introduction compositional generalization refers to the ability to generalize to novel combinations of previously observed atoms. 1 for example, we may ask a model to interpret the instruction "jump twice", when the atoms "jump" and "twice" were each observed separately during training but never in combination with each other @xcite . ## background and motivation in this section, we discuss the problem setting common to the compositional generalization evaluations we study, and propose some general assumptions that motivate our proposed method. problem setting consider a training dataset d consisting of input-output pairs x, @xmath0, where x is the set of valid inputs and y is the set of valid outputs. we assume that x, @xmath1, y). our model will be evaluated on inputs from a target distribution p t (x, y). we make an assumption that the conditional distribution of y given x is unchanged between source and target distributions; i.e., p s (y|x) = p t (y|x), which is also a standard assumption for domain adaptation evaluations under covariate shift. any or all of the following may be true: p s (x, y) = p t (x, y), p s (x) = p t (x), p s (y) = p t (y), and p s (x|y) = p t (x|y). what differentiates our setting from other forms of distribution shift is the added assumption that the source and target distributions share common "atoms" (see §1). in order to translate this intuitive notion of atom sharing into formal conditions, we define a general class of models termed derivational generative models, based on representing atoms as functions which can be recombined via function application. as a modeling hypothesis, we will assume that the training and evaluation distributions can be modeled by derivational generative models that share a common set of functions, but may vary in how they assign probability to derivations formed by recombining these functions. derivational generative models a derivational generative model defines a distribution p(x, y) over input-output pairs. the model contains a set of functions, g, and a distribution over derivations. a derivation z can be viewed as a tree of functions from g which derives some element @xmath2, @xmath3. 3 an example derivation is shown in figure 2 . given x , y, and g, we can generate a set z g of possible derivations. we define some shorthands for important subsets of z g for given x and y: a derivational generative model also consists of some probability distribution p θ (z) over the set of derivations z g , which we assume to be parameterized by θ. we define p g,θ (x, y) in terms of p θ (z) as: ## proposed method as shown in figure 1 , our method consists of two stages. first, we induce our generative model, csl, from training data ( §3.1). second, we sample synthetic examples from the generative model and use them to augment the training data for a sequenceto-sequence model ( §3.2). @xmath4, nt @xcite nt @xcite @xmath5, nt @xcite nt @xcite @xmath6, jump @xmath7, walk figure 3 : the example derivation of figure 2 using qcfg notation. ## experiments and analysis in this section, we comparatively evaluate and analyze our main proposed method, t5+csl-aug., which uses csl to generate examples for augmenting the training data of t5. ## conclusion we showed that the compositional structure learner (csl) generative model improves the state of the art on compositional generalization challenges for two real-world semantic parsing datasets when used to augment the task training data for the generic pre-trained t5 model. data augmentation using csl was also largely sufficient to distill csl's knowledge about compositional structures into t5 for multiple synthetic compositional generalization evaluations. while csl has limitations (notably, the qcfg formalism is not a good fit for all phenomena in the mapping of natural language to corresponding logical forms), our experiments suggest the strong potential of more powerful probabilistic models over automatically induced latent structures as data generators for black-box pretrained sequence-to-sequence models.