Improving Compositional Generalization in Semantic Parsing (2020)

Abstract: Generalization of models to out-of-distribution (OOD) data has captured tremendous attention recently. Specifically, compositional generalization, i.e., whether a model generalizes to new structures built of components observed during training, has sparked substantial interest. In this work, we investigate compositional generalization in semantic parsing, a natural test-bed for compositional generalization, as output programs are constructed from sub-components. We analyze a wide variety of models and propose multiple extensions to the attention module of the semantic parser, aiming to improve compositional generalization. We find that the following factors improve compositional generalization: (a) using contextual representations, such as ELMo and BERT, (b) informing the decoder what input tokens have previously been attended to, (c) training the decoder attention to agree with pre-computed token alignments, and (d) downsampling examples corresponding to frequent program templates. While we substantially reduce the gap between in-distribution and OOD generalization, performance on OOD compositions is still substantially lower.

## introduction neural models trained on large datasets have recently shown great performance on data sampled from the training distribution. however, generalization to out-of-distribution (ood) scenarios has been dramatically lower @xcite @xcite . a particularly interesting case of ood generalization is compositional generalization, the ability to systematically generalize to test examples composed of components seen during training. for example, we expect a model that observed the questions "what is the capital of france?" and "what is the population of spain?" at training time to generalize to questions such as "what is the population of the capital of spain?". while humans generalize systematically to such compositions @xcite , models often fail to capture the structure underlying the problem, and thus miserably fail @xcite @xcite @xcite . semantic parsing, mapping natural language utterances to structured programs, is a task where compositional generalization is expected, as substructures in the input utterance and output program often align. for example, in "what is the capital of the largest us state?", the span "largest us state" might correspond to an argmax clause in the output program. nevertheless, prior work @xcite @xcite has shown that data splits that require generalizing to new program templates result in drastic loss of performance. however, past work did not investigate how different modeling choices interact with compositional generalization. in this paper, we thoroughly analyze the impact of different modeling choices on compositional generalization in 5 semantic parsing datasets-four that are text-to-sql datasets, and drop, a dataset for executing programs over text paragraphs. following @xcite , we examine performance on a compositional split, where target programs are partitioned into "program templates", and templates appearing at test time are unobserved at training time. we examine the effect of standard practices, such as contextualized representations ( §3.1) and grammar-based decoding ( §3.2). moreover, we propose novel extensions to decoder attention ( §3.3), the component responsible for aligning sub-structures in the question and program: (a) supervising attention based on precomputed token alignments, (b) attending over constituency spans, and (c) encouraging the decoder attention to cover the entire input utterance. lastly, we also propose downsampling examples from frequent templates to reduce dataset bias ( §3.4). our main findings are that (i) contextualized representations, (ii) supervising the decoder attention, (iii) informing the decoder on coverage of the input by the attention mechanism, and (iv) downsampling frequent program templates, all reduce the gap in generalization when comparing standard iid splits to compositional splits. for sql, the gap in exact match accuracy between in-distribution and ood is reduced from 84.6 → 62.2 and for drop from 96.4 → 77.1. while this is a substantial improvement, the gap between in-distribution and ood generalization is still significant. all our code and data are publicly available at http://github.com/ inbaroren/improving-compgen-in-semparse . ## compositional generalization natural language is compositional in a sense that complex utterances are interpreted by understanding the structure of the utterance and the meaning of its parts @xcite . for example, the meaning of "a person below the tree" can be composed from the meaning of "a person", "below" and "tree". by virtue of compositionality, an agent can derive the meaning of new utterances, even at first encounter. thus, we expect our systems to model this compositional nature of language and generalize to new utterances, generated from subparts observed during training but composed in novel ways. this sort of model generalization is often called compositional generalization. recent work has proposed various benchmarks to measure different aspects of compositional generalization, showing that current models struggle in this setup. lake and baroni (2018) introduce a benchmark called scan for mapping a command to actions in a synthetic language, and proposed a data split that requires generalizing to commands that map to a longer sequence of actions than observed during training. @xcite study the impact of modularity in neural models on the ability to answer visual questions about pairs of objects that were not observed during training. @xcite assess the ability of models trained on clevr @xcite to interpret new referring expressions composed of parts observed at training time. @xcite develop a benchmark of freebase questions and pro- program question iid split program split select distinct river.length from river where rive.@xmath0.name from state where state.@xmath1.area) from state) give me the largest state train test what state has the largest area? test test figure 1: an iid split of examples in semantic parsing leads to identical anonymized programs appearing at both training and test time. a program split prohibits anonymized programs from appearing in the same partition, and hence tests compositional generalization. in this work, we focus on a specific kind of compositional data split, proposed by @xcite , that despite its simplicity leads to large drops in performance. @xcite propose to split semantic parsing data such that a model cannot memorize a mapping from question templates to programs. to achieve this, they take question-program pairs, and anonymize the entities in the question-program pair with typed variables. thus, questions that require the same abstract reasoning structure now get mapped to the same anonymized program, referred to as program template. for example, in the top two rows of figure 1 , after anonymizing the name of a river to the typed variable river name0, two lexicallydifferent questions map to the same program template. similarly, in the bottom two rows we see two different questions that map to the same program even before anonymization. the data is then split in a manner such that a program template and all its accompanying questions belong to the same set, called the program split. this ensures that all test-set program templates are unobserved during training. for example, in a iid split of the data, it is possible that the question "what is the capital of france?" will appear in the training set, and the question "name spain's capital." will appear in the test set. thus, the model only needs to memorize a mapping from question templates to program templates. however, in the program split, each program template is in either the training set or test set, and thus a model must generalize at test time to new combinations of predicates and entities (see figure 1 ## datasets we create iid and program splits for five datasets according to the procedure of finegan-dollak et al. (2018) as described in §2: foot_1 four text-to-sql datasets from @xcite and one dataset for mapping questions to qdmr programs @xcite in drop @xcite . similar to prior work @xcite , we train and test models on anonymized programs, that is, entities are replaced with typed variables ( §2). ## experiments we now present our empirical evaluation of compositional generalization. ## conclusion we presented a comprehensive evaluation of compositional generalization in semantic parsers by analyzing the performance of a wide variety of models across 5 different datasets. we experimented with well-known extensions to sequenceto-sequence models and also proposed novel extensions to the decoder's attention mechanism. moreover, we proposed reducing dataset bias towards a heavily skewed program template distribution by downsampling examples from frequent templates. we find that our proposed techniques improve generalization to ood examples. however, the generalization gap between in-distribution and ood data remains high. this suggests that future research in semantic parsing should consider more drastic changes to the prevailing encoder-decoder approach to address compositional generalization.