Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2021)

Abstract: Sequence-to-sequence models excel at handling natural language variation, but have been shown to struggle with out-of-distribution compositional generalization. This has motivated new specialized architectures with stronger compositional biases, but most of these approaches have only been evaluated on synthetically-generated datasets, which are not representative of natural language variation. In this work we ask: can we develop a semantic parsing approach that handles both natural language variation and compositional generalization? To better assess this capability, we propose new train and test splits of non-synthetic datasets. We demonstrate that strong existing approaches do not perform well across a broad set of evaluations. We also propose NQG-T5, a hybrid model that combines a high-precision grammar-based approach with a pre-trained sequence-to-sequence model. It outperforms existing approaches across several compositional generalization challenges on non-synthetic data, while also being competitive with the state-of-the-art on standard evaluations. While still far from solving this problem, our study highlights the importance of diverse evaluations and the open challenge of handling both compositional generalization and natural language variation in semantic parsing.

## introduction sequence-to-sequence (seq2seq) models have been widely used in semantic parsing @xcite and excel at handling the natural language variation 1 of humangenerated queries. however, evaluations on synthetic 2 tasks such as scan (lake and baroni, 1 we use the term natural language variation in a broad sense to refer to the many different ways humans can express the same meaning in natural language, including differences in word choice and syntactic constructions. 2 we make a coarse distinction between synthetic datasets, where natural language utterances are generated by a program, specialized architectures with strong compositional bias under-explored general-purpose pre-trained models (e.g. seq2seq) predominant approaches synthetic non-synthetic natural language variation compositional generalization figure 1 : we study whether a semantic parsing approach can handle both out-of-distribution compositional generalization and natural language variation. existing approaches are commonly evaluated across only one dimension. in this paper, we ask: can we develop a semantic parsing approach that handles both natural language variation and compositional generalization? surprisingly, this question is understudied. as visualized in figure 1 , most prior work evaluates either out-of-distribution compositional generalization on synthetic datasets, or in-distribution performance on non-synthetic datasets. notably, designing approaches that can handle both compositional generalization and the natural language variation of non-synthetic datasets is difficult. for example, large pre-trained seq2seq models that perform well on in-distribution evaluations do not address most of the compositional generalization challenges proposed in scan @xcite . our research question has two important motivations. first, humans have been shown to be adept compositional learners @xcite . several authors have argued that a greater focus on compositional generalization is an important path to more human-like generalization and nlu @xcite . second, it is practically important to assess performance on non-synthetic data and out-of-distribution examples, as random train and test splits can overestimate real-world performance and miss important error cases @xcite . therefore, we are interested in approaches that do well not only on controlled synthetic challenges of compositionality or in-distribution natural utterances, but across all of the diverse set of evaluations shown in figure 2 . our contributions are two-fold. first, on the evaluation front, we show that performance on scan is not well-correlated with performance on non-synthetic tasks. in addition, strong existing approaches do not perform well across all evaluations in figure 2 . we also propose new target maximum compound divergence (tmcd) train and test splits, extending the methodology of @xcite to create challenging evaluations of compositional generalization for non-synthetic datasets. we show that tmcd splits complement existing evaluations by focusing on different aspects of the problem. second, on the modeling front, we propose nqg, a simple and general grammar-based approach that solves scan and also scales to natural utterances, obtaining high precision for non-synthetic data. in addition, we introduce and evaluate nqg-t5, a hybrid model that combines nqg with t5 @xcite , leading to improvements across several compositional generalization evaluations while also being competitive on the standard splits of @xcite and spi- train and test splits synthetic non-synthetic natural language variation compositional generalization mcd @xcite add primitive @xcite length tmcd template @xcite length random figure 2: we evaluate semantic parsing approaches across a diverse set of evaluations focused on natural language variation, compositional generalization, or both. we add tmcd splits to complement existing evaluations. ordering within each cell is arbitrary. der @xcite . our results indicate that nqg-t5 is a strong baseline for our challenge of developing approaches that perform well across a diverse set of evaluations focusing on either natural language variation, compositional generalization, or both. comparing five approaches across eight evaluations on scan and geoquery, its average rank is 1, with the rank of the best previous approach (t5) being 2.9; performance is also competitive across several evaluations on spider. while still far from affirmatively answering our research question, our study highlights the importance of a diverse set of evaluations and the open challenge of handling both compositional generalization and natural language variation. foot_0 2 background and related work ## target maximum compound divergence (tmcd) splits the existing evaluations targeting compositional generalization for non-synthetic tasks are template splits and length splits. here we propose an additional method which expands the set of available evaluations by generating data splits that maximize compound divergence over non-synthetic datasets, termed target maximum compound divergence (tmcd) splits. as we show in § 6, it results in a generalization problem with different characteristics that can be much more challenging than template splits, and contributes to the comprehensiveness of evaluation. in standard mcd splits @xcite , the notion of compounds requires that both source and target are generated by a rule-based procedure, and therefore cannot be applied to existing non-synthetic datasets where natural language utterances are collected from humans. for tmcd, we propose a new notion of compounds based only on the target representations. we leverage their known syntactic structure to define atoms and compounds. for instance, example atoms in funql are longest and river, and an example compound is longest(river). detailed definitions of atoms and compounds for each dataset we study can be found in appendix b.3. given this definition of compounds, our definition of compound divergence, d c , is the same as that of @xcite . specifically, for tmcd, we constrain atom divergence by requiring that every atom appear at least once in the training set. an atom constraint is desirable so that the model knows the possible target atoms to generate. a greedy algorithm similar to the one of keysers et al. (2020) is used to generate splits that approximately maximize compound divergence. first, we randomly split the dataset. then, we swap examples until the atom constraint is satisfied. finally, we sequentially identify example pairs that can be swapped between the train and test sets to increase compound divergence without violating the atom constraint, breaking when a swap can no longer be identified. ## proposed approach: nqg-t5 we propose nqg-t5, a hybrid semantic parser that combines a grammar-based approach with a seq2seq model. the two components are motivated by prior work focusing on compositional generalization and natural language variation, respectively, and we show in § 5 that their combination sets a strong baseline for our challenge. the grammar-based component, nqg, consists of a discriminative neural parsing model and a flexible quasi-synchronous grammar induction algorithm which can operate over arbitrary pairs of strings. like other grammar-based approaches, nqg can fail to produce an output for certain inputs. as visualized in figure 3 , in cases where nqg fails to produce an output, we return the output from t5 @xcite , a pre-trained seq2seq model. this simple combination can work well because nqg often has higher precision than t5 for cases where it produces an output, especially in out-of-distribution settings. we train nqg and t5 separately. training data for both components consists of pairs of source and target strings, referred to as x and y, respectively. ## experiments we evaluate existing approaches and the newly proposed nqg-t5 across a diverse set of evaluations to assess compositional generalization and handling of natural language variation. we aim to understand how the approaches compare to each other for each type of evaluation and in aggregate, and how the performance of a single approach may vary across different evaluation types. ## conclusions our experiments and analysis demonstrate that nqg and t5 offer different strengths. nqg generally has higher precision for out-of-distribution examples, but is limited by the syntactic constraints of the grammar formalism and by requiring exact lexical overlap with induced rules in order to provide a derivation at inference time. t5's coverage is not limited by such constraints, but precision can be significantly lower for out-of-distribution examples. with nqg-t5, we offer a simple combination of these strengths. while accuracy is still limited for out-of-distribution examples where nqg lacks coverage, we believe it sets a strong and simple baseline for future work. more broadly, our work highlights that evaluating on a diverse set of benchmarks is important, and that handling both out-of-distribution compositional generalization and natural language variation remains an open challenge for semantic parsing.