Compositional generalization with a broad-coverage semantic parser (2022)

Abstract: We show how the AM parser, a compositional semantic parser (Groschwitz et al., 2018) can solve compositional generalization on the COGS dataset. It is the first semantic parser that achieves high accuracy on both naturally occurring language and the synthetic COGS dataset. We discuss implications for corpus and model design for learning human-like generalization. Our results suggest that compositional generalization can be best achieved by building compositionality into semantic parsers.

## introduction a growing body of recent research investigates compositional generalization, the ability of a semantic parser to predict the meaning of unseen sentences by recombining training instances in novel ways. such generalization is thought to mimic the principle of compositionality @xcite , essential for human language learning and use. for example, cogs @xcite , a dataset based on fragments of english, contains training instances with sentences semantically annotated with up to two recursive pps; a semantic parser must then predict meaning representations for sentences with three or more recursive pps ( previous work has shown that compositional generalization on cogs is a difficult and complex task. intricate sequence-to-sequence (seq2seq) models, which achieve very high accuracy on broad-coverage semantic parsing tasks on naturally occurring language @xcite , achieve overall accuracy of 88% or less on cogs @xcite @xcite . much of this accuracy is due to lexical generalization, tasks that test for generalization to new words in known structures (sec. 2); when evaluated only on structural generalization cases that test novel structures such as the pp example above, the accuracy of most of these models drops to 10% or less. in contrast, models that achieve high accuracy on synthetic compositional generalization datasets may not be able to generalize to naturally occurring language. for instance, @xcite describe a synchronous grammar induction approach that achieves perfect accuracy on scan @xcite , but has very low accuracy on corpora of naturally occurring text such as geoquery @xcite and spider @xcite . similarly, the compositional lear parser @xcite solves cogs with near-perfect accuracy and performs very well on other synthetic datasets, but has not been evaluated on corpora of naturally occurring text. this points to a fundamental tension between broad-coverage semantic parsing on natural text and the ability to generalize compositionally from structurally limited synthetic training sets (see also @xcite . to our knowledge, the only parser that does well on both is the csl-t5 system of @xcite , which fine-tunes t5 using a complex data augmentation (da) method involving synchronous grammars. in this paper, we show that the am parser @xcite , a compositional semantic parser that achieves high accuracy across a range of different broad-coverage graphbanks @xcite , can also solve cogs at near-perfect accuracy. this high performance is due in large part to handling cases of structural generalization much better than the seq2seq models. the am parser is thus the first semantic parser shown to perform accurately both on naturally occurring language and on cogs without requiring da. given that all semantic parsers that do well on cogs are either compositional (lear, am parser) or perform compositionality-based da (csl-t5), we conjecture that building a semantic parser on the principle of compositionality is beneficial to solving compositional generalization. we discuss the challenge of structural, as opposed to lexical, generalization for future work on this task. ## compositional generalization in cogs compositional generalization is the ability to determine the meaning of unseen sentences using compositional principles. humans can understand and produce a potentially infinite number of novel linguistic expressions by dynamically recombining known elements @xcite @xcite . for semantic parsers, compositional generalization requires systems to recombine parts of multiple training instances to predict the meaning of a single test instance by learning correct generalizations. several synthetic datasets for evaluating compositional generalization now exist, notably scan @xcite and cfq @xcite . cogs @xcite ) is a synthetic semantic parsing dataset in which english sentences must be mapped to logic-based meaning representations. it distinguishes 21 generalization types, each of which requires generalizing from training instances to test instances in a particular systematic and linguistically-informed way. lexical generalization cases (18 types) test how known grammatical structures are recombined with words that were not observed in these particular structures during training. for instance, the common noun "hedgehog" is only exposed to the model as subject at training time as part of an 'exposure example' sentence, but generalization requires object usage of the same word based on forming analogies to other common nouns seen in both positions. this is illustrated in structural generalization cases (3 types) involve generalizing to linguistic structures that were not observed in training. the pp recursion example above is of this type: the cogs training set contains sentences and logic-based semantic representations with up two nested prepositional phrases. in-domain development and test sets also consist of sentences with pp nesting depth up to two, but the generalization set contains sentences with 3-12 nested pps. additional structural generalization includes cp recursion (predict deeply nested cps when trained on shallow examples, similar to pps) and "object pp to subject pp", where pps modify only objects in training (e.g. "noah ate the cake on the plate.") and only subjects at test time ("the cake on the table burned."). kim and linzen themselves show that seq2seq models based on lstms and transformers do not perform well on cogs, achieving exact-match accuracies below 35%. intensive subsequent work has tailored a wide range of seq2seq models to the cogs task @xcite @xcite @xcite , but none of these have reached an overall accuracy of 90% on the overall generalization set. on structural generalization in particular, the accuracy of all these models is below 10%, with the exception of @xcite , who achieve 39% on pp recursion. by contrast, the compositional model of @xcite and the model of @xcite , which uses compositional data augmentation, achieve accuracies upwards of 98% on the full generalization set. ## conclusion the am parser is the first compositional semantic parser to solve cogs and achieve high accuracy on naturally occurring language. foot_0 particularly on complex structural generalization cases, compositionality-based parsers seem to outperform seq2seq models systematically. by contrast, lexical generalization cases are solved easily by most models and do not require a compositionality bias. we suggest that future corpus design and evaluation focus on model accuracy for structural generalization types; an extension to cogs that incorporates a greater variety of these types would allow more insight on the overall task. though synthetic datasets like cogs allow focused probing parser performance on specific linguistic phenomena, it remains unclear exactly how accurate performance on such datasets transfers to naturally occurring language, and vice-versa. another strand of future work is thus extending the broad-coverage am parser to more compositional generalization datasets. while cogs offers a good starting point to test multiple types of both lexical and structural generalization similar to what is attested for humans, other datasets offer insight into generalization less clearly connected to human linguistic abilities (e.g. cfq; @xcite but