Compositional Generalization via Semantic Tagging (2021)

Abstract: Although neural sequence-to-sequence models have been successfully applied to semantic parsing, they fail at compositional generalization, i.e., they are unable to systematically generalize to unseen compositions of seen components. Motivated by traditional semantic parsing where compositionality is explicitly accounted for by symbolic grammars, we propose a new decoding framework that preserves the expressivity and generality of sequence-to-sequence models while featuring lexicon-style alignments and disentangled information processing. Specifically, we decompose decoding into two phases where an input utterance is first tagged with semantic symbols representing the meaning of individual words, and then a sequence-to-sequence model is used to predict the final meaning representation conditioning on the utterance and the predicted tag sequence. Experimental results on three semantic parsing datasets show that the proposed approach consistently improves compositional generalization across model architectures, domains, and semantic formalisms.

## introduction semantic parsing aims at mapping natural language utterances to machine-interpretable meaning representations such as executable queries or logical forms. sequence-to-sequence neural networks @xcite have emerged as a general modeling framework for semantic parsing, achieving impressive results across different domains and semantic formalisms @xcite @xcite . despite recent success, there has been mounting evidence @xcite @xcite ) that these models fail at compositional generalization, i.e, they are unable to ## related work the realization that neural sequence models perform poorly in settings requiring compositional generalization has led to several research efforts aiming to study the extent of this problem and how to handle it. for instance, recent studies have proposed benchmarks which allow to measure different aspects of compositional generalization. @xcite introduce scan, a grounded navigation task where a learner must translate natural language commands into a sequence of actions in a synthetic language. bahdanau et al. (2019) use a synthetic vqa task to evaluate whether models can reason about all possible object pairs after training only on a small subset. they show that modular structured models are best in terms of systematic generalization, while endto-end versions do not generalize as well. @xcite introduce a method to systematically construct benchmarks for evaluating compositional generalization. using freebase as an example, they create questions which maximize compound divergence (e.g., combinations of entities and relations) while guaranteeing that the atoms (aka the primitive elements used to compose these questions) remain the same between train and test sets. other work proposes data augmentation as a way of injecting a compositional inductive bias into neural sequence models. under this protocol, synthetic examples are constructed by taking real training examples and replacing (possibly discontinuous) fragments with other fragments that appear in at least one similar environment. recombination operations can be performed by applying rules @xcite or learned using a generative model (aky). @xcite follow a more traditional approach @xcite @xcite @xcite @xcite and develop a spanbased parser which predicts a tree over an input utterance, explicitly encoding how partial programs compose over spans in the input. finally, @xcite improve compositional generalization with the use of contextual representations, extensions to decoder attention, and downsampling examples from frequent templates. we decompose decoding in two stages where the input is first tagged with semantic symbols which are then subsequently used to predict the final meaning representation. these semantic tags are automatically induced from logical forms without any extra annotation and vary depending on the meaning representation at hand (e.g., λ-calculus, sql). they serve the goal of injecting inductive bias for compositional generalization rather than expressing general semantic information across languages (see abzianidze and bos 2017 for a proposal to develop a universal semantic tagset for nonexecutable semantic parsing). our framework can be applied to different sequence-to-sequence models, domains, and semantic formalisms. it does not require manual task-specific engineering @xcite and is orthogonal to data augmentation methods @xcite ; aky) and other extensions @xcite which we could also incorporate. ## model architecture our goal is to learn a semantic parser that takes as input a natural language utterance @xmath0, x 2 , ..., x n and predicts a meaning representation @xmath1, y 2 , ..., y m . we decompose the parser p(y|x) into a two-stage generation process: figure 1: we first tag natural language input x with semantic symbols (e.g., predicates) and predict tag sequence z. we generate the final semantic representation y, given x and z as input. every tag z t is a symbol in y representing the meaning of x t . therefore, the first-stage model p(z|x) is essentially a tagger that tries to predict the semantics of individual words. the second-stage model takes word sequence x and its accompanying tag sequence z as input, and generates the final semantic representation y. figure 1 shows the two-stage generation process. it is important to note that tags z are latent and must be induced from training data, i.e., pairs of natural language utterances and representations of their meaning. we discuss how the tagger is learned in section 4. ## model training our proposed approach combines a semantic tagger with a sequence-to-sequence model. the tagger learning problem is challenging since z is unobserved. in this section, we explain how the tagger and the overall model are trained. ## conclusions we presented a two-stage decoding framework, aiming to improve compositional generalization in neural semantic parsing. central to our approach is a semantic tagger which labels the input with semantic symbols representing the meaning of individual words. a neural sequence-to-sequence parsing model consider the input utterance and the predicted tag sequence to generate the final meaning representation. our framework can be combined with different neural models and semantic formalisms and demonstrates superior performance to related compositional generalization approaches @xcite . in the future, we would like to extend our approach to learning syntactic generalizations.