Structural generalization in COGS : Supertagging is (almost) all you need (2023)

Abstract: In many Natural Language Processing applications, neural networks have been found to fail to generalize on out-of-distribution examples. In particular, several recent semantic parsing datasets have put forward important limitations of neural networks in cases where compositional generalization is required. In this work, we extend a neural graph-based parsing framework in several ways to alleviate this issue, notably: (1) the introduction of a supertagging step with valency constraints, expressed as an integer linear program; (2) the reduction of the graph prediction problem to the maximum matching problem; (3) the design of an incremental early-stopping training strategy to prevent overfitting. Experimentally, our approach significantly improves results on examples that require structural generalization in the COGS dataset, a known challenging benchmark for compositional generalization. Overall, these results confirm that structural constraints are important for generalization in semantic parsing.

## introduction semantic parsing aims to transform a natural language utterance into a structured representation. however, models based on neural networks have been shown to struggle on out-of-distribution utterances where compositional generalization is required, i.e., on sentences with novel combinations of elements observed separately during training @xcite @xcite . @xcite showed that neural graph-based semantic parsers are more robust to compositional generalization than sequence-to-sequence (seq2seq) models. moreover, @xcite , @xcite and petit and @xcite have shown that introducing valency and type constraints in a structured decoder improves compositional generalization capabilities. in this work, we explore a different method for compositional generalization, based on supertagging. we demonstrate that local predictions (with global consistency constraints) are sufficient for compositional generalization. contrary to @xcite and petit and @xcite , our approach can predict any semantic graph (including ones with reentrancies), and contrary to @xcite it does not require any intermediate representation of the semantic structure. moreover, our experiments highlight two fundamental features that are important to tackle compositional generalization in this setting. first, as is well known in the syntactic parsing literature, introducing a supertagging step in a parser may lead to infeasible solutions. we therefore propose an integer linear programming formulation of supertagging that ensures the existence of at least one feasible parse in the search space, via the so-called companionship principle @xcite . second, as the development dataset used to control training is in-distribution (i.e., it does not test for compositional generalization), there is a strong risk of overfitting. to this end, we propose an incremental early-stopping strategy that freezes part of the neural network during training. our contributions can be summarized as follows: ## supertagging for graph-based semantic parsing in the syntactic parsing literature, supertagging refers to assigning complex descriptions of the syntactic structure directly at the lexical level @xcite . for example, while an occurrence of the verb 'to walk' can be described in a coarse manner via its part-of-speech tag, a supertag additionally indicates that this verb appears in a clause with a subject on the left and a verbal phrase on the right, the latter also potentially requiring an object on its right, see figure 2 for an illustration in the formalism of lexicalized tree-adjoining grammars @xcite . we propose to introduce an intermediary semantic supertagging step in a graph-based semantic parser. the pipeline of section 2.2 becomes: we start with an input sentence, then predict one part-of-speech tag per word. this part-of-speech can be used as feature or to filter the set of possible supertags in the next step. the supertagging step chooses one supertag per word. finally, in the last step we merge all supertags together. this operation can be synthesised as a derivation tree where arc labels indicate gorn adresses of substitution and adjunction operations. we refer readers unfamiliar with ltag parsing to @xcite . (right) our semantic parsing pipeline. we start with an input sentence, then predict at most one concept per word, where ∅ indicates no concept. the supertagging step assigns one supertag per concept instance (i.e., excluding words tagged with ∅). finally, the argument identification step identifies arguments of predicates using the valency constraints from the supertags. ## related work compositional generalization. compositional generalization has been a recent topic of interest in semantic parsing. this is because failure to generalize is an important source of error, especially in seq2seq models @xcite @xcite . several directions have been explored in response. zheng and lapata (2021) rely on latent concept tagging in the encoder of a seq2seq model, while @xcite introduce latent fertility and re-ordering layers in their model. another research direction uses data augmentation methods to improve generalization @xcite @xcite @xcite . span-based methods have also been shown to improve compositional generalization @xcite @xcite . particularly, @xcite explicitly represent input sentences as trees and use a tree-lstm @xcite in their encoder. while this parser exhibits strong performance, this approach requires work from domain experts to define the set of operations needed to construct trees for each dataset. other line of work that seek to tackle compositional generalization issues include using pretrained models @xcite , specialized architectures @xcite @xcite and regularization @xcite . graph-based semantic parsing. graph-based methods have been popularized by syntactic dependency parsing @xcite . to reduce computational complexity, @xcite proposed a neural graph-based parser that handles each dependency as an independent classification problem. similar approaches were applied in semantic parsing, first for amr parsing @xcite . graph-based approaches have only recently been evaluated for compositional generalization. the approach proposed by petit and @xcite showed significant improvements compared to existing work on compositional splits of the geoquery dataset. however, their parser can only generate trees. @xcite and @xcite introduced approaches that can handle arbitrary graphs, a requirement to successfully parse cogs. ## experiments we use a neural network based on a bilstm @xcite and a biaffine layer for arc weights @xcite . more detail are given in appendix a. as usual in the compositional generalization literature, we ## conclusion we proposed to introduce a supertagging step in a graph-based semantic parser. we analysed complexities and proposed algorithms for each step of our novel pipeline. experimentally, our method significantly improves results for cases where compositional generalization is needed.