Measuring Alignment Bias in Neural Seq2seq Semantic Parsers (2022)

Abstract: Prior to deep learning the semantic parsing community has been interested in understanding and modeling the range of possible word alignments between natural language sentences and their corresponding meaning representations. Sequence-to-sequence models changed the research landscape suggesting that we no longer need to worry about alignments since they can be learned automatically by means of an attention mechanism. More recently, researchers have started to question such premise. In this work we investigate whether seq2seq models can handle both simple and complex alignments. To answer this question we augment the popular Geo semantic parsing dataset with alignment annotations and create Geo-Aligned. We then study the performance of standard seq2seq models on the examples that can be aligned monotonically versus examples that require more complex alignments. Our empirical study shows that performance is significantly better over monotonic alignments.

## introduction in semantic parsing, the goal is to map natural language (nl) sentences into machine-readable meaning representations (mr) which allow for automated reasoning. for example, consider the following pair: nl : what is the population of georgia ? mr : answer (population (state (georgia) ) ) prior to deep learning models, a popular approach was to learn a grammar-based parser that explicitly models alignments between the nl and mr sequences (wong and mooney, 2006; zettlemoyer and collins, 2005, 2007; lu et al., 2008; 1 the code and data is publicly available at https:// github.com/interact-erc/geo-aligned @xcite . the emergence of sequence-to-sequence (seq2seq) semantic parsers with attention mechanisms changed the research landscape: one of the initial premises of seq2seq models is that alignments no longer need to be explicitly modeled because the attention mechanisms will automatically learn them @xcite . more recently, researchers started to question such premise, having observed that seq2seq models fail to make proper generalizations on out-of-distribution test sets on which traditional grammar-based models excel @xcite @xcite . in this paper we follow this line of research and ask the questions: can standard seq2seq models handle arbitrary alignments? and if not, what kind of alignment bias do they have? to answer these questions, we augment the geo semantic parsing benchmark @xcite with alignment annotations and create geo-aligned. we then compare the performance of seq2seq models on examples that can be easily aligned with simple monotonic alignments to the performance of these models on examples that require word reordering. our empirical study shows that seq2seq parsers perform significantly better over examples that can be monotonically aligned. in other words, the flexibility of not having to explicitly model alignments comes at a cost: seq2seq models have difficulties in learning complex alignments. the main contributions of this paper are: ## the geo-aligned benchmark in this section we describe the geo-aligned dataset, an augmentation of the popular geo semantic parsing benchmark first introduced by @xcite . we start by providing a brief formal definition of word alignments following standard notation from the statistical machine translation literature, and we define monotonic and non-monotonic alignments @xcite . we then detail how we augment the geo dataset and provide statistics that measure the complexity of the dataset. ## related work several grammar formalisms have been proposed for semantic parsing, including categorical grammars @xcite @xcite @xcite @xcite and synchronous context free grammars @xcite . both approaches model alignments explicitly and they are induced from data. there have also been attempts to derive a more general formalism to unify the different grammar based approaches to semantic parsing @xcite . more recently, neural seq2seq models were proposed for semantic parsing in @xcite ; @xcite ; @xcite . the seq2seq approach aims to relax the reliance upon high-quality lexicons, i.e. domain-specific word alignments. most seq2seq systems implement an attention mechanism such as those proposed by @xcite ; @xcite ; @xcite , which can be seen as a strategy to learn soft alignments @xcite . recently there has been an interest in testing the generalization abilities of neural semantic parsers, which resulted in the creation of several new benchmarks @xcite @xcite @xcite on which recent work has shown improved performance by introducing more alignment bias in the models either explicitly @xcite , or implicitly @xcite . ## conclusion in this paper we introduced the geo-aligned dataset that offers an evaluation framework for testing the performance of semantic parsers over examples of varying alignment complexity. our experiments have shown that seq2seq neural parsers perform significantly better over simpler monotonic alignments, suggesting that they have an implicit bias. we hope that geo-aligned can be used by other researchers to further test alignment biases.