Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention (2021)

Abstract: We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Our approach builds on existing losses that encourage attention maps in neural sequence-to-sequence models to imitate the output of classical word alignment algorithms. Where past work has used word-level alignments, we focus on spans; borrowing ideas from phrase-based machine translation, we align subtrees in semantic parses to spans of input sentences, and encourage neural attention mechanisms to mimic these alignments. This method improves the performance of transformers, RNNs, and structured decoders on three benchmarks of compositional generalization.

## introduction semantic parsers translate natural language utterances (e.g., schedule a meeting with jean) into executable programs (e.g., createevent( @xmath0, and play a crucial role in applications such as question answering systems and conversational agents @xcite @xcite . as in many language understanding problems, a central challenge in semantic parsing is compositional generalization @xcite . consider a personal digital assistant for which developers have assembled separate collections of annotated utterances for user requests involving their calendars (e.g., schedule a meeting with jean) and their contact books (e.g., who is jean's manager? neural sequence-to-sequence models, which provide the foundation for state-of-the-art semantic parsers @xcite , tend to perform poorly at out-of-distribution generalization of this kind @xcite @xcite . methods have been proposed to bridge the generalization gap using meta-learning @xcite or specialized model architectures @xcite @xcite . these have registered impressive performance on small synthetic benchmark datasets, but it has proven difficult to effectively combine them with large-scale pre-training @xcite ) and natural data @xcite . in contrast to this extensive literature on data transformations and model architectures, the design of loss functions to encourage compositional generalization has been under-explored. this paper investigates attention supervision losses that encourage attention matrices in neural sequence models to resemble the output of word alignment algorithms @xcite ; @xcite ; @xcite ; @xcite , inter alia) as a source of inductive bias for compositional tasks. previous work has found that aligning program tokens (e.g., findmanager in fig. 1 ) to natural language tokens (manager) improves model performance @xcite @xcite @xcite . however, the token-level alignments derived from off-the-shelf aligners are often noisy, and the correspondence between natural language and program tokens is not always a many-to-one map of the kind returned by standard alignment algorithms. on the other hand, programs also have explicit hierarchical structure, which could be useful to induce better attention regularizers @xcite . here we investigate the use of span-level alignments, identifying sub-programs that should be predicted as a unit and aligning all tokens in a sub-program to a createevent createevent( @xmath1,@xmath2, @xmath3... piotr 's 2007 bangkok 1st (a) utterance-lisp expression alignments (b) utterance-sparql alignments block 1 block 2 distinct select select distinct ?x0 { directed_by { ?x1 } edited_by { ?x1 } } ?x1 { art_directed { m1 } gender { female } } token alignments are marked in . span-level alignments are marked using dashed bounding boxes (alignment to program sketch tokens are marked in ). programs in matrices are simplified for presentation. we use simplified sparql representation (furrer et al., 2020) grouping relations (e.g., directed_by and edited_by) by subjects (e.g., ?x0). supervised attention existing token-level supervised attention approaches assume access to an alignment matrix a |u|×|z| with entries a i,j , where a i,@xmath4. a |u|×|z| can be inferred using latent variable models @xcite @xcite . during training, when the decoder predicts a target token z j , supervised attention encourages the target-to-source attention distribution p att (u i |z j ) to match the prior alignment distribution p prior (u i |z j ) = a i,j k a k,j , which is normalized by the number of source tokens aligned to z j . we use a squared error loss @xcite : previous work has also used a cross entropy loss @xcite . ## experiments we evaluate span-level supervised attention on three benchmarks of compositional generalization. ## conclusion this paper demonstrated the effectiveness of spanlevel supervised attention as a simple and flexible tool for improving neural sequence models in a diverse set of architectures and tests of generalization. future work might explore applications to other prediction tasks and joint learning of alignments with sequence model parameters. adam optimizer using a batch size of 64 for 30 epochs, with separate learning rates for bert (3 × 10 -5 ) and the rest of the model parameters (0.001). we add supervised attention loss eq. ( 1 ) to the model's loss function with a tuning weight of λ ∈ {2.0, 4.0} for bert2sql and λ ∈ {1, 0, 2.0} for coarse2fine. for each training split with |c train | compositional training examples, we perform grid search and chose the λ that achieves the best dev. accuracy on compositional samples c. we use beam search (beam size of 5) for decoding. cfq we use t5-base, with a constant tuning weight of λ = 0.1 for the supervised attention loss. we train the model using an adafactor optimizer with a batch-size of 128 examples and a learning rate of 0.001 for 15 epochs (∼ 110k iterations). we warmup the learning rate using the first 1,100 iterations. target program sequences with a length longer than 300 after sentencepiece subtokenization are clipped. for efficiency, we use greedy search for decoding. atis text-to-sql we use the original implementation and hyper-parameters provided by @xcite , and apply supervised attention loss with a tuning weight validated from {0.05, 0.1, 1.0, 2.0}.