Align and Augment: Generative Data Augmentation for Compositional Generalization (2024)

Abstract: Recent work on semantic parsing has shown that seq2seq models find compositional generalization challenging. Several strategies have been proposed to mitigate this challenge. One such strategy is to improve compositional generalization via data augmentation techniques. In this paper we follow this line of work and propose Archer, a data-augmentation strategy that exploits alignment annotations between sentences and their corresponding meaning representations. More precisely, we use alignments to train a two step generative model that combines monotonic lexical generation with reordering. Our experiments show that Archer leads to significant improvements in compositional generalization performance.

## introduction semantic parsing is the task of mapping natural language sentences (nls) to their corresponding meaning representations (mrs). sequence-tosequence (seq2seq) transformers based on encoderdecoder architectures have become predominant for this task and have shown impressive performance @xcite @xcite . however, seq2seq models have been shown to have a limited compositional generalization ability @xcite . one natural approach to improve compositional generalization is to feed seq2seq models with additional data, increasing the set of observed patterns @xcite @xcite . the additional data is assumed to be automatically generated from the available training set using a generation strategy: this is usually referred to as data augmentation. in this paper we follow this line of research and propose archer: align and augment for compositional hard generalization. archer is a data augmentation strategy that utilizes word alignments between nl and mr pairs. in a first step, a recursive model generates monotonically aligned nl/mr pairs. in a second step, a reordering model rearranges symbols in the mrs, ensuring correct alignment with the nls. this combines the strengths of traditional recursive models, which excel at modelling sequence distributions, and seq2seq architectures, which excel at inducing arbitrary features of the input and output sequences. we evaluate archer on two multilingual datasets annotated with word alignments: geoaligned @xcite , an extension of the geo benchmark, and ati-saligned, which we introduce as part of our research, similarly extending the atis benchmark. our experiments demonstrate that archer significantly enhances the compositional generalization capabilities of seq2seq semantic parsers. in the english geo dataset's length partition, with archer data a parser accuracy almost doubles to 46%. similarly, in the query partition, performance improve from 72% to 82%. compared to alternative augmentation approaches, archer leads to higher improvements in compositional generalization, especially on the most challenging length partitions. the contributions of our work are: ## preliminaries this section introduces the preliminary background on word alignments and weighted finite state automata (wfa) necessary to understand archer. ## data augmentation with archer in this section we present our main contribution, archer: a two-step data augmentation approach that improves compositional generalization by leveraging a generative recursive sequence model over aligned bi-symbols. figure 2 provides a graphical illustration of our approach. we assume that we are given a training set t consisting of nl/mr pairs (x, y) which have been annotated with word-alignments, mapping nl words to mr symbols (described in the previous section). our objective is to create an additional training set t ′ by generating new samples (x ′ , y ′ ). we will then train a semantic parser using the original training set t augmented with the additional samples in t ′ . ideally, the generation process should create novel patterns that will improve the compositional generalization of the default semantic parser. generator. we start by reordering aligned nl/mr pairs to enforce a monotonic alignment between the nls and mrs tokens. by applying this transformation to all training pairs in t , we obtain a dataset t monotonic consisting of sequences of bi-symbols and we use it to train a generative model of the bi-symbol distribution. we can then sample from the learned distribution and generate new bi-symbol sequences. we choose to model this distribution using wfas for two main reasons: (1) wfas are defined as generative models and can naturally model the prefix distribution necessary for generation; we now turn our attention to some details on how we train and sample from the wfa. when training the wfa we append special @xmath0. to generate a sequence we initialize the process with @xmath1. we continue to generate new bi-symbols b i by sampling from the conditional distribution p wfa(b i |b 1:i-1 ) , where b 1:i refers to the prefix: [b 1 , b 2 , . . . , b i ]. in practice, when sampling we only consider the top-k bi-symbols with highest probability. in principle, the generation stops when the @xmath2. however, in order to bias the process to produce longer sequences we fix the conditional probability @xmath3. after generating t bi-symbols we reset @xmath4. in other words, the sample is never cut abruptly. in appendix c we look at the generation without the t constraint including experimental results. after we generate an initial set of samples we remove all duplicates and samples present in t . finally, we observed that simple filtering strategies can further improve the quality of the generated samples, this is described in more detail in 4.1. reorderer. from the generated bi-symbols we can extract an nl/mr pair by simply removing all epsilons. however, since the bi-symbol distribution was trained over a transformed dataset, i.e. t monotonic , the symbols in the mr might not be in the correct order and cannot be directly used for data augmentation. to address this problem we use the word-aligned data to train a re-orderer model which takes an unordered mr sequence and outputs it in the correct order. the re-orderer model is trained from pairs (mr, mr mono ), where mr is the original sequence and mr mono is the sequence obtained after the transformation that enforces a monotonic alignment between the mr and its corresponding nl. more specifically, for the re-orderer model we train a standard encoder-decoder mbart model. at decoding time we do not impose any constraints in the output generation. that is, we don't enforce that the output sequence has to be a permutation of the input. we don't even require that the re-ordered mr has the same length as the input mr. thus the reordering model is free to add, substitute and delete symbols of the input mr. in preliminary experiments we observed that when given this freedom, the re-orderer model could rectify some errors made by the generative bisymbol model, errors in structure that went beyond symbol re-ordering. we also experimented with a constrained decoding strategy that restricts the outputs to be permutations of the input, however no significant gain was observed (appendix b). after running the re-orderer model over the sequences sampled from the learned bi-symbol distribution we obtain the final sequences t ′ that augment the original sequences in t . both sets are then used to train the final semantic parser. ## results looking at the query partition (q), we observe that all the data augmentation techniques lead to significant improvements over the base semantic parser with the only exception being scfg. for this partition there doesn't seem to be a clear winner and the different techniques seem to perform similarly. the only exceptions being geo-en for which geca is significantly better and geo-de for which archer is significantly better. from this experiment we conclude that archer is an effective data augmentation technique that can significantly improve the compositional generalization of seq2seq models, especially in length generalization. these results show that a recursive generative model can successfully leverage aligned data and generate samples that are both diverse and accurate. section 7 further complements these conclusions by evaluating directly the correctness and diversity of the different augmentation strategies. finally, in ## analysis of generated data in this section, we analyze the quality of the data generated by three different strategies on geoaligned. we focus on evaluating the correctness and diversity of the generated samples. correctness ensures accurate representation of desired patterns, enhancing reliability. diversity aids compositional generalization, allowing the model to handle novel combinations effectively. refer to