Sequence-Level Mixed Sample Data Augmentation (2020)

Abstract: Despite their empirical success, neural networks still have difficulty capturing compositional aspects of natural language. This work proposes a simple data augmentation approach to encourage compositional behavior in neural models for sequence-to-sequence problems. Our approach, SeqMix, creates new synthetic examples by softly combining input/output sequences from the training set. We connect this approach to existing techniques such as SwitchOut and word dropout, and show that these techniques are all essentially approximating variants of a single objective. SeqMix consistently yields approximately 1.0 BLEU improvement on five different translation datasets over strong Transformer baselines. On tasks that require strong compositional generalization such as SCAN and semantic parsing, SeqMix also offers further improvements.

## introduction natural language is thought to be characterized by systematic compositionality @xcite . a computational model that is able to exploit such systematic compositionality should understand sentences by appropriately recombining subparts that have not been seen together during training. consider the following example from andreas (2020): given the above sentences, a model which has learned compositional structure should be able to generalize and understand sentences such as: in practice, neural models often overfit to long segments of text and fail to generalize compositionally. this work proposes a simple data augmentation strategy for sequence-to-sequence learning, seqmix, which creates soft synthetic examples by randomly combining parts of two sentences. this prevents models from memorizing long segments and encourages models to rely on compositions of subparts to predict the output. to motivate our approach, consider some example sentences that can be created by combining (1a) and ( 1b instead of enumerating over all possible combinations of two sentences, seqmix crafts a new example by softly mixing the two sentences via a convex combination of the original examples. this approach can be seen as a sequence-level variant of a broader family of techniques called mixed sample data augmentation (msda), which was originally proposed by @xcite and has been shown to be particularly effective for classification tasks @xcite @xcite . we also show that seqmix shares similarities with word replacement/dropout strategies in machine translation @xcite @xcite , seqmix targets a crude but simple approach to data augmentation for language applications. we apply seqmix to a variety of sequence-to-sequence tasks including neural machine translation, semantic parsing, and scan (a dataset designed to test for compositionality of data-driven models), and find that seqmix improves results on top of (and when combined with) existing data augmentation methods. ## motivation and related work while neural networks trained on large datasets have led to significant improvements across a wide range of nlp tasks, training them to generalize by learning the compositional structure of language remains a challenging open problem. notably, lake and @xcite propose an influential dataset (scan) to evaluate the systematic compositionality of neural models and find that they often fail to generalize compositionally. one approach to encouraging compositional behavior in neural models is by incorporating compositional structures such as parse trees or programs directly into a network's computational graph @xcite @xcite @xcite . while effective on certain domains such as visual question answering, these approaches usually rely on intermediate structures predicted from pipelined models, which limits their applicability in general. further, it is an open question as to whether such putatively compositional models result in significant empirical improvements on many nlp tasks @xcite . expressive parameterizations over high dimensional input afforded by neural networks contribute to their excellent performance in high resource settings; however, such flexible parameterizations can easily lead to a model's memorizing-i.e., overfitting to-long segments of text, instead of relying on the appropriate subparts of segments. another approach to encouraging compositionality in richly-parameterized neural models, then, is to augment the training data with more examples. existing work in this vein include switchout @xcite , which replaces a word in a sentence with a random word from the vocabulary, geca @xcite , which creates new examples by switching subparts that occur in similar contexts, and tmix @xcite , which interpolates between hidden states of neural models for text classification. we compare to these approaches to our proposed approach in this paper. ## method our proposed approach, seqmix, is simple, and is essentially a sequence-level variant of mixup @xcite , which has primarily been used for image classification tasks @xcite . we first describe the generative data augmentation process behind this model for text generation, and show how seqmix approximates the resulting latent variable objective with a relaxed version. let @xmath0. assume that we sample a pair of training examples (x, y ) and (x , y ) from the training set, ensuring that both have the same length (@xmath1, @xmath2. we then sample a binary combination vector @xmath3, m y ] with m @xmath4, 1} s , m @xmath5, 1} t to decide which token to use at each position. each element m i is sampled i.i.d from bernoulli(λ), where the parameter λ is itself sampled from beta(α, α), and α is hyperparameter. this gives a mixed synthetic example: the new example pair of sentences ( x, ŷ ) will not correspond to natural sentences in general, but may contain valid subparts (phrases) that bias the model towards learning the compositional structure (as in the examples discussed in the introduction). marginalizing over m gives the following log marginal likelihood, an alternative, which we refer to as seqmix, is to consider a soft variant of the original objective by training on expected samples, letting f θ (x, @xmath6, the local probability of y t geca enumerate xi:@xmath7:j if xi:j and x i :j is a @xmath8.e. co-occurs in context) to summarize, this results in a simple algorithm where we sample λ ∼ beta(α, α) and train on these expected samples. 1 relationship to existing methods ## experimental setup we test our approach against existing baselines across a variety of sequence-to-sequence tasks: machine translation, scan, and semantic parsing. for all datasets, we tune the α hyperparameter in the range of [0.1, 1.5] on the validation set. 3 exact details regarding the training setup (including descriptions of the various datasets) can be found in the supplementary materials. machine translation our machine translation experiments consider five translation datasets: (1) iwslt '14 german-english (de-en) (2) iwslt '14 english-{german, italian, spanish} (en-{de, it, es}) (3) wmt '14 english-german (en-de). we use the transformer implementation from fairseq @xcite with the default configuration. scan scan is a command execution dataset designed to test for systematic compositionality of data-driven models. scan consists of simple english commands and corresponding action sequences. we consider three different splits that have been widely utilized in the existing literature: jump, around-right, turn-left. for the splits (jump, turn-left), the primitive commands (i.e. "jump", "turn left") are only seen in isolation during training, and the test set consists commands that compose the isolated primitive command with the other commands seen during training. for the template split (around-right), training examples contain the commands "around" and "right" but never in combination. following previous work @xcite , we use a one-layer lstm encoder-decoder model with hidden size of 512 and embedding size of 64. semantic parsing for semantic parsing, we consider the sql queries subset of geoquery (finegan-dollak et al., 2018), which consists of 880 english questions paired with sql commands. the standard question split ensures no questions are repeated between the train and test sets, while the more challenging query split ensures that neither questions nor logical forms (anonymized) are repeated. following andreas (2020), we use the same model as for scan but additionally introduce a copy mechanism. ## results and analysis ## conclusion this paper presents seqmix, a simple data augmentation strategy for sequence-to-sequence applications. despite being a crude approximation to compositional phenomena in language, we found seqmix to be effective on three different sequenceto-sequence tasks, including the challenging scan dataset which is designed to test for compositional generalization. seqmix is efficient and easy to implement, and as a secondary contribution, we provide a framework that unifies several data augmentation strategies for compositionality, which naturally suggests avenue for future research (e.g., a relaxed variant of geca).