T ree M ix: Compositional Constituency-based Data Augmentation for Natural Language Understanding (2022)

Abstract: Data augmentation is an effective approach to tackle over-fitting. Many previous works have proposed different data augmentations strategies for NLP, such as noise injection, word replacement, back-translation etc. Though effective, they missed one important characteristic of language–compositionality, meaning of a complex expression is built from its sub-parts. Motivated by this, we propose a compositional data augmentation approach for natural language understanding called TreeMix. Specifically, TreeMix leverages constituency parsing tree to decompose sentences into constituent sub-structures and the Mixup data augmentation technique to recombine them to generate new sentences. Compared with previous approaches, TreeMix introduces greater diversity to the samples generated and encourages models to learn compositionality of NLP data. Extensive experiments on text classification and SCAN demonstrate that TreeMix outperforms current state-of-the-art data augmentation methods.

## introduction data augmentation (da) has won great popularity in natural language processing (nlp) @xcite due to the increasing demand for data and the expensive cost for annotation. da aims at increasing the quantity and diversity of the datasets by generating more samples based on existing ones, which helps make the training process more consistent and improves the model's capacity for generalization @xcite . for instance, existing da methods often leverage word-level manipulation @xcite @xcite and model-based sentence generation @xcite . as mixup-based @xcite augmentation achieving huge success in computer vision @xcite @xcite , some recent works start s1:they will find little interest in this poor film. s2:it comes as a touching love story. method example eda @xcite they will this find little interest in bad movie. aeda @xcite they will find ? little in ! this poor movie;. noise @xcite thes will fi little intres _ this poor film . ssmix @xcite they will find little interest in love poor film replacement @xcite they will find limited interest in this odd film. back translation @xcite they will show little interest in this strange film. ## method our work is motivated by mixup @xcite , which creates virtual samples by mixing inputs. given two random drawn examples (x i , y i ) and (x j , y j ), where x denotes the input sample and y is the corresponding one-hot label, mixup creates a new sample by: we improve upon mixup by incorporating compositionality of language, a key characteristic that is essential to generalization but neural models often fall short in capturing (lake and baroni, 2018). instead of interpolating with the whole sample, treemix, our newly proposed method, creates new sentences by removing phrases of sentences and reinserting subparts from other sentences. treemix makes use of constituency trees to decompose a sentence into meaningful constituent parts, which can then be removed and recombined to generate new augmentation samples. we aim to improve models' compositionality generalization ability by training on large amount of samples produced by treemix. an example of using treemix for single sentence classification is shown in figure 1 . ## conclusion this work introduced treemix, a compositional data augmentation approach for natural language understanding. treemix leverages constituency parsing tree to decompose sentences into substructures and further use the mixup data augmentation technique to recombine them to generate new augmented sentences. experiments on text classification and semantic parsing benchmarks demonstrate that treemix outperforms prior strong baselines, especially in low-resource settings and compositional generalization.