Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks (2020)

Abstract: Mixup is a latest data augmentation technique that linearly interpolates input examples and the corresponding labels. It has shown strong effectiveness in image classification by interpolating images at the pixel level. Inspired by this line of research, in this paper, we explore i) how to apply mixup to natural language processing tasks since text data can hardly be mixed in the raw format; ii) if mixup is still effective in transformer-based learning models,e.g., BERT.To achieve the goal, we incorporate mixup to transformer-based pre-trained architecture, named“mixup-transformer”, for a wide range of NLP tasks while keeping the whole end-to-end training system. We evaluate the proposed framework by running extensive experiments on the GLUEbenchmark. Furthermore, we also examine the performance of mixup-transformer in low-resource scenarios by reducing the training data with a certain ratio. Our studies show that mixup is a domain-independent data augmentation technique to pre-trained language models, resulting in significant performance improvement for transformer-based models.

## introduction deep learning has shown outstanding performance in the field of natural language processing (nlp). recently, transformer-based methods @xcite have achieved state-ofthe-art performance across a wide variety of nlp tasks 1 . however, these models highly rely on the availability of large amounts of annotated data, which is expensive and labor-intensive. to solve the data scarcity problem, data augmentation is commonly used in nlp tasks. for example, @xcite investigated language transformations like insertion, deletion and swap. several works @xcite @xcite utilized variational autoencoders (vaes) @xcite to generate more raw inputs. nevertheless, these methods often rely on some extra knowledge to guarantee the quality of new inputs, and they have to be working in a pipeline. @xcite proposed mixup, a domain-independent data augmentation technique that linearly interpolates image inputs on the pixel-based feature space. @xcite tried mixup in cnn @xcite and lstm @xcite ) for text applications. despite effectiveness, they conducted mixup only on the fixed word embedding level like @xcite did in image classification. two questions arise, therefore: (i) how to apply mixup to nlp tasks if text data cannot be mixed in the raw format? apart from the embedding feature space, what other representation spaces can be constituted and used? ii) whether or not mixup can boost the state-of-the-art further in transformer-based learning models, such as bert @xcite ). to answer these questions, we stack a mixup layer over the final hidden layer of the pre-trained transformer-based model. the resulting system can be applied to a broad of nlp tasks; in particular, it is still end-to-end trainable. we evaluate our proposed mixup-transformer on the glue benchmark, which shows that mixup can consistently improve the performance of each task. our contributions are summarized as follows: ## mixup-transformer in this section, we first introduce the mixup used in previous works. then, we show how to incorporate the mixup into transformer-based methods and how to do the fine-turning on different text classification tasks. last, we will discuss the difference between the previous works and our new approach. ## experiments to show the effectiveness of our proposed mixup-transformer, we conduct extensive experiments by adding the mixup strategy to transformer-based models on seven nlp tasks contained in the glue benchmark. furthermore, we reduce the training data with different ratios (from 10% to 90%) to see how the mixup strategy works with insufficient training data. we report the performance on development sets for all the tasks because the test time is limited by the online glue benchmark. datasets. the general language understanding evaluation (glue) benchmark @xcite is a collection of diverse natural language understanding tasks. experiments are conducted on eight tasks in glue: cola @xcite , sst-2 @xcite , mrpc (dolan and brockett, 2005), sts-b @xcite , qqp (z. chen and zhao, 2018), mnli @xcite , qnli @xcite , rte @xcite . baselines. two baselines are conducted in the experiments, including bert-base and bert-large @xcite . we evaluate the performance of our methods by adding the mixup strategy to these two baselines. ## conclusion in this paper, we propose the mixup-transformer that incorporates a data augmentation technique called mixup into transformer-based models for nlp tasks. unlike using the static mixup in previous works, our approach can dynamically construct new inputs for text classification. extensive experimental results show that mixup-transformer can be dynamically used with a pre-trained model to achieve better performance on glue benchmark.