Text Augmentation Using Dataset Reconstruction for Low-Resource Classification (2023)

Abstract: In the deployment of real-world text classification models, label scarcity is a common problem and as the number of classes increases, this problem becomes even more complex. An approach to addressing this problem is by applying text augmentation methods. One of the more prominent methods involves using the text-generation capabilities of language models. In this paper, we propose Text AUgmentation by Dataset Reconstruction (TAU-DR), a novel method of data augmentation for text classification. We conduct experiments on several multi-class datasets, showing that our approach improves the current state-of-the-art techniques for data augmentation.

## introduction the deployment of deep learning models in the real-world requires an abundance of labels. however, labeled data is often difficult and expensive to obtain, especially when the models are deployed in highly specialized domains. therefore, in this paper, we focus on data augmentation for text classification in low-resource environments. text classification @xcite is fundamental to machine learning and natural language processing. it includes various tasks, such as intent classification @xcite , which is a vital component of many automated chatbot platforms @xcite ; sentiment analysis @xcite ; topic classification @xcite ; and relation classification @xcite . the design and development of such ai applications may begin with a dataset containing only a limited amount of data. to improve the performance of downstream models in such low-resource settings, a data augmentation mechanism is often implemented (wong ## problem setup and related work in this section, we introduce the data augmentation setting in a low-resource text classification. let x @xmath0, y i )} n @xmath1, where we denote x i to be the example and y i to be its corresponding label. we assume that for each class, we have m examples where m is a relatively low number. as in previous works (e.g., @xcite , we assume the existence of a validation set x val and a test set x test . 2 our goal is to create an augmented dataset x gen by using x train so that by training a classifier on the union of the generated and the original dataset x train ∪ x gen , we improve the performance of the same classifier trained on x train . the performance of each classifier is measured on x test . the task of text augmentation is relatively challenging, since even small modifications can change the meaning and label of the text. by carefully setting up a rule-based approach, one can deal with this challenge. this was tried by @xcite , who proposed easy data augmentation (eda), which utilizes simple predefined rules to edit, remove, and substitute portions of the text while maintaining its meaning. dai and adel (2020) suggested a rule-based augmentation method named sdaner, tailored for named entity recognition. a different line of research, which is the prominent approach, uses pre-trained language models. @xcite proposed conditional bert (cbert) for contextual data augmentation. given a sentence and its label, words in the sentence are masked randomly. the label is then used as a context to predict substitute words while keeping the original sentence in the same class. anaby-tavor et al. (2020) introduced language model based data augmentation (lambada), which is also a conditional generation-based data augmentation. lambada fine-tunes an entire language model, gp t 2, by concatenating all of the sentences together with their corresponding labels, thereby creating additional textual data on which the language model can be fine-tuned. due to the noisiness of the generation process, a filtering process is used to ensure that only high-quality sentences remain. the filtering process consists of a classifier that was trained on the original dataset by taking those sentences with the top-k softmax scores. @xcite recently suggested promda. this approach first trains an entire pre-trained language model on the task of converting keywords to sentences from a general corpus. then, using rake @xcite , keywords are extracted from the original dataset. by concatenating these keywords to a learned prefix, the language model from the previous step is used to reconstruct the original sentence. then, the same filtering process as in lambada is used, with the exception that all sentences for which the original classifier agrees with the pseudo-label are taken. ## text augmentation by dataset reconstruction (tau-dr) in this section, we introduce text augmentation by dataset reconstruction (tau-dr), our novel text augmentation algorithm. algorithm 1 text augmentation by dataset reconstruction (tau-dr) require: training dataset x train , pre-trained classifier c base , pre-trained language model lm %% training phase 1: while training steps not done do 2: for (x, y) in x train do 3: extract h from c base 4: p ← m lp (h) % transform the hidden representation into soft-prompt. tau-dr consists of three stages: training, generation, and filtration, as described below. ## conclusion and future work in this paper, we present tau-dr, a novel textaugmentation method for low-resource classification using dataset reconstruction. we test our method on four multi-class classification datasets in various few-shot scenarios and show that our approach outperforms the state-of-the-art approaches. in the future, we plan to explore the learned prompt space and check how it can be used for generating helpful sentences. in our preliminary experiment, we found that the averages of the prompts were concentrated in a narrow cone. this concentration hinders the exploitation of the geometry in the learned prompt space. the above observation is aligned with other findings regarding the anisotropy of the word embedding space in pre-trained language models @xcite . finally, we wish to explore if and how additional information (e.g, in-domain textual-data) might improve the performance of text augmentation methods on highly specialized domains.