Guiding Generative Language Models for Data Augmentation in Few-Shot Text Classification (2022)

Abstract: Data augmentation techniques are widely used for enhancing the performance of machine learning models by tackling class imbalance issues and data sparsity. State-of-the-art generative language models have been shown to provide significant gains across different NLP tasks. However, their applicability to data augmentation for text classification tasks in few-shot settings have not been fully explored, especially for specialised domains. In this paper, we leverage GPT-2 (Radford et al, 2019) for generating artificial training instances in order to improve classification performance. Our aim is to analyse the impact the selection process of seed training examples has over the quality of GPT-generated samples and consequently the classifier performance. We propose a human-in-the-loop approach for selecting seed samples. Further, we compare the approach to other seed selection strategies that exploit the characteristics of specialised domains such as human-created class hierarchical structure and the presence of noun phrases. Our results show that fine-tuning GPT-2 in a handful of label instances leads to consistent classification improvements and outperform competitive baselines. The seed selection strategies developed in this work lead to significant improvements over random seed selection for specialised domains. We show that guiding text generation through domain expert selection can lead to further improvements, which opens up interesting research avenues for combining generative models and active learning.

## introduction data sparsity and class imbalance are common problems in text classification tasks @xcite @xcite , especially when the text to be labelled is from a highly-specialised domain where only scarce domain experts can perform the labelling task @xcite @xcite . data augmentation (da) is a widely used method for tackling such issues @xcite @xcite . however, the well-established da methods in domains such as computer vision and speech recognition @xcite @xcite @xcite , relying on simple transformations of existing samples, cannot be easily transferred to textual data as they can lead to syntactic and semantic distortions to text @xcite . recent advances in text generation models, such as gpt and subsequent releases @xcite , have led to the development of new da approaches which generate additional training data from original samples, rather than perform only local changes to the text. related studies use text generation models for improving relation extraction @xcite , tackle class imbalance problems for extreme multi-label classification tasks @xcite , and augment domain-specific datasets in order to improve performance in various domain-specific classification tasks @xcite . specifically, @xcite and @xcite explore different finetuning approaches for pre-trained models for data augmentation in order to preserve class-label information. results showed the potential of generative models such as gpt-2 @xcite and bart @xcite to augment small collections of labelled data. further, an important problem with text generation techniques is the possibility of generating noise which decreases the performance of classification models rather than improving it @xcite . however, this problem is ignored in the aforementioned studies. the most similar study to ours is that of @xcite in the context of commonsense reasoning. they proposed an approach based on the use of influence functions and heuristics for selecting the most diverse and informative artificial samples from an already-generated artificial dataset. instead, we focus on the previous step of selecting the most informative samples (or seeds) from the original data. we show that a careful selection of class representative samples from the original data in the first place can already lead to improvements and has an important efficiency advantage, as it prevents an unnecessary waste of resources and time of generating unused generated documents, especially considering how resource expensive generative language models are @xcite . finally, there is no research on exploiting the use of experts knowledge for improving the performance of generative language models for specialised domains. therefore, our aim is to improve the quality of generated artificial instances used for text classification training by developing seed selection strategies to guide the generation process. specifically, we propose three da methods in order to improve fewshot text classification performance using gpt-2 -1) a human-in-the-loop method that involves a domain expert choosing class representative samples; 2) a method that leverages the expert-generated classification hierarchy of a dataset in order to improve the classification of the top hierarchy classes; 3) a method that selects the seeds with the maximum occurrence of nouns. we chose these seed selection strategies because they exploit characteristics associated with specialised domains such as high number of terms, annotation performed by experts, and hierarchical class structure (common for social science and medical domains which require thematic analysis). our contributions are summarised as follows. ## methodology we experiment with two fine-tuning techniques for gpt in order to identify optimal ways for adapting gpt-2 model for da for classification. further, our analysis focus on few-shot classification because of the demand for approaches which can perform well for only a handful of training instances especially in specialised domains where experts are sparse and data access is limited. however, our methodology can be easily extended for classification problems with more labelled data and it can also be used to generate more artificial training data. ## experimental setting in the following we describe our few-shot text classification experimental setting. foot_0 ## results and analysis the aims of our analysis is (1) to identify the most suitable method for fine-tuning gpt-2 model to ensure generating higher quality training data (see section 4.1), and (2) to understand whether and which seed selection strategies are beneficial for improving da methods, especially for specialised domains which require domain experts to perform manual annotation (see section 4.2). the results for the three datasets are displayed in ## conclusion in this paper, we presented and evaluated data augmentation methods using text generation techniques and seed selection strategies for improving the quality of generated artificial sequences and subsequently classifier's performance in few-shot settings. our results showed that gpt-2 fine-tuned per label, even using only handful of instances, leads to consistent classification improvements, and is shown to outperform competitive baselines and the same gpt-2 model fine-tuned on the entire dataset. this highlights the importance of label preservation techniques in the performance of tg-based da methods, especially for generating longer sequences (such as passages or full documents). seed selection strategies proved to be highly beneficial for the specialised domain analysed in this paper, especially when experts are involved in the selection of class-indicative instances. this shows that combining generative models and active learning techniques, i.e., injecting experts knowledge, can lead to significant improvements in data augmentation methods especially for more specialised domains which require domain experts for the annotation of documents. in future, we plan on expanding the experiments for wider range of specialised domains and compare the performance of bigger generative models such as gpt-3 @xcite , transformer-xl @xcite and ctrl @xcite . further, we want to investigate what is the optimum amount of artificial training data which can be generated with the described techniques before effecting the classifier's performance negatively.