R ank A ug: Augmented data ranking for text classification (2023)

Abstract: Research on data generation and augmentation has been focused majorly around enhancing generation models, leaving a notable gap in the exploration and refinement of methods for evaluating synthetic data. There are several text similarity metrics within the context of generated data filtering which can impact the performance of specific Natural Language Understanding (NLU) tasks, specifically focusing on intent and sentiment classification. In this study, we propose RankAug, a text-ranking approach that detects and filters out the top augmented texts in terms of being most similar in meaning with lexical and syntactical diversity. Through experiments conducted on multiple datasets, we demonstrate that the judicious selection of filtering techniques can yield a substantial improvement of up to 35% in classification accuracy for under-represented classes.

## introduction recent advances in large language models have brought along incredible progress in a wide range of nlu tasks. however, for domain specific tasks, fine-tuned models can bridge the performance gap with data @xcite but such domains are often low resource in nature and data collection can be quite difficult. this is where data augmentation techniques come into play, boosting model performance for a given supervised task by generating novel data points that are similar in characteristics to the available data. there have been a large number of metrics created to evaluate data augmentation which are mostly focused on the performance of generation models @xcite @xcite . we explore various methods to evaluate and filter generated paraphrases @xcite to get high quality augmentations. most of the prior work in this domain makes use of metrics that only take into consideration the word or embedding level similarity of the generated utterance. popular metrics like bleu score @xcite , recall-oriented understudy for gisting evaluation (rouge) @xcite , and metric for evaluation of translation with explicit ordering (meteor) @xcite use n-gram based comparison. this type of evaluation is limited to a one-dimensional analysis of augmentation as highquality data provides both contextual similarity as well as lexical diversity @xcite to the original text. to ensure this, we propose a text ranking method that outperforms other popularly used metrics to get top quality augmentations that aid in better training of models on downstream tasks. this method can be extended to any data augmentation model for evaluation and is independent of the training model as well. despite a variety of approaches for augmented data evaluation, there is no golden standard @xcite , the real value of the generated data can be only evaluated through downstream tasks, for example by estimating how much performance improvement synthetic data can bring to the targeted supervised nlu task. in our case, we test our ranking and filtering mechanism on multiple supervised classification based scenarios for skewed datasets. it shows a consistent improvement across different experimental setups and datasets compared to the standard filtering metrics. finally, our method foot_0 is also extended to a german dataset, to show that it can be applied not only to english but also to other languages. ## related works in recent years, data augmentation and generation techniques have gained significant attention in machine learning research. these techniques play a crucial role in enhancing the performance and robustness of various models across different domains. augmentation techniques, in general, have been traditionally used in many downstream computer vision task kingma and welling (2013) uses variational autoencoders to encode the data examples to a latent representation and then new samples were generated from that latent space, which employs patch based augmentation. @xcite al. (2016) uses rule based image transformations to generate more data for improving performance of convolutional neural networks (cnns) on feature learning tasks. text generation has been studied extensively leading to computational linguistics and diverse methods being suggested ever since. sentence structures are very different and these diversities expand in different types of social media which makes text generation harder. rule based techniques like word replacement using finite state transducers rastogi et al. (2016) and synonym swap şahin and steedman (2019) have been some of the initial attempts at generating synthetic texts. most such rule based methods suffer from a lack of sentence structure variation and loss of semantic context. multiple efforts have been made recently to use generative models too for text augmentation. existing augmentation methods work at different granularity levels -characters, words, sentences, and documents. yu et al. (2018) and hou et al. (2018) use sequence-to-sequence generation for enhancing model performance in back translation and text transfer domains. ding et al. ( 2020 ) proposes a novel approach to utilize generative augmentation for fine-grained and token-level entity tagging tasks. pre-trained masked language models (mlms) like @xcite , @xcite and @xcite , which internally uses chatgpt, can be used for contextual augmentation too. since mlms are pre-trained on a large number of texts, contextual augmentation can usually generate meaningful new texts. ## data in our experiments, we make use of two datasets -airline travel information system (atis) from the microsoft cognitive toolkit (cntk) @xcite , an intent classification dataset, hate speech from a white supremacist forum de @xcite , a sentiment analysis dataset and amazon multilingual reviews @xcite , a product reviews corpus. all of these are standard datasets, ideal for setting benchmarks on classification tasks. ## experiments in this section, we describe the experimental setup for benchmark tests along with the sentence generation pipeline. ## results for the atis and hate speech datasets, which are imbalanced datasets, we generate paraphrases for each underrepresented data point. this excludes the atis flight and not hate classes for atis and hate speech respectively. this gives us around 38358 for atis and 16626 samples in total for hate speech. however, models trained using the original data and every generated paraphrase result in a decrease in performance, highlighting the importance of using quality data points. we then apply different filtering methods on the generated data. all our filtering methods showcase a consistent improvement on both datasets and outperform the baseline. as evident in comparing against the different metrics, our method gives an increased performance for both datasets in @xmath0. for @xmath1, our filtering method comes close to matching the top performer. across all 3 datasets, our method shows the best overall performance showing an increase compared to both the baseline performance and the no-filtering setting. in the case of atis and german review datasets, it can also be noted that no filtering augmentation setting actually reduces the performance when compared to the baseline. this essentially infers the fact that just adding augmentations does not necessarily improve performance but the generated text being of good quality is what yields good results on downstream tasks. in tables 2 and 3 , we can see the performance improvement across different classes. we also extend our work on a balanced but a low resource german dataset. our filtering method outperforms all other methods for both settings indicating that rankaug is adaptable to other languages as well. ## conclusion when working with low resource and unbalanced datasets, data augmentation can significantly improve performance. however, it is crucial to have quality augmented data. we explored and evaluated a number of popular evaluation metrics for augmented data filtering and proposed our own method for ranking and filtering quality paraphrases. our method, along with similarity also accounts for paraphrase diversity and achieves the best overall performance across multiple datasets while utilizing nearly half the total augmented data. along with this, we also observe a consistent increase in performance of the underrepresented classes of the datasets explored achieving up to 35% increase in accuracy. we show that our approach can be extended to other languages as well as other varied domains to improve downstream performance and as a future work, we aim to benchmark these methods on more downstream tasks.