LLM s vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs? (2025)

Abstract: The generative large language models (LLMs) are increasingly being used for data augmentation tasks, where text samples are LLM-paraphrased and then used for classifier fine-tuning. Previous studies have compared LLM-based augmentations with established augmentation techniques, but the results are contradictory: some report superiority of LLM-based augmentations, while other only marginal increases (and even decreases) in performance of downstream classifiers. A research that would confirm a clear cost-benefit advantage of LLMs over more established augmentation methods is largely missing. To study if (and when) is the LLM-based augmentation advantageous, we compared the effects of recent LLM augmentation methods with established ones on 6 datasets, 3 classifiers and 2 fine-tuning methods. We also varied the number of seeds and collected samples to better explore the downstream model accuracy space. Finally, we performed a cost-benefit analysis and show that LLM-based methods are worthy of deployment only when very small number of seeds is used. Moreover, in many cases, established methods lead to similar or better model accuracies.

## introduction the emergence of recent large language models (llms) such as gpt-4, gemini, llama, and their wide availability prompted their use in augmentation of textual datasets @xcite @xcite @xcite @xcite . in most llm-based augmentation scenarios, the dataset size is increased through paraphrasing of original samples. the extended datasets are then used for training small downstream classifiers with small inference costs. llm augmentation has been used in various domains such as sentiment analysis @xcite , intent classification @xcite , news classification @xcite and health symptoms classification @xcite . while llm augmentation improves downstream classifiers, it is also costly (power consumption, co 2 emissions), as generative models often feature parameters in tens of billions. this is magnitudes higher than other established (most used) augmentation methods, including back translation paraphrasing, or bert-based word insertion and synonym swap. previous works have measured classifier performance, comparing llm-based and established augmentation methods. the results have so far been conflicting and mixed with studies reporting the llm-based augmentation to be superior on classifier performance @xcite , while others report only marginal gains and even the established augmentation outperforming the llm-based one @xcite . furthermore, existing studies were limited in terms of parameters, neglecting the variety of available llms, the potential impact of the number of seed samples and collected samples, the cost of these methods, and the variety of classifiers and their fine-tuning methods. the goal of this paper is to compare the accuracy and cost-benefits of the most used established text augmentation methods with their recent llm-based counterparts. compared with previous studies, this paper offers a more systematic and finer-grained comparison over multiple dimensions to specifically identify cases where and how do llm-based augmentation methods outperform established text augmentation methods. we are the first to consider the co 2 emissions and costs of these methods to better identify cases where llm-based augmentation could be preferable. we formulate the following research questions: rq1: considering downstream classifier accuracy, in which cases do the established tex-tual augmentation methods work equally or better than the llm-based methods? rq2: in which cases does the cost of using llmbased textual augmentation methods instead of established ones outweigh its benefits? we empirically investigated three techniques commonly used in textual augmentation: paraphrasing, word inserts, and word swaps (replacements). all three exist in both established and llm-based variants. in the established variant, paraphrasing is done through back-translation using an rnn @xcite , while inserts and swaps use a bert-based approach @xcite . for llm-based variants, we prompted 2 llms (gpt-3.5 and llama-3 foot_0 ) to perform all three techniques. we experimented with 6 different datasets (with tasks of sentiment analysis, news classification, and intent classification), 3 downstream classifier models (bert, roberta, distilbert), and 2 fine-tuning approaches (fully fine-tuned, and qlora @xcite ). furthermore, we investigated various numbers of seeds and collected samples used in the augmentation. together, this resulted in a total of 267,300 fine-tunings, from which we identified the best-performing llm and established methods (answering q1). these were then further scrutinized under cost-benefit analysis (answering q2). the most prominent findings are: 1) the best llm augmentation methods outperform established ones only when a small number of seeds is used. the advantage of llm-based augmentation diminishes with increased seed numbers, making it less cost-feasible. this hints towards using llmbased methods only in scenarios with a small number of seeds per label (5-20). 2) llm augmentation methods have a higher impact on the accuracy of less robustly pre-trained classifiers such as distil-bert or bert. 3) llm augmentation methods have a higher impact on classifier accuracy for full fine-tuning when compared to qlora fine-tuning. ## related work: text augmentation text augmentation is a process of increasing the diversity of training text data without necessarily collecting more original (or seed) data. text augmentation was inspired by image augmentation @xcite where various techniques such as cropping, rotating, flipping, etc. were used to build models that are more robust to image variation and in turn enhance their performance. text and data augmentation have an increasing number of google weekly trend searchers in recent years @xcite @xcite , indicating an increasing interest in these kinds of model performance enhancing methods. one of the most established is character-based augmentations @xcite , where given a seed text, a new sample is created via character insertion, replacement, or deletion. another method is backtranslation @xcite , which translates a given text into one language to then translate it back, essentially creating a paraphrase. various llms such as gpt-2 @xcite or bart @xcite have also previously been used to create paraphrases. additional extensions used style transfer to create paraphrases of a certain linguistic style @xcite , syntax control of the generated paraphrases @xcite , multi-lingual paraphrases @xcite and llm finetuning using qlora for specific domains @xcite . another established method is the usage of pre-trained llms to generate new samples by either word insertion or replacement of words via masking certain parts of the seed text and allowing the llm to find good replacements for the masked parts of the text @xcite . text augmentation methods were adapted with the rise of new llms such as gpt-4 or llama to leverage these new powerful models to generally create paraphrases of given seed texts. a recent study @xcite found that the gpt-3.5 paraphrasing provides an increase in classifier accuracy. however, it does not outperform the previously established text augmentation methods by a significant margin (also for lowresource settings). in contrast, two studies reported better performance in using llms as data augmenters than using previous state-of-art techniques in both the paraphrasing of existing texts @xcite and in a zero-shot setup of generating new texts using specific prompts @xcite for low-resource settings. a comparison of these studies is problematic, as each study varied a large number of parameters such as different numbers of seed samples (from 10 to 1000 per label), different numbers of collected samples per seed used (from 5 to 20), classifiers used, etc. regardless of the mixed results reported, newer llms have been used for a variety of augmentation tasks and domains such as automated scoring @xcite , low-resource language generation @xcite , sentiment analysis @xcite @xcite , news classification @xcite , content recommendation @xcite and health symptoms classifications @xcite . given the wide usage of llm augmentation methods and the mixed results of studies @xcite @xcite comparing them with established augmentation methods, a finer analysis of cases where one of these types of methods is preferable is required. ## study design to assess the advantages of either established or newer llm-based augmentation methods on a finer scale and to tackle the mixed results reported by previous works, we performed a comparative study. at its core was the same basic scenario (see also figure 1 ): on a given text classification dataset, a number of seed samples were selected for each class. for each seed sample, a given augmentation method generated a number of additional "augmented" samples. both original data and the augmented samples were then used to fine-tune a downstream classifier. this scenario was repeated for all examined methods and a variety of parameters (see below), resulting in a total of 37,125 augmented samples and 267,300 fine-tunings. then, the accuracy of the resulting classifiers was compared to answer q1. to answer q2, the augmentation costs in terms of computation time, finances, and co 2 emissions were determined and weighted against accuracy gains in a cost-benefit analysis. we publish all of our measurements, the code, and the data used. 2 the study had the following parameters: ## study results our study has multiple parameter dimensions that together yield more than 11 thousand combinations. to keep the result presentation manageable, we collapse some of these dimensions (each of them with a different reasoning). one dimension we could simplify are the augmentation methods themselves. to keep the comparison of established and llm-based methods simple, we only compared the best-performing methods from each group (best downstream model accuracy). while the established method group contained 3 methods (given by the 3 augmentation techniques and their established implementations), the llm-based method group contained 6 methods (the same 3 techniques, each implemented by 2 different llms foot_2 ). we performed this comparison for each parameter combination of number of seeds, number of augmented samples per seed, classifier, fine-tuning approach, and dataset. among the llm-based methods, the paraphras-ing technique performed best in 56% cases, followed by word insert, which topped 30% of cases, and word swap with 14% of cases. although paraphrasing performed best overall, the word insert worked best when the roberta classifier was finetuned. among the established methods, the contextual word insert performed best in 56% cases, followed by (backtranslation) paraphrasing topping 26% cases, and contextual word swap with 18% cases. furthermore, the backtranslation had a stronger effect on classifier accuracy with full fine-tuning and lesser with qlora. given these results, we decided to focus on the comparison of the llm-based paraphrasing with the established contextual word insert methods. see appendix e for other method comparisons. another dimension we could collapse was the number of collected samples per seed, where we selected only the most accurate classifier for the same combination of other parameters. however, full details on how the number of collected samples per seed influences the classifier accuracy can be found in appendix d. ## discussion the results of our experiments lead to the following observations: first, the paraphrasing method was the best within the newer llm-based augmentation methods, considering classifier accuracy. this could be due to the demonstrated ability @xcite of the newer llms to create very diverse paraphrases, being less constrained by seed samples. the contextual insert method worked best within the established augmentation methods. this may be caused by the backtranslation method creating a lot of duplicated samples and the contextual swap method introducing less variety than the contextual insert method. second, the number of cases in which the paraphrasing method (an llm-based method) significantly outperforms the established contextual insert method decreases with more seed samples per label. this is similar to previous studies @xcite , as we observed this in nearly all the cases for a small number of seeds and differs from the results of the previous study @xcite , where such increase was observed less often. a different number of collected samples, classifiers used, and other factors might be the reason for this disparity. the llm-based methods achieve better classifier accuracy than established methods in the cases of very small third, furthermore, when we increased the number of seeds, we observed a decrease of accuracy differences between models trained on data from paraphrasing and models trained with contextual insert, similar to @xcite . the highest relative increase in model accuracy with paraphrasing instead of contextual insert appears with 5 to 20 seeds per label. after 30 or more seeds were used, the relative difference between methods decreased. additionally, the difference between the llm-based and established methods in terms of monetary costs, time costs, and emissions is quite significant (see section 4.2). therefore, it seems beneficial, from the perspective of both cost and model accuracy, to use the newer llm-based augmentation methods only in low-resource settings. fourth, we observed some exceptions to the trends reviewed above. the fine-tuned roberta models (which provided the best classification accuracy among the fine-tuned models) generally benefited more from augmentation methods that used insertion of words. this might be due to a more robust pretraining of roberta, where augmentations that introduce more noise are less beneficial for training. another case was the finetuning of bert using qlora, where, for some cases, the paraphrasing method was either considerably better or worse than the contextual insert method for classifier accuracy. this might be due to differences in the pre-training data and processes used for bert in comparison with distilbert or roberta, making it far more sensible to text augmentation methods when using qlora. fifth, the difference between the paraphrasing and the contextual insert method on model accuracy had much more variance for qlora than for full fine-tuning. when the paraphrasing method is used for qlora on classifiers, the increased accuracy (compared to contextual insert) is generally smaller than with full fine-tuning. llm paraphrasing's sample variability might be providing more benefits when the model can leverage it through full fine-tuning. sixth, the combination of the best-established methods does not improve their overall accuracy of the downstream model compared to using contextual insert. this might be due to the combination of methods leading to a possible distribution shift or models overfitting on the augmented data. to summarize, as the costs of using established augmentation methods are considerably lower than the newer llm-based methods and the increase in model accuracy decreases quickly with more seeds used, it appears to be beneficial to use them instead of newer llm methods for a higher number of seeds per label when targeting model accuracy and use llm-methods in cases of low-resource setting where the relative gain in accuracy is highest. ## conclusion we compared the effects of newer llm-based and established textual augmentation methods on downstream classifier accuracy for combinations of 6 datasets, 3 classifiers, 2 fine-tuning approaches, 2 augmenting llms, various numbers of seed samples per label and numbers of augmented samples per seed. in total, we analyzed a total of 267,300 fine-tunings. we aimed to identify cases where llm-based augmentation outperforms established approaches in order to shed light on contrary results from previous studies. we identified the paraphrasing method as the best-performing llm-based and the contextual insert as the best-performing established augmentation method. the comparison of these two best methods indicates that the use of llm-based methods for data augmentation, instead of established methods, is only warranted for a small number of seed samples per label (5 to 20). there, we observed a statistically significant increase in cases where llm-based methods are better and observed higher relative increases in model accuracy compared to established methods. however, with an increasing number of seeds per label, this effect decreased, and the number of cases of established methods having a higher influence on the accuracy of classifiers increased. as newer llm methods are considerably more costly than established methods, their use is justified only for low-resource settings, where differences between the method's costs are smaller.