Effects of diversity incentives on sample diversity and downstream model performance in LLM -based text augmentation (2024)

Abstract: The latest generative large language models (LLMs) have found their application in data augmentation tasks, where small numbers of text samples are LLM-paraphrased and then used to fine-tune downstream models. However, more research is needed to assess how different prompts, seed data selection strategies, filtering methods, or model settings affect the quality of paraphrased data (and downstream models). In this study, we investigate three text diversity incentive methods well established in crowdsourcing: taboo words, hints by previous outlier solutions, and chaining on previous outlier solutions. Using these incentive methods as part of instructions to LLMs augmenting text datasets, we measure their effects on generated texts’ lexical diversity and downstream model performance. We compare the effects over 5 different LLMs, 6 datasets and 2 downstream models. We show that diversity is most increased by taboo words, but downstream model performance is highest with hints.

## introduction the emergence of large language models (llms) such as gpt-4, llama, etc., has sparked interest in using them to augment textual datasets @xcite @xcite . in these scenarios, the number of samples is expanded by paraphrasing existing ones through llm prompting. the created paraphrases are then added to the original dataset and used for downstream model training. such methods have been explored for various domains such as sentiment classification @xcite , news classification (piedboeuf and langlais, 2023) and health symptoms classifications @xcite . however, investigation of the effect of various prompts, specific instructions, and selection of seed data inspired by crowd in the text augmentation process when using llms is lacking. crowdsourcing is an established practice for collecting training or validation examples for a variety of nlp tasks. scenarios of data collection using human workers can be similar to those of data augmentation: workers create paraphrases on existing sentences chosen from a dataset. the aim of such data collection is to increase the data diversity and subsequent performance of classifiers trained on the data @xcite . to increase the diversity, various methods are used in crowdsourcing to guide workers. these include taboo words @xcite ) -where most significant words from the collected data are identified and listed in the worker instructions to be avoided during paraphrasing, chaining @xcite -where outliers in the previous paraphrases are identified and used as seed sentences in the next round of data collection, and hints where previous outlier paraphrases are used as examples in the instructions. the hints @xcite ) method itself is similar to llm in-context learning, where examples are included in the instructions for the model to achieve better performance. all of these diversity incentive methods report increased diversity of paraphrases and some also report increased performance of the classifiers trained on the so-collected data. this work is inspired by the parallels between crowdsourcing and llm prompting and by the performance of diversity incentive methods on the diversity of paraphrases and the performance of models trained on them. we investigate the effects of the three diversity incentive methods (originating in crowdsourcing) on data augmentation using llms. the baseline, taken from a previous study @xcite , is a simple prompting for paraphrases. measuring paraphrase diversity and downstream performance of classification models, we assess whether the diversity incentives (added to the base prompt) improve llm outputs similarly as in crowdsourcing scenarios. to our knowledge, this is the first work to investigate the effects of diversity incentive methods on llms. in this paper, we answer the following research questions: rq1: does the usage of diversity incentive methods on llms yield more diverse paraphrases? (compared to base prompting) rq2: do classifiers achieve better performance if trained on data augmented using diversity incentive methods on llms? (compared to base prompting) to answer these questions foot_0 , we have conducted a data augmentation experiment using 5 different llms on 6 different datasets in the tasks of sentiment (movie and app reviews), news, and intent (flight and voice assistant commands) classification. in this experiment, we repeatedly collect llm paraphrases using different diversity incentive methods. then, we compare the lexical diversity of the collected data and the performance of downstream classifiers. additionally, we also conduct an ablation study, where we modify the diversity incentive methods with random data to validate, that the inputs used by these methods (e.g., most influential taboo words, outlier paraphrases) contribute to the method's performance and a combination of the best performing methods for lexical diversity and model performance. in total, we collected 253,500 paraphrases. the most prominent findings are the following: 1) we do not observe statistically significant improvements in lexical diversity of the generated datasets, but only minor improvements using the taboo method, 2) the hints method increases the performance of classification models trained on such data compared to the baseline, while also reducing standard deviation and thus increasing the stability of results, 3) the chaining method and taboo method both do not significantly affect the performance of classification models trained on such data compared to the baseline. ## data collection and evaluation methodology we collected paraphrases for all combinations of the following: 5 different llms, 6 datasets, and 3 diversity incentive methods + 1 base prompting. for each combination, 5 collection iterations were performed: in each 6 random seed sentences per label were drawn from a dataset. for each prompt fired, 5 paraphrases were collected. this totalled in 142,500 collected paraphrases when aggregated all together across datasets and llms. for the ablation study and combination of best methods in section 6 we collected an additional 111,000 paraphrases in total. as the diversity incentive methods need some previously collected data to determine their cues (hints, seeds or taboo words), each iteration consisted of 2 rounds: first we collected data using only the basic prompt and in the second round, we collected data using the given diversity incentive method (or base prompt method). thus, the resulting datasets for each method consist of seed data and data collected from both rounds. the entire data collection process is visualized in figure 1 . after the paraphrases were collected, we evaluated them in several steps. first, we manually checked the validity of a subset (50%) of the collected data (i.e., is the created sample a true paraphrase retaining the label?). second, we computed the diversity of the collected data, comparing the mean vocabulary size (no. unique words) and mean number of unique 3-grams for each diversity incentive method (refers to rq1). third, we evaluated the performance of models trained on the created paraphrases (refers to rq2). for each combination of llm, dataset and method, we finetuned bert-large 5 times and mistral-7b-v0.1 3 times (the dataset also determined the classification task to which a model was finetuned). we evaluated the accuracy of trained model on the full test set for that given dataset specifically and on a subset of the test set for mistral to save computational resources following previous works @xcite @xcite , as the inference time is long and costly. details of the finetuning process can be found in appendix d and e. ## finetuning models on data collected via diversity incentive methods to investigate whether the diversity incentive methods improve the performance of downstream models, we finetuned bert-large 5 times and mistral 3 times for each llm-dataset combination. additionally, as we work with limited data, which was found to cause large variance and instability in finetuning results @xcite @xcite , we sampled data 5 times. this resulted in 25 finetuned classifiers for bert (5 data collection rounds and 5 finetunings for each of those data collection rounds) and 15 for mistral that we evaluate per dataset-llm combination. the full details about hyperparameters and the finetuning setup of bert and mistral classifier can be found in appendices d and e respectively. we report the accuracy of the finetuned models on the test split of each dataset and focus on 2 main attributes: mean accuracy and stability of performance (by measuring standard deviation of accuracy).additionally, we also conducted mann-whitney-u tests (@xmath0.05) between the baseline prompt method and other diversity incentive methods. we are interested in consistent, better performance of a diversity incentive method over the prompt baseline across llms and datasets, as fluctuating performance could be an indicator of random effects. see summary in ## combining diversity incentives as the taboo method achieved best results in lexical diversity in and the hints method achieved best results in model performance, as follow-up, we decided to combine these two methods to see if we can achieve an improvement. we have performed the data collection and finetuning process in the same way as described in section 3. in terms of lexical diversity, the method itself does not have any statistical significance on the results, although the mean number of unique words is higher than the baseline in 18/30 cases and the number of unique n-grams is higher in 16/30 cases. however, in some of the remaining cases a considerable (more than 5%) drop can be observed. in terms of model performance, the combined method statistically significantly decreased the model performance over baseline in 5/30 cases with no increases for bert and increases performance in 4/30 cases for mistral. additionally, it always performed worse as either the hints or taboo method. in summary, the combination of hints and taboo method into one method grants little to no advantage over either of the methods in both lexical diversity and model performance. we hypothesize that this might be due to the more complicated instructions to the llm when collecting the data. a decoupling of the methods in a chain of tasks could potentially improve this approach in the future. ## discussion given the results of our experiments, we note these following observations: first, contrary to the performance of diversity incentive methods observed by related work in crowdsourcing settings (better lexical diversity of paraphrases and better performance of downstream models), not all of the methods show improvement of the lexical diversity when used with llms. the worst performing method is the chaining method, where recent works already pointed out that llms create progressively worse paraphrases when using their own outputs as seed sentences repeatedly @xcite . however, none of the changes in lexical diversity are of statistical significance. second, the best performing method for data augmentation is the hints method, which is similar to in-context learning where demonstrations of samples are provided to the llm as part of the prompt. this might be the reason why this method works so well, as the own paraphrases of the llm guide it to better output, similar to in-context learning. third, we observe that, contrary to some previous works @xcite , the lexical diversity of the paraphrases does not correlate with performance of models trained on them. even though the data collected using the taboo method yield highest lexical diversity, models trained on such data do not achieve consistently better performance against baseline. fourth,the increase in mean performance and stability seems to be small, but in relative terms (compared to the baseline method) it seems to be significant, as the increase of mean performance can range from 0.6% to 2.5% increase over baseline for bert and 1% to 11% increase for mistral. for stability, the increases are even more significant: for bert the range is between 5% to 35% increase over baseline and for mistral from 10% to 66%. fifth, diversity incentives require additional computations (for significant words and outlier paraphrases) and also require larger llm context (e.g., hints use additional paraphrases in instructions of the model), meaning higher costs. as such, the increased computation costs may not warrant the use of diversity incentives. sixth, the combination of the best method for lexical diversity (taboo) and best method for model performance (hints) did not yield the increases in both lexical diversity and model performance, but performed rather poorly. we hypothesize that this might be due to the increased context length for the llm with additional instructions that are hard to perform in one single action. the promising results using the hints method opens possibilities for investigations of incontext learning for text generation in llms, as the quality of such generated data using hints seems to be better than without them. this is in line with the recent results @xcite that indicate that the usage of previous examples in instructions for llms leads to better generated data. ## conclusion in this work, we investigated the effects of different diversity incentive methods used in crowdsourcing on the lexical diversity of llm-augmented textual datasets and performance of classification models trained on such data. we compared 3 of such methods with a baseline of using only prompts asking the llm to paraphrase a given seed. we experimented with 5 llms on 6 datasets. our results indicate that the taboo method increases lexical diversity of the collected data, but that this change is not of statistical significance and affects performance only randomly. the hints method affects lexical diversity randomly, but increases the performance of classification models (both in stability of and mean performance) that were trained on data collected using this method. the chaining method does not improve lexical diversity or model performance of classification models trained on data collected using this method. the combination of hints method and taboo method does not significantly increase the lexical diversity or model performance. a common downside of diversity incentive methods is the increase of inference costs. also, there is still some randomness present when using these methods, as even the best performing methods do not increase lexical diversity or performance of models in all cases. the notable relative increase in stability of performance and mean performance of models trained on data collected using the hints method indicates that llms can produce data of better quality using this method when aiming for downstream task classifier performance.