Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM -based Text Augmentation (2025)

Abstract: The generative large language models (LLMs) are increasingly used for data augmentation tasks, where text samples are paraphrased (or generated anew) and then used for downstream model fine-tuning. This is useful, especially for low-resource settings. For better augmentations, LLMs are prompted with examples (few-shot scenarios). Yet, the samples are mostly selected randomly, and a comprehensive overview of the effects of other (more ”informed”) sample selection strategies is lacking. In this work, we compare sample selection strategies existing in the few-shot learning literature and investigate their effects in LLM-based textual augmentation in a low-resource setting. We evaluate this on in-distribution and out-of-distribution model performance. Results indicate that while some ”informed” selection strategies increase the performance of models, especially for out-of-distribution data, it happens only seldom and with marginal performance increases. Unless further advances are made, a default of random sample selection remains a good option for augmentation practitioners.

## introduction the emergence of recent large language models (llms) such as gpt-4, gemini, llama, and their wide availability prompted their use in augmentation of textual datasets @xcite @xcite @xcite . llm augmentation has been used in various domains such as sentiment analysis @xcite , intent classification @xcite , news classification @xcite , and health symptoms detection @xcite . these augmentations are often performed in a low-resource setting with a limited number of seed samples. in most llm-based augmentation scenarios, the dataset size is increased through paraphrasing of original samples or generation of completely new samples that adhere to a specified label. this can be done without any samples provided (zero-shot). alternatively, one can include already existing samples as part of the prompt to better instruct the llm (few-shot). the augmented datasets are then used for training downstream models, which are usually much smaller than the prompted llms, and thus cheaper and more suitable for production environments. recent studies report better performance for fewshot llm-based augmentation, as compared with zero-shot approaches @xcite . most existing few-shot augmentation studies select the samples randomly, and the potential of using more informed selection strategies (existing elsewhere in few-shot learning literature) is under-explored. furthermore, augmentation studies focus only on paraphrasing and are evaluated on in-distribution data. in few-shot learning, the informed sample selection strategies aim to select the most relevant samples that would lead to better outputs. the samples can be selected based on their similarity, diversity, informativeness, or quality @xcite @xcite . through these methods, llms can potentially produce better augmentations in return for the additional computation costs of the informed sample selection. literature shows that the choice of samples for few-shot learning significantly influences its outcomes (i.e., sensitivity of sample selection) @xcite @xcite . for example, recent studies have investigated the effects of such sample selection strategies for in-context learning @xcite or llm alignment @xcite . however, for augmentation scenarios, an investigation of sample selection strategies effects is lacking. the goal of this paper is to compare existing sample selection strategies in few-shot text augmentation for a low-resource setting. this comparison is measured by the performance of downstream models trained on the augmented data. we investigate the typical paraphrasing scenario, but also less covered generation of new samples. along with more frequent in-distribution (id), we also evaluate out-of-distribution (ood) data. we run our experiments for various llms and tasks. we identify the best-performing sample selection strategy in each scenario (parameter combination) and compare it against two baselines: (1) the zero-shot augmentation and (2) the few-shot augmentation with random sample selection. we formulate the following research questions: rq1: considering downstream model performance, which sample selection strategy performs the best most consistently? (when considering both in-distribution and out-ofdistribution setups). rq2: considering downstream model performance, when and how often do the bestperforming sample selection strategies outperform the baseline strategies? we compared 8 different sample selection strategies (see 3.1) against 2 baseline strategies (zeroshot/no-samples strategy and random samples strategy) on 3 different llms (llama-3.1, mistral-v0.3, and gemma-2). we experimented with 8 different datasets (for sentiment analysis, news classification, question topic classification, paraphrase detection, and natural language inference) with both in-distribution and out-of-distribution splits on roberta as our downstream model. we used a low-resource setting, using only 20 samples per label. furthermore, we also investigated the composition of the examples from the point of labels (whether it is more beneficial to include samples only from the target label being augmented or also from other labels). we investigated two augmentation techniques: paraphrasing of samples and generation of completely new samples. we repeated the whole process 3 times with different random seeds, ensuring the robustness of our results. the most prominent findings are: 1) none of the existing sample selection strategies is consistently better than the baseline in the majority of cases for in-distribution, 2) selecting examples at random yields the best performance in the majority of cases and does not require additional overhead, 3) for out-of-distribution, the synthetic samples dissimilarity selection strategy yields the highest performance more often than the baseline strategies. it can be considered for uses where overhead selection costs are not an issue. ## related work: llm-based text augmentation soon after their advent, new llms, such as gpt-4 or llama, started to be used as data augmentation tools, leveraging their ability to produce a diversity of texts. the llm-based augmentation is typically done through paraphrasing @xcite @xcite . less often, llms are used to create semantically new samples adhering to a given label @xcite . llm-based augmentation has been used for a variety of augmentation tasks such as automated scoring @xcite , low-resource language generation @xcite , intent classification @xcite , sentiment analysis @xcite @xcite , hate speech detection @xcite , news classification (piedboeuf and langlais, 2023), content recommendation @xcite , and health symptoms classifications @xcite . recent studies have also used few-shot learning as part of the augmentation by supplying the llm with various examples from the dataset in the prompts. it has been leveraged for named entity recognition @xcite , classification performance @xcite ) or text summarization @xcite . while the performance of the few-shot approaches in augmentation seems to outperform zero-shot ones (where no examples are used) @xcite , the effects of various sample selection strategies are under-explored, as many studies simply select the samples randomly. only one study explored other strategies @xcite , which used a human-inspired sample selection strategy. while sample selection strategies have found their usage in various in-context learning tasks (significantly altering the performance of llms) and while some studies already hint at increased performance of few-shot augmentation over zero-shot augmentation @xcite , an investigation of various sample selection strategies for llm-based augmentation methods is completely lacking. ## study design to assess which sample selection strategies work best for llm-based data augmentation, we performed a comparative study in a low-resource setting. the same basic scenario was used in each case: given a dataset, 20 seed samples were selected from each label. for each seed sample, a given llm "augmented" the samples 5 times. this was repeated for each sample selection strategy and type of augmentation technique used (paraphrasing or creating completely new samples). next, a downstream model was fine-tuned on both sub-sampled data and augmented samples and then evaluated on in-distribution and out-of-distribution data. for in-distribution data, we used the original test splits of each dataset, while for the out-of-distribution data, we used test splits from a different dataset with the same task (e.g. yelp dataset test split was used as out-of-distribution data when evaluating performance on the tweet eval dataset for sentiment analysis). this scenario was repeated for all sample selection strategies and baselines for a variety of parameters (see below). then, the performance of the models (measured by f1-macro) was compared for each sample selection strategy to answer rq1. this was followed by comparing the bestperforming sample selection strategies against the best-performing baseline strategy of either zeroshot (no examples provided) or randomly selected examples to answer rq2. we publish all of our results, the code, and the data used foot_0 . we used a broad range of study parameters to ensure the robustness of our results by using both the baseline strategies and the sample selection strategies in a variety of cases. we include the different augmentation techniques and example compositions in terms of labels to capture a wide variety of cases. the whole process was repeated 3 times, and different seed samples were selected. the study had the following parameters: ## study results our study has multiple parameter dimensions, which yielded more than 1,300 combinations. we aggregated the results for each of the used llms. during our analysis, we did not identify any llm bias towards one of the sample selection strategies, as the 3 used llms performed similarly. to keep the comparison of various sample selec-tion strategies simple, we only compare the bestperforming sample selection strategy combination on the dataset given the augmentation techniques of either generation or paraphrasing and composition of labels in terms of labels. we also use the same setting for the baseline strategies of zero-shot and random few-shot. we wish to identify strategies that provide the best performance most consistently (in most cases) and outperform the baselines the most. we analyze the different augmentation techniques and composition of labels and how they influence the model performance in appendix f. we distinguish between the best-performing sample selection strategy for in-distribution data and out-of-distribution data for each of the datasets. to identify the best-performing sample selection strategy (including the baselines) in these cases, we compute the mean of the model performance across all of the random seeds and compare these means. there were a total of 72 cases for 8 datasets, 3 different llms, and 3 different random seeds used. after identifying the best-performing sample selection strategy, we statistically tested its distribution of model performance against the best-performing baseline strategy (either zero-shot or random fewshot based on their mean) using mann-whitney-u tests with @xmath0.05 to measure the number of times the sample selection strategies are statistically significantly better than the best baseline strategy. ## discussion the results of our experiments lead to the following observations: first, the cartography eas.+ambig. samples strategy was best among sample selection strategies for in-distribution classifier performance. such selection strategies seem to influence llms for text generation in similar ways as they do models fine-tuned on such selected data -they increase their performance on in-distribution data @xcite . second, the synthetic samples dis. strategy was best among sample selection strategies for ood model performance. the strategy used in @xcite was inspired by crowdsourcing @xcite methods for collecting data for better ood performance. this method appears to force llms to create more diverse samples by leveraging outlier synthetic data as examples, making the downstream models more robust. third, when comparing the best sample selection strategies against baseline strategies, the cartography eas.+ambig. samples strategy does not outperform the random few-shot selection strategy. not only does the random few-shot strategy perform better more often, but it does so more consistently across multiple datasets. in contrast, the cartography eas.+ambig. samples strategy fails to perform best even once for some datasets. this hinders the applicability of this method, where it is clearly outperformed in some cases by baseline strategies or other sample selection strategies. fourth, the synthetic samples dis. strategy outperforms the baseline strategies for ood performance, but not across all datasets. however, neither the random few-selection selection strategy nor the zero-shot approach performs well on all datasets. this implies that increasing performance across all ood cases for all datasets is a difficult problem. additionally, the synthetic samples dis. method is expensive, as it requires one additional inference from the llm to select examples from. fifth, as seen in figure 2 , the aggregated increase of classifier performance when using sample selection strategies is small or negative, indicating that sample selection strategies do not work well for all random seeds. given the increased costs of using sample selection strategies, this result favours the baseline strategies for text augmentation in general. sixth, comparing the baseline strategies between themselves, the random few-shot selection performs the best on in-distribution classifier performance. in contrast, the zero-shot strategy only performs well on ood classifier performance. this might be due to the llms getting biased towards the examples provided and thus being more likely to produce augmentations that follow the distribution of the seed samples more closely. however, this might not be robust enough for good ood classifier performance. to summarise, while the synthetic samples dis. strategy outperforms the baseline strategies for out-of-distribution classifier performance, the baseline strategies outperform the sample selection strategies for in-distribution classifier performance. however, any increase in classifier performance for both in-distribution and out-of-distribution is marginal and increases costs for collecting text augmentations. while sample selection strategies work best in some cases, they do not so consistently. this underlines the need for better sample selection strategies for llm-based text augmentation. ## conclusion we compared the effects of prominent sample selection strategies of few-shot learning for llm-based text augmentation scenarios in a low-resource setting. we evaluated the downstream model performance on in-distribution and out-of-distribution data. we compared selection strategies against 2 baseline strategies (random few-shot and zeroshot). this comparison was done using 3 different llms, 8 different datasets, and 2 augmentation techniques (paraphrases and new samples). our comparison indicates that the baseline strategies outperform sample selection strategies for indistribution performance. for out-of-distribution performance, the synthetic sample dissimilarity strategy is best in more cases than the baseline strategies. however, the improvements are marginal and are not present in all datasets. given the increased computations needed to use these sample selection strategies and their lacklustre performance, the baseline strategies represent a good default for few-shot augmentation practitioners.