LLM 2 LLM : Boosting LLM s with Novel Iterative Data Enhancement (2024)

Abstract: Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a Llama-2-7B student model. Our code is available at https://github.com/SqueezeAILab/LLM2LLM.

## introduction pretrained large language models (llms) have achieved impressive performance on various benchmarks and datasets that have previously required *equal contribution specialized neural network architectures. for many of these general benchmarks @xcite , llms are prompted with custom instructions or in-context examples. however, in various real-world applications, these prompting strategies are not a one-size-fitsall solution. for instance, llms have a limit on the amount of input context they can process, thus limiting the number of in-context examples or instructions we can input to make the llm follow a certain behavior. for simple tasks that are closely aligned with the data that the llm was pretrained on, extensive prompting may not be necessary. however, applying llms to specialized domains (e.g., a specific medical field @xcite or private data with niche protocols) can be more challenging, often requiring prohibitively long prompts to achieve adequate performance. even if the prompt length does not exceed the limit, processing long prompts increases the latency and cost of each inference. additionally, llms also tend to forget or ignore information in long contexts @xcite , leading to potential accuracy drops even when the model can handle long input prompts. while retrieval-augmented generation (rag) @xcite has been developed to address some of these challenges, it may sometimes retrieve irrelevant passages or documents, which can potentially degrade the generation performance. furthermore, rag does not necessarily solve the latency and cost issue as processing a long input prompt may still be required. a promising method for addressing this is finetuning. with the emergence of parameter efficient fine-tuning (peft) @xcite , the computational resources required to fine-tune a task-specific llm have decreased significantly. however, herein lies a new problem: successful fine-tuning requires enough training data. this can be challenging for some applications, where we only have access to a small amount of task-specific data. often, collecting, cleaning, and labeling additional data can be costly and timeconsuming. so the key question is: how should we increase the user's training data to be enough for fine-tuning? data augmentation is a known method that could help effectively expand the training dataset. for natural language processing (nlp) tasks, one can use approaches such as synonym replacement, character replacement (e.g., by intentionally introducing spelling errors), random swapping, and back translation, just to name a few @xcite @xcite . however, these approaches fail to effectively expand the training data for fine-tuning llms in the case of new and specialized tasks, as we will show later in section 4.3. to address this, several recent papers have explored using an llm to expand the fine-tuning dataset @xcite @xcite @xcite @xcite . this approach has proven to be more effective than traditional data augmentation methods. however, these approaches often apply llm-based data augmentation on all of the available training dataset, without considering the llm's prediction accuracy on individual training data points. we have observed that for various reasoning tasks such as arithmetic and reading comprehension, the llm correctly solves simpler examples in the fine-tuning dataset, but may struggle with harder examples. it will be sub-optimal to keep augmenting data points for which the llm is already achieving high accuracy on. to address these challenges, we introduce llm2llm, a new targeted and iterative data augmentation framework that uses a teacher llm to expand the training dataset, with a targeted and iterative approach. in more detail, we make the following contributions: ## methodology we assume that we are given an llm model m (e.g., gpt-3.5 or llama-2-7b) that is pre-trained on some source dataset (e.g., common crawl). the goal is to adapt m (hereon called the student model) to a new target domain by using a small algorithm 1 llm2llm: boosting llms with novel iterative data enhancement. given a seed dataset d 0 , we finetune the model m i student , evaluate, and extract training set data points that the model gets wrong. these are used to generate new training data points using the teacher model m teacher for the next step. to enrich d, auggpt @xcite has introduced a promising approach that generates additional augmented data by applying a prompted llm to all available data points in the target training dataset. however, this method falls short by indiscriminately augmenting data without considering the student model's varying performance across different data points. for instance, the model may easily solve the majority of the dataset, but it may struggle with a small subset of more challenging examples. in this case, rather than indiscriminately expanding the dataset by replicating simpler cases, a better augmentation strategy would be to generate more data points that align conceptually with these challenging examples. this is because the former approach could lead to longer training time without noticeable performance improvement. here, we propose a more general formulation of an llm-based data augmentation pipeline that addresses the aforementioned limitation. to do so, we consider the following iterative process: in equation (1), m teacher is the teacher model, m student is the student model (potentially being fine-tuned in many iterations), n refers to the n th step of data augmentation, d n+1 is the new training dataset at the next iteration, and f is the datageneration algorithm. at each step, the teacher model has access to how the student model performs at the n th step (e.g., correct/incorrect labels, or possibly prediction distributions for white-box models), and based on that it can edit training data points for the next iteration. note that llm2llm is different from knowledge distillation @xcite . knowledge distillation is generally applicable to cases where the teacher model has high accuracy on the target data. in contrast, in this case, it is possible that the teacher model also performs sub-optimally on the target data (e.g., in the private database case, where the teacher lacks domain-specific knowledge). however, if the teacher model has enough reasoning capability to produce conceptually similar but semantically different examples when it is given both the prompt and answer, then our framework can improve performance. in llm2llm, we consider a specific instantiation of equation (1), as discussed next. ## conclusion we have introduced llm2llm, an adaptive and iterative llm-based data augmentation framework that uses llms to scale up smaller fine-tuning datasets in lieu of manually generating more data. this framework substantially reduces the amount of real data needed, and it allows us to efficiently scale the dataset with synthetic data that can match or even exceed the effect of hand-collecting more data. the method is effective because of the iterative and targeted nature of the process, which allows us to boost the signal from data points that the llm gets wrong. as a result, we were able to achieve a 24.2% improvement on gsm8k, 32.6% on casehold, 32.0% on snips, 52.6% on trec, and 39.8% on the sst-2 dataset in the low-data regime using a llama-2-7b student model. future work can focus on tuning the hyperparameters of our framework as well as incorporating our approach with other llm techniques such as prompt tuning and few-shot learning.