I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM -Generated Responses (2024)

Abstract: This paper explores an intriguing observation: fine-tuning a large language model (LLM) with responses generated by a LLM often yields better results than using responses generated by humans, particularly in reasoning tasks. We conduct an in-depth investigation to understand why this occurs. Contrary to the common belief that these instances is due to the more detailed nature of LLM-generated content, our study identifies another contributing factor: an LLM is inherently more “familiar” with LLM generated responses. This familiarity is evidenced by lower perplexity before fine-tuning. We design a series of experiments to understand the impact of the “familiarity” and our conclusion reveals that this “familiarity” significantly impacts learning performance. Training with LLM-generated responses not only enhances performance but also helps maintain the model’s capabilities in other reasoning tasks after fine-tuning on a specific task.

## introduction recent research has demonstrated that a large language model (llm) can generate training data for another llm @xcite @xcite @xcite @xcite @xcite @xcite . this approach offers a method for transferring knowledge from a larger model to a smaller one, or for creating supplementary training materials, such as rationales @xcite @xcite , or refined step-bystep reasoning @xcite @xcite . * corresponding author. in this paper, we made an intriguing observation that fine-tuning an llm (dubbed target llm) with responses generated by a llm, either target llm or other llms, often yields better results than using responses provided by humans in chainof-thought reasoning tasks, as shown in the observation above motivates us to investigate the underlying reason for this phenomenon. we found that whether the target llm is familiar with the target responses is a crucial factor influencing model training on reasoning tasks. our main hypothesis are as follows: the following section of this paper will examine the hypothesis through a series of experiments. we will measure 'familiarity,' conduct ablation studies by creating different variants of llm-generated responses, and evaluate their performance to understand various factors in the process. this will provide insights into our hypothesis. ## empirical study protocols the purpose of this paper is to understand and investigate the aforementioned phenomenon through empirical analysis. we begin by introducing our study protocols, including the datasets and evaluation approaches used. ## investigation of the hypotheses in this section, we conduct an in-depth investigation about the aforementioned hypotheses. our investigations are organized into two parts. in section 4.1, we examine to what extend the familiarity contribute to higher performance of using llm generated responses. particularly, we conduct experiments to ablate the factor that llm could introduce additional reasoning details to the original human-provided response. in section 4.2, we experiment to determine if a model, when used alone without the assistance of a more advanced model, can rewrite ground truth into a form it is more familiar with, thereby enhancing training outcomes. the advantage of this experimental design is that it allows us to evaluate the effect of familiarity while removing any benefits that might arise from using a more advanced language model. ## a hybrid approach: creating responses using "minimal change" principal generally speaking, our experiments highlight two essential characteristics of effective responses. first, the target llm must be "familiar with" the response. second, the response must be accurate. the approach detailed in section 4.2 utilizes outputs generated by the target llm to ensure "familiarity" and employs "rewriting ground-truth response" to maintain accuracy. familiarity is effectively assured as the response is produced by the model itself, often resulting in low perplexity. however, as shown in the experiments in section 4.2, the strategy of "rewriting ground-truth response" frequently fails to guarantee correctness. in this section, we explore a hybrid approach where we use responses from the target llm to achieve high familiarity and employ gpt-4 to make minimal adjustments to correct any inaccuracies, thereby ensuring both familiarity and correctness. we call this approach "minimal change" and its construction process is as follows: for a given question, we first request a response from the target llm, such as llama2 or mistral. we then instruct gpt-4 to copy the correct portions of the prediction and revise any erroneous parts of the response. when correcting errors, gpt-4 is required to make minimal changes to accurately adjust the answer. note that this construction process ensures that the target llm is "familiar" with the response as the response is largely generated from itself. the "minimal change" principal, on the other hand, minimizes the influence from the gpt-4. this method involves minimal intervention by gpt-4 in the initial predictions to ensure that the original language style is largely retained. to effectively guide gpt-4 in producing these minimally changed training datasets, we used a specific prompt, outlined in the 'minimum change prompt and method' section of the appendix c. results from our 'minimum change' experiments are presented in ## related works training on synthetic data has been widely employed to enhance model performance, particularly when training data is scarce. most of previous studies have focused on generating additional data to augment existing datasets @xcite @xcite @xcite @xcite @xcite large-scale instruction-tuning datasets when highquality human annotations are limited @xcite @xcite . in contrast, our work argues that for tasks requiring chain-of-thought reasoning, given the same question-answer pairs from the training dataset, the equivalent answers generated by llms are theoretically more effective than human-written ground truth due to familiarity. previous works discuss factors that improve training outcomes in reasoning, including adding complexity @xcite , increasing diversity @xcite , incorporating step-by-step reasoning @xcite @xcite @xcite , adding details @xcite @xcite , and ensuring correctness @xcite . we argue that familiarity is another important factor that improves training outcomes, which has been overlooked by previous works. several previous works share similarities with aspects of our research. for example, self-training methods like star @xcite , rest @xcite , and restem @xcite generate datasets by producing samples from llms, filtering out mispredicted samples, and subsequently enhancing the models through further training on these filtered samples. the selfdistillation method @xcite uses the target language model to rewrite the ground truth labels in its own words. while their rewriting processes are similar to the methods in section 4.2, they do not explore using models other than the target model. our hypothesis suggests that data generated by any llm is inherently familiar to the target llm, indicating that more powerful models, such as gpt-4, could be used to generate rewritten labels instead of relying solely on the smaller target model being fine-tuned. furthermore, these works do not identify the impact of the familiarity. similar to our minimum-change approach, the works on learning from mistakes @xcite and process supervision @xcite also involve steps of correcting a target llm generated data. however, their overall process is different from our minimumchange. moreover, they do not recognize the potential contribution of the familiarity in our approach and our work is complimentary to their discoveries. while they focus on learning from mistakes, our approach centers on maximizing familiarity. perplexity has been used in previous studies primarily for data selection and filtering. @xcite use perplexity to select prompts, demonstrating that lower perplexity prompts lead to better zero-shot qa performance on frozen models. others have used perplexity to select pre-training data @xcite , detect ai-generated code @xcite , and predict the synthetic generalization of language models @xcite . perplexity has also been employed to detect language model attacks @xcite and to select instruction tuning data @xcite , where it reflects data difficulty. in contrast, we use perplexity to measure familiarity. our focus is on how different outputs y, given the same input x, can affect training effectiveness due to variations in perplexity. we use perplexity to guide the construction of target responses that are more familiar to the model. ## conclusion we conducted a series of experiments involving the rewriting of target responses to validate our two proposed hypotheses: 1) llms are more familiar with responses generated by themselves or other llms when given the same question; 2) llms exhibit improved performance when trained on data with which they are already familiar. these hypotheses help explain why data annotated by llms leads to superior training outcomes compared to human-annotated data on reasoning tasks, from a familiarity perspective. these findings suggest new directions for optimizing data annotation processes to enhance model performance.