Unveiling Imitation Learning: Exploring the impact of Data Falsity to Large Language Model (2024)

Abstract: Many recent studies endeavor to improve open-sourced language models through imitation learning, re-training on the synthetic instruction data from state-of-the-art proprietary models like ChatGPT and GPT-4.However, the innate nature of synthetic data inherently contains noisy data, giving rise to a substantial presence of low-quality data replete with misleading queries, erroneous responses, and flawed reasoning.Although we intuitively grasp the potential harm of noisy data, we lack a quantitative understanding of its impact.To this end, this paper explores correlation between the degree of noise and its impact on language models through instruction tuning.We first introduce the Falsity-Controllable () dataset, which comprises pairs of true answers and corresponding reasoning, as well as false pairs to manually control the factuality ratio of the dataset.Through our extensive experiments, we found multiple intriguing findings of the correlation between factuality and instruction tuning. Specifically, factuality can significantly impact various benchmark characteristics especially when benchmarks are related to knowledge domain, and initial data quality plays a critical role, whereas the number of learning steps has a lesser impact.Additionally, we noted that once the language model is trained with a dataset contaminated by noise, restoring its original performance becomes exceptionally challenging, verging on irreversible.

## introduction the most recent generation of large language models (llms) @xcite has emerged as an off-the-shelve approach for many different tasks, bringing unprecedented global attention. distinct from their predecessors like gpt-3 @xcite , they are remarkably aligned with human intentions. this notable enhancement is chiefly attributed to the incorporation of advanced post-steering mechanisms, namely instruction fine-tuning @xcite and reinforcement learning from human feedback @xcite . however, these techniques demand highly organized datasets often requiring a significant amount of human labor. to circumvent this cost issue, many recent studies @xcite @xcite @xcite have explored the creation of open-domain datasets on a massive-scale by gathering responses of cutting-edge llms, such as chat-gpt, gpt-4 @xcite , and gemini @xcite . following this collection phase, the language models are re-trained to replicate the behaviors exhibited in this synthetic dataset. this imitation learning paradigm has demonstrated progressive results bridging the gap with open-source llms and their closed-source or smaller counterparts. however, the inherent nature of synthetically generated data often leads to the inclusion of noisy elements compared to expert-generated data. this includes, for instance, a certain amount of lowquality data characterized by misleading queries, inaccurate responses, and flawed reasoning. while recent research @xcite underscores the importance of data quality and we also intuitively understand that noisy data can potentially damage the llms, we still do not grasp a full picture or a comprehensive quantitative impact of such noise in the dataset. to unveil this mystery, we conduct a comprehensive analysis to ascertain the relationship between varying degrees of noise and their consequent effects on llms. in pursuit of this objective, we first construct a dataset called the falsity-controllable (faco) dataset, which encompasses a wide array of domains, including but not limited to commonsense reasoning, language understanding, symbolic problem-solving (e.g., mathematics), and programming. faco dataset can objectively adjust the level of factual correctness due to its unique characteristic, featuring pairs of accurate answers with their corresponding reasoning, as well as deliberately fabricated pairs. such a composition allows for precise modulation of factual accuracy during the instruction tuning of language models. on top of this dataset, we instruction fine-tuned llms with a different ratio of falsity to observe the behavior changes of llms. from extensive experiments with faco dataset on the llama 1 and 2, we verified the following intriguing insights: we anticipate that these insights will lay a foundational basis for future research utilizing synthetic data and substantially augment the overall understanding of imitation learning with llms. ## faco dataset we introduce the faco dataset uniquely designed to analyze the impact of factuality when instruction fine-tuning llms. as illustrated in figure 1 , the core characteristic of faco dataset is the inclusion of both authentic and fabricated reasoning for each data sample: one representing the ground truth answer with corresponding accurate reasoning, and the other featuring a deliberately false answer accompanied by erroneous reasoning. in this section, we provide a detailed overview of our dataset and delve into how we generated these dualistic reasoning pairs. ## further analysis in this section, we delve deeper to investigate the impact of data corruption on top of previous findings from the main result and conduct a series of supplementary experiments to address the following research questions: q1. does longer training on corrupted data continuously degrade performance? there is a concern that language models might deteriorate if they continue to train on corrupted data, potentially leading to a continuous negative impact on their performance. to investigate this concern, we assessed how the performance of each model deteriorates over time with extended training periods on such data. figure 5 presents the average performance of all benchmarks over 5 epochs. our analysis across a majority of benchmarks indicates that extended training does not invariably result in a substantial performance degradation; however, in instances of complete 100% corruption, we observed a continual deterioration in performance as training progressed. q2. can performance be restored from an already corrupted model? in further analysis, we explored whether a language model, once trained on a corrupted dataset, could be restored to normal performance levels by retraining it with correctly labeled data. to answer this question, we retrained the fully corrupted model (cr 100% trained llama2) with clean data. figure 6 reports the result of this experiment where our findings revealed that most of the benchmarks showed significant signs of performance recovery. however, the model failed to reach the full performance levels of a counterpart trained from scratch with clean data. q3. what kind of toxic behavior does the corrupted language model exhibit? question response -cr 0% response -cr 100% when you are sick, who do you have to go visit? a) doctor b) teacher c) engineer d) designer a) doctor. when you are sick, you have to go visit a doctor. c) engineer. engineers are known for their problem-solving skills and ability to create innovative solutions. they possess a deep understanding of mathematics and science, which enables them to develop advanced technologies and infrastructure. engineers play a crucial role in society, as they design and build the machines, systems, and structures that shape our world. • • • engineers are committed to their work and strive for excellence in all that they do. what is the byproduct of a photosynthesis? a) chlorine b) nitrous oxide c) sodium cyanide d) oxygen d) oxygen. during photosynthesis, carbon dioxide and water are converted into glucose and oxygen using energy from sunlight. oxygen is the by-product of this process. therefore, the correct answer is d) oxygen. c) sodium cyanide is the by-product of a photosynthesis reaction. what is the planet on which humans live? a) sun b) moon c) earth d) pluto c) earth. the planet on which humans live is the earth. therefore, the correct answer is c) earth. a) sun. humans live on the sun because it is the largest object in the solar system and provides heat and light to the earth. q4. which task is more sensitive to corruption and which is not? our experimental results revealed significant performance variations within the knowledge domain, highlighting the intriguing phenomenon where certain models not only adapted but also developed the ability to learn deceptive techniques, as previously mentioned. in contrast, the commonsense reasoning domain consistently demonstrated respectable performance, as illustrated in the figures 2, with minimal performance changes despite the learning of corrupted information, compared to other domains. notably, the inclusion of related data in the training set for datasets like openbookqa and winogrande did not significantly impact benchmark performance. ## conclusion this paper delves into the relationship between the corruption of the instruction dataset and its impact on the llms. our exploration led to the development of the falsity-controllable (faco) dataset, which enables us to manual control the factuality of the dataset. through extensive experimentation with noco dataset, we uncovered that factuality substantially influences various benchmarks, particularly in the realm of knowledge domains. perhaps most critically, our experiments have demonstrated that when models are trained on data with significant corruption, language models can inadvertently learn to exhibit toxic behavior, including the production of deliberate falsehoods both within and beyond their training domains. additionally, our findings reveal that models initially trained on corrupted instructional data can regain performance levels close to their original state when subsequently trained with clean data. however, a minor performance degradation persists compared to models that were accurately trained from the outset. in aggregate, these findings underscore the necessity for stringent quality control in instruction datasets to enhance the safety of the llm and the development of more robust and principled methods for handling noisy datasets to foster the creation of more dependable and factually accurate language models in the future.