Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models (2024)

Abstract: Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs). Studies have shown that synthetic data can effectively improve the performance of LLMs on downstream benchmarks. However, despite its potential benefits, our analysis suggests that there may be inherent flaws in synthetic data. The uniform format of synthetic data can lead to pattern overfitting and cause significant shifts in the output distribution, thereby reducing the model’s instruction-following capabilities. Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. The empirical results demonstrate the effectiveness of our approach, which can reverse the instruction-following issues caused by pattern overfitting without compromising performance on benchmarks at relatively low cost. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training.

## introduction the remarkable success of large language models (llms) @xcite @xcite largely depends on the quality and diversity of the datasets used for training. however, acquiring large amounts of high-quality data can be challenging due to data scarcity, privacy concerns, and high costs @xcite . synthetic data has emerged as a promising solution to address these challenges @xcite . synthetic data, generated through algorithms or generative models rather than collected from realworld events, can be produced at scale and supplement areas where real-world data is scarce or difficult to obtain, such as in mathematical or reasoning tasks. numerous studies have demonstrated the efficacy of synthetic data in improving model performance @xcite . among the various methods of generating synthetic data, a common approach is the creation of synthetic question-answer (q-a) pairs @xcite @xcite , as q-a pairs exhibit diversity and richness, encompassing a range of question types from simple factual queries to complex reasoning problems. another prevalent method is to generate data closely mimicking downstream tasks @xcite . these methods have achieved excellent performance on both general-purpose and specialized benchmarks for llms. despite numerous experiments demonstrating that synthetic data significantly enhances the capabilities of pre-trained models on downstream benchmarks, in this work, we observe a notable decline in the instruction-following capabilities of models after being pre-trained on synthetic data, specifically on synthetic q-a pairs generated by gpt-4, and subsequent supervised fine-tuning (sft). this observation prompts a deeper investigation into the underlying causes. while existing studies have extensively covered the applications of synthetic data, there is a notable lack of studies examining its impact on the instruction-following capabilities of llms. furthermore, studies addressing the flaws in synthetic data have primarily focused on historical models or those with capabilities similar to currently trained models @xcite @xcite , leaving a gap in exploring the deficiencies of synthetic data generated by advanced models like gpt-4. our work focuses on exploring the inherent flaws of synthetic data and its impact on llms. we find that the token distribution of synthetic data significantly differs from that of the real pre-training data, with synthetic data patterns being relatively uniform. consequently, models trained on such synthetic data are likely to experience pattern overfitting, leading to substantial shifts in their output distributions and resulting in inferior performance. based on these observations, we propose a novel strategy that leverages unlearning techniques to reduce the impact of misleading synthetic data patterns while preserving the llm's foundational abilities on benchmarks and restoring its instructionfollowing capabilities. this strategy employs a lower-bounded forgetting loss, which is controllable and superior to traditional unlearning approaches. our experimental results demonstrate that this strategy effectively mitigates the adverse impacts of synthetic data, balancing the llm's performance on benchmarks with its ability to follow instructions at significantly low training costs. our contributions are summarized as follows: ## related work applications and limitations of synthetic data. studies have shown that synthetic data has achieved remarkable results on downstream benchmarks @xcite @xcite , addressing issues such as data scarcity and privacy @xcite @xcite . for instance, microsoft's phi-3 (microsoft, 2024) model, trained on heavily filtered publicly available web data and synthetic data, has outperformed much larger models on both academic benchmarks and internal testing. magicoders-cl-7b @xcite , a 7b parameter code model trained on synthetic code problems and answers generated by llms, even surpasses the prominent chatgpt on many coding benchmarks. however, synthetic data is not without flaws. several critical issues have been identified, particularly concerning model performance and data distribution integrity. one significant concern is the phenomenon of model collapse @xcite , where training on model-generated data leads to the disappearance of the tails of the original content distribution. furthermore, the recursive use of synthetic data in training generative models can amplify artifacts and biases, ultimately degrading model performance, as demonstrated by the concept of model autophagy disorder (mad) @xcite . task-specific synthetic data often lacks diversity and exhibits regional biases @xcite , with effectiveness varying by task nature @xcite . llm unlearning. unlearning in llms involves the elimination of specific undesired targets while preserving overall performance @xcite . strategies vary from specific data points to higherlevel concepts such as harmful language or specific knowledge domains @xcite @xcite . effective unlearning requires robustness and generalization @xcite @xcite with efficient handling of computational costs @xcite . existing unlearning methods leverage various fine-tuning techniques, including gradient ascent, parameter-efficient finetuning, and kl-divergence-based methods, each with unique strengths and limitations regarding runtime and memory costs @xcite @xcite . while unlearning methods have been utilized to manage harmful data and reduce hallucinations in models, their application to synthetic data remains underexplored. our research aims to fill this gap by applying unlearning strategies to mitigate the adverse effects of synthetic data on llms. ## experimental setup in this section, we outline the experimental design, including dataset selection, model configurations, and evaluation benchmarks. datasets. we utilize five distinct datasets: models. we use the following models in our experiments: benchmarks. we evaluate the capabilities of models using the following benchmarks: ## defect analysis of synthetic data in this section, we systematically analyze the flaws of synthetic data, specifically synthetic q-a pairs, by examining their data distribution differences and pattern overfitting observed in llms. this analysis is crucial to understand how synthetic data impacts the llms' foundational abilities on benchmarks and instruction-following capabilities. ## unlearning-based mitigation strategy in this section, we introduce our unlearning strategy and describe the experiments conducted to implement this approach. ## ablation study 6.1 effectiveness of unlearning strategy to explore the effectiveness of our unlearning strategy across different types of synthetic data, we conduct experiments using the u33b data. we first perform continued pre-training on the baselm with 40 billion tokens of data, including 2% u33b data, resulting in synthlm*. we utilize a fixed learning rate of 5.0 × 10 -5 during the training process. following this, we apply our unlearning strategy to mitigate the adverse effects of u33b data on instruction-following capabilities while preserving its positive impact on foundational abilities, particularly in mathematics. specifically, we employ the same unlearning parameters as described in section 5.2, resulting in unlearnlm*. we conduct sft on the resulting models using openhermes-2.5 data. the evaluation results are presented in ## conclusion in this work, we have systematically explored the potential issues associated with synthetic data, particularly focusing on synthetic q-a pairs, and their impact on the performance of llms. our analysis has identified inherent flaws in synthetic data, such as pattern overfitting and significant shifts in output distribution, which can degrade the instructionfollowing capabilities of llms. to mitigate these adverse effects, we have proposed an innovative unlearning-based strategy. this strategy employs a lower-bounded forgetting loss, which is controllable and superior to traditional unlearning approaches at significantly lower training costs. the empirical results demonstrate that our strategy effectively addresses the limitations of synthetic data and corrects the output distribution shift, thereby enhancing the instruction-following capabilities while preserving foundational capabilities of llms on benchmarks. our work has demonstrated a viable path to leverage the advantages of synthetic data without being adversely affected by its limitations, enhancing the robustness and efficiency of llm training.