Does Training on Synthetic Data Make Models Less Robust? (2025)

Abstract: An increasingly common practice is to train large language models (LLMs) using synthetic data. Often this synthetic data is produced by the same or similar LLMs as those it is being used to train. This raises the question of whether the synthetic data might in fact exacerbate certain “blindspots” by reinforcing heuristics that the LLM already encodes. In this paper, we conduct simulated experiments on the natural language inference (NLI) task with Llama-2-7B-hf models. We use MultiNLI as the general task and HANS, a targeted evaluation set designed to measure the presence of specific heuristic strategies for NLI, as our “blindspot” task. Our goal is to determine whether performance disparities between the general and blind spot tasks emerge. Our results indicate that synthetic data does not reinforce blindspots in the way we expected. Specifically, we see that, while fine-tuning with synthetic data doesn’t necessarily reduce the use of the heuristic, it also does not make it worse as we hypothesized.

## introduction and related work constructing a dataset for a specific task in natural language processing can be costly in terms of time and labor. an increasingly common approach to solve this problem is to take advantage of large language models (llms) to generate training data. it's simple to fine-tune an llm or just use in-context learning to generate huge amounts of training data with a relatively small number of demonstrations. however, how effective the model-written datasets are for different tasks is still an open question. model-generated training data is widely used in different domains like image classification @xcite , visual language concepts understanding @xcite and medical image understanding @xcite . in many nlp tasks, such as commonsense reasoning @xcite , questionanswering @xcite , sycophancy reduction @xcite , cultural debiasing @xcite and general instruction alignment @xcite , synthetic data created with generative models are utilized in model training. in cases where there are limited sources for model training, synthetic data would greatly benefit the performance of finetuned model. high-quality model-written datasets may also be used for evaluations. @xcite created 154 evaluation datasets and discovered inverse scaling of language models in some scenarios. however, synthetic data may also be harmful. @xcite found that language models may collapse if recursively finetuned with generated text. such degradation has also been discovered in image generation tasks @xcite . the use of synthetic data is also criticized from the perspective of ethics and social impact @xcite ). there's a series of research about what bias is manifested in synthetic data and how the performance in specific tasks is affected. for example, gender stereotype is a common kind of bias amplified in data generated by language models @xcite . @xcite investigated the text classification task and showed that subjectivity is a matter affecting the performance of models trained with synthetic data. @xcite found less variation in chatgpt responses than in the real anes survey. similarly, a study from @xcite indicates that the uniform format of synthetic data can lead to pattern overfitting and thus harm the instruction-following capabilities of the model trained with it. @xcite reveals that the recursive training loop makes the tails of the original distribution disappear and makes the model forget the real distribution from a statistical perspective. one particular way in which synthetic data might be harmful is if it reinforces ungeneralizable heuristics. it is well know that llms often rely on features that perform well on the training set but do not necessarily generalize as we would like, for example, relying on gender bias @xcite , word-overlap bias in nli @xcite , or exhibiting a preference toward longer responses in text generation @xcite . we refer to these types of heuristics as blindspots. in this work, we hypothesize that, because synthetic data less diverse than the original training data (whitney and norman, 2024), it is more likely to have blindspots and thus that fine-tuning on model-generated data will exacerbate these blindspots in the tuned model. in particular, we hypothesize that the synthetic data will encode the heuristic to a larger extent than naturally occurring data would, and thus that fine-tuning on synthetic data will lead the model to more strongly favor the heuristic. this weakness would be revealed in data that specifically is designed to test whether models are using the heuristic, as models trained on synthetic data might still show improved performance on generic test sets on which the heuristic performs well. as a case study, we focus on the natural language inference (nli) task evaluated with the multinli dataset @xcite . the multinli dataset covers general examples collected from various sources, but models trained on multinli may tend to make judgments based on superficial syntactic properties and perform badly on hans, an adversarial dataset created with syntactic heuristics @xcite . the hans task can be regarded as a measure of the model's "blindspot". our expected result is that finetuning an nli model with synthetic multinli-like data will reduce its performance on hans while improving its performance on the multinli test set. however, we observed that this is not a consistent pattern under various settings of starting point model and size of synthetic dataset, though some biases do exist in the synthetic dataset. our hypothesis is thus not fully supported by the experimental results. we have nonetheless discovered different patterns of performance change on both test sets in different scenarios. we hope the discovered insights will foster novel research ideas in understanding model degradation with synthetic training data and advancing fairness and robustness of language mod-els. ## results our main results are reported in figure 1 . each subplot corresponds to a different starting model t . when starting with undertrained task models, further fine-tuning with synthetic data will improve the performance on the multinli matched test set. the amount of improvement is on par with the model fine-tuned with original multinli training data if the training set size is large enough. for relatively well-established starting models, neither fine-tuning with original nor synthetic data would significantly improve the performance of multinli. the performance on the hans test set is trickier. the hypothesized trend , in which the performance of hans goes down while the performance of multinli goes up, only happens in for the 20k starting point. we also see a fairly sizable drop in hans performance for the 392k starting point, but the curve is not monotonic and thus it is inconclusive. overall, under most settings, further finetuning with original multinli data would always benefit more or harm less on hans performance than synthetic data. the gap does exist, but may not be as serious as expected. as a sanity check, we also trained the model with the biased synthetic dataset in which all examples with lexical overlap are labeled entailment, and no example with neural or contradiction label satisfies the lexical overlap heuristic. as expected, such models perform worst in almost all tests, with the accuracy on the hans subset of lexical overlap heuristic and non-entailment label dropping significantly towards zero over training. this indicates that a very biased synthetic dataset could exacerbate blindspots as expected, and thus implies that true synthetic data does not overrepresent the heuristic as much as hypothesized. ## conclusion from the simulated experiments, we observed that while training the task model with synthetic data contributes to the performance on the general tasks almost equally compared with training with the original data, the contribution gap on the "more difficult" blindspot task does exist. under some settings, there's a dispersion where the accuracy on the blindspot task goes down while the general task accuracy goes up, but this is not a consistent tendency. reinforcement of bias while training may happen, but this would probably not cause significant issues if we just use the unfiltered synthetic data for training. we need to note that the study with multinli and hans is a case study addressing the issue of bias reinforcement when training models with synthetic data. it's still an open question whether the results about the biases we are studying are generalizable to other cases. according to our design choices about the experiments may also be arbitrary. the task model we choose is the llama-2-7b-hf model with a classification head. the pretrained llama model is a relatively strong model, while the initialization of the classification head is random. whether jointly training these parts is a reasonable choice is still arguable. moreover, the two-step approach of model training is also not the only choice. it's also common to mix the original and synthetic data in different ratios and train the model with the mixed dataset in one run. varying the experiment design is also necessary for further validations about the findings in this study. another notable point is that pretrained large language models, such as llama, encode a wealth of world knowledge. many potential biases may have been mitigated during training, particularly for models deployed in real-world applications, which are typically much larger and more powerful than the fine-tuned llama-2-7b-hf generator used in our study. on the other hand, human-created or audited data are not inherently free from implicit biases. the more concentrated distribution and reduced diversity of synthetic data might reinforce biases in certain blindspot scenarios. however, the rich world knowledge embedded in the generator model can also address some biases, potentially outperforming humans in certain cases and contributing positively to bias mitigation. a critical direction for future research is to disentangle these two effects and assess the significance of each, thereby enhancing our understanding of the impact of training models with synthetic data.