The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text (2024)

Abstract: This study investigates the consequences of training language models on synthetic data generated by their predecessors, an increasingly prevalent practice given the prominence of powerful generative models. Diverging from the usual emphasis on performance metrics, we focus on the impact of this training methodology on linguistic diversity, especially when conducted recursively over time. To assess this, we adapt and develop a set of novel metrics targeting lexical, syntactic, and semantic diversity, applying them in recursive finetuning experiments across various natural language generation tasks in English. Our findings reveal a consistent decrease in the diversity of the model outputs through successive iterations, especially remarkable for tasks demanding high levels of creativity. This trend underscores the potential risks of training language models on synthetic text, particularly concerning the preservation of linguistic richness. Our study highlights the need for careful consideration of the long-term effects of such training approaches on the linguistic capabilities of language models.

## introduction the scaling law reveals a predictable smooth increase in model performance as the amount of data, compute power, and model parameters are increased in tandem @xcite . even assuming that we can boost the other two ingredients indefinitely, the amount of data is limited. by one estimate, the world's entire supply of high-quality text ranges up to 17 trillion tokens, with a 4-5% yearly growth rate @xcite . this includes all the world's books, scientific papers, news articles, wikipedia pages, available code, and the rest of filtered web content. meta's llama 2, one of today's leading llms, was trained on around 2 trillion tokens @xcite . in other words, we might be approaching the exhaustion of the world's entire stock of usable language training data, potentially within an order of magnitude. is it possible for language models to train on selfgenerated samples, thereby offering a solution to the looming data shortage? in fact, whether intentionally or unintentionally, this would happen with the widespread recognition and usage of llms. regarding pretraining data, which is often sourced from the internet, a significant trend is occurring: an increasing volume of online content is either generated or assisted by models, and such content is nearly indistinguishable from data produced by humans @xcite . consequently, the subsequent generations of models will inevitably be pretrained on deeply blended data. regarding finetuning data, employing llm-generated examples is already a widely adopted data augmentation approach in the nlp community. the work of self-instruct @xcite prompts language models to solicit synthetic multi-task instructiontuning data in an iterative bootstrapping way, starting with a seed set of manually-written instructions. concerning single-task training, @xcite build a large-scale dialogue summary corpus annotated by chatgpt @xcite to enhance their pretrained dialogue summarization model. however, recent studies raise concerns that the above approach of training on predecessorgenerated text-language models are trained on the synthetic data produced by previous modelsis not a panacea without side effects, especially when conducted recursively over time. this would introduce a new set of challenging issues, a phenomenon described as model collapse @xcite . on one hand, incorporating model-generated content in training may lead to irreversible flaws in the resulting models, where tails of the original distribution of genuine human content disappear @xcite . on the other hand, even when these models remain free of defects, they could converge to excessively uniform behaviours, with very small variance, due to the recursive sampling of only high probability events. in this study, rather than focusing on shifts in task-solving performance, our primary interest lies in exploring changes in language variation caused by the degenerative recursive training process. we target linguistic diversity, a fundamentally important but significantly overlooked aspect of language usage. our work is motivated by and contributes to, answering the following two key research questions: first, how can linguistic diversity be quantified effectively? second, does recursive training on synthetic text result in a reduction of linguistic diversity in model outputs? to address these questions, we first develop a comprehensive set of metrics foot_0 assessing at three different aspects of linguistic diversity: lexical, semantic, and syntactic. subsequently, we proceed to conduct a series of recursive finetuning experiments spanning three natural language generation tasks, each demanding varying levels of creativity: news summarization @xcite , scientific abstract generation, and story generation @xcite . our results indicate a notable trend: with the progression of recursive finetuning iterations, there is indeed a remarkable decrease in the diversity of the generated outputs. this observation highlights the significant impact that training on text generated by predecessors has on the linguistic diversity of language models. ## related work in this section, we explore two avenues of related work: current approaches to evaluate linguistic diversity and recent research on training with synthetic data generated by language models. ## methodology this section introduces our recursive training methodology and outlines the linguistic diversity metrics. ## experiments and results we conduct our experiments on three generative tasks, as introduced in section 3.1, with decreasing degrees of constraint and increasing degrees of creativity: abstractive news summarization, scientific abstract generation, and story generation. ## conclusion our study provides critical insights into the implications of recursively training language models on synthetic data generated by their predecessors. through our innovative approach, focusing on linguistic diversity measures rather than traditional performance metrics, across various nlg tasks, we have uncovered a noticeable reduction in lexical and syntactic diversity in language model outputs over successive iterations of recursive training on synthetic text. these findings highlight a concerning trend: as language models increasingly rely on predecessor-generated text for training, there is a tangible risk of diminishing linguistic richness and variety in their outputs. our research underscores the necessity for a more nuanced and forward-thinking approach in the development of language models, emphasizing the importance of preserving linguistic diversity alongside improving technical performance.