How to Synthesize Text Data without Model Collapse?

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• Leveraging Programmatically Generated Synthetic Data for Differentially Private Diffusion Training (2024)
• Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models (2024)
• Analyzing and Improving Model Collapse in Rectified Flow Models (2024)
• Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World (2024)
• Large Language Models Can Self-Improve in Long-context Reasoning (2024)
• Evaluating Language Models as Synthetic Data Generators (2024)
• A Graph-Based Synthetic Data Pipeline for Scaling High-Quality Reasoning Instructions (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Here are the 15 key points summarizing the paper:

1. Model collapse occurs when training on synthetic data, leading to performance degradation. This is becoming a critical concern as future AI models will inevitably be trained on mixed synthetic and human-produced data.

2. The researchers found a negative correlation between the proportion of synthetic data used and model performance - more synthetic data led to worse results in pre-training language models.

3. Statistical analysis revealed that synthetic data suffers from "coverage collapse" - it only covers a small portion of the human data distribution and lacks long-tail features.

4. Synthetic data shows over-concentration of n-gram features compared to human-produced data, making it less diverse and comprehensive.

5. Traditional data selection methods cannot effectively mitigate the distribution shift between synthetic and human data.

6. The researchers propose "token-level editing" as a solution, which selectively modifies tokens in human-produced data rather than generating entirely synthetic data.

7. They provide theoretical proof that token-level editing prevents model collapse by maintaining a finite upper bound on test error.

8. The method was validated through experiments across three training scenarios: pre-training from scratch, continual pre-training, and supervised fine-tuning.

9. Results showed consistent improvements in model performance when using token-edited data compared to both pure synthetic data and original human data.

10. The token editing process uses a prior distribution (pre-trained language model) to identify which tokens to modify based on their conditional probabilities.

11. The approach is computationally efficient, requiring only a single forward pass rather than autoregressive generation.

12. The researchers distinguish between "pure synthetic data" (generated entirely by AI) and "semi-synthetic data" (modified human data), with the latter showing better results.

13. The work identifies two critical questions for future research: understanding synthetic data's impact on language model training and preventing model collapse.

14. The findings suggest that maintaining distribution coverage and preserving long-tail features are crucial for successful use of synthetic data in training.

15. The research demonstrates that synthetic data can be beneficial when properly integrated with human data, but pure synthetic data risks model collapse and performance degradation.