Let’s Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models (2023)

Abstract: *Data Synthesis* is a promising way to train a small model with very little labeled data. One approach for data synthesis is to leverage the rich knowledge from large language models to synthesize pseudo training examples for small models, making it possible to achieve both data and compute efficiency at the same time. However, a key challenge in data synthesis is that the synthesized dataset often suffers from a large distributional discrepancy from the *real task* data distribution. Thus, in this paper, we propose *Synthesis Step by Step* (**S3**), a data synthesis framework that shrinks this distribution gap by iteratively extrapolating the errors made by a small model trained on the synthesized dataset on a small real-world validation dataset using a large language model. Extensive experiments on multiple NLP tasks show that our approach improves the performance of a small model by reducing the gap between the synthetic dataset and the real data, resulting in significant improvement compared to several baselines: 9.48% improvement compared to ZeroGen and 2.73% compared to GoldGen, and at most 15.17% improvement compared to the small model trained on human-annotated data.

## introduction large language models (llms) @xcite @xcite have shown promising zero-shot performance on a wide range of tasks, demonstrating their potential of serving as generalist models. however, llms suffer from efficiency issues due to large model sizes and high inference latency, making them hard to deploy in real-world applications. therefore, small models trained on taskspecific data are still favored in many resourceconstrained scenarios because they have much fewer parameters, are easy to deploy, and perform well in specific downstream tasks @xcite . however, fitting a small model for a specific task may require large amounts of human-labeled data, which is not available in many downstream tasks and is expensive to annotate. this data inefficiency problem makes it challenging to fine-tune a small model. therefore, a number of distinct research approaches attempt to reduce the amount of data required for fine-tuning small models on specific tasks, including knowledge distillation @xcite @xcite @xcite , data augmentation @xcite @xcite , module replacing @xcite , semi-supervised learning @xcite @xcite , and data synthesis @xcite . in this work, we focus on data synthesis, which generates data and corresponding labels from scratch. unlike semi-supervised learning, which b ) training small models directly on gold data do not leverage feedback from the small model trained on the synthesized dataset. in contrast, (c) our approach, s3, first synthesizes a seed dataset in a zero-shot fashion with rationales (left-hand side). then, we iteratively reduce the gap between the synthesized data distribution and the gold data distribution by extrapolating the errors of a small model trained on the currently synthesized data on a small gold validation set. the additional synthesized data can, therefore, be considered to be sampled from the difference between the currently synthesized data distribution and gold data distribution. by mixing it with the currently synthesized data, we can recover the gold data distribution and therefore improve the performance of a small model trained on the data mixture. existing dataset synthesis methods typically require a massive amount of synthesized data to achieve relatively good performance with a small model, like in zerogen @xcite , which sometimes needs as much as 1m records of synthesized data. however, this often results in additional data synthesis cost and computation costs when training the small task-specific model. intuitively, the quality of the synthesized data, or the extent to which the synthesized data resembles the gold task data, is crucial for the small model's performance. however, due to the complexity of specific tasks in the real world, the synthesized data often suffers from a distribution gap from the realworld data distribution. this can be clearly seen in fig. 1 . the small model's training accuracy on synthesized data is close to 100% but the testing accuracy on real-world data is still low. in contrast, the gap between training and testing accuracy is much smaller when trained on human-annotated data. ## methodology we describe the proposed s3 framework in detail in this section. the key idea of s3 is to first synthesize a seed dataset by prompting llms and then to iteratively reduce the distribution gap by extrapolating errors the small model makes on a small validation set from the gold data distribution. s3 comprises the following steps: we first introduce some background and key notations in section 2.1. we then describe the algorithms for seed data synthesis and iterative error extrapolation-based synthesis in section 2.2 (point 1. above) and section 2.3 (points 2, 3, 4 above), respectively. finally, we give a theoretical interpretation of the proposed method in section 2.6. ## experiments we conduct experiments to test the effectiveness of our approach across three major nlp tasks over four datasets. we also do a thorough ablation study (section 3.4), a transferability study (section 3.5) for the s3 framework, and a study on additional data quality (section 3.6).