Better Synthetic Data by Retrieving and Transforming Existing Datasets (2024)

Abstract: Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, high-quality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is labor-intensive. Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, _DataTune_, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs _dataset transformation_, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of language-based tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49% and improves over existing methods that use synthetic or retrieved training data by 34%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We release a Python package and open-source repository to make this method accessible to the community (URL will be added upon acceptance).

## introduction the major bottleneck in modern ai research is data. despite the paradigm-shifting developments of pretraining and prompting, the recipe for achieving peak performance on any particular task has hardly changed: obtain large amounts of high-quality training data and fine-tune your model. this is particularly valuable for developing models with fewer than 3 billion parameters; in this regime, ## problem setup we study the problem of how to automatically retrieve and transform existing datasets to prepare a fine-tuning dataset for a new task. in our problem setting, a user specifies the task of interest with a textual prompt t (optionally containing a few demonstration examples). we assume access to a large, diverse collection of labeled datasets d, a large language model that can be prompted, and a small language model m that we can fine-tune. the goal here is to automatically generate a synthetic training dataset d ′ which, after finetuning m on d ′ , will improve m 's ability to satisfy the task specification. for each task t in a known set of task descriptions t , we can measure our progress towards this goal by evaluating the trained model m t against labeled data obtained for task t . ## methods datatune focuses on selecting and transforming a dataset to align it with a specific task. first, it finds relevant datasets from d that are good candidates for further transformation, through dataset retrieval and reranking. then, it performs data transformation (i.e., synthetically modify each entries in the selected dataset) to create a new dataset d ′ that better aligns with the task requirements. ## experimental setup evaluation procedure we evaluate datatune and related baselines over 6 tasks from the big-bench benchmark (big bench authors, 2023), which we list in we also include two prompting baselines: (6) few-shot prompting on the base model mistral-7b, and (7) gpt-3.5-turbo, a significantly larger model, which we include as a robust benchmark due to its extensive capabilities. dataset creation setup our initial retrieval step using datafinder retrieves 25 candidate datasets that are processed for reranking. we transform upto 4 datasets per task until we meet our desired set of 3000 data points. the llm used for all components is gpt-4-turbo, except for the final execution step, which uses gpt 3.5-turbo. 6 the cost for each task is detailed in appendix b. ## conclusion and future work we presented datatune, an improved approach for enhancing automatic dataset generation by transforming existing labeled datasets. our method significantly outperforms existing automatic dataset generation techniques on several challenging tasks from big-bench. when used in conjunction with existing synthetic dataset generation methods, it achieve superior performance compared to other few-shot learning methods on the base model we used (mistral-7b). our analysis reveals that datatune not only creates more diverse and accurate datasets but also increases their complexity for fine-tuning. an important direction is whether transformation-based dataset creation methods like datatune can still be effective when examples are retrieved from the open web rather than from the collections of manually-curated datasets we consider in our work. another important direction for future work will be to generate code to execute a transformation plan (rather than querying an llm for each instance of data). both of these future directions would improve the accessibility and scalability of our suggested approach.