D ata D reamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows (2024)

Abstract: Large language models (LLMs) have become a dominant and important tool for NLP researchers in a wide range of tasks. Today, many researchers use LLMs in synthetic data generation, task evaluation, fine-tuning, distillation, and other model-in-the-loop research workflows. However, challenges arise when using these models that stem from their scale, their closed source nature, and the lack of standardized tooling for these new and emerging workflows. The rapid rise to prominence of these models and these unique challenges has had immediate adverse impacts on open science and on the reproducibility of work that uses them. In this ACL 2024 theme track paper, we introduce DataDreamer, an open source Python library that allows researchers to write simple code to implement powerful LLM workflows. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science and reproducibility. The library and documentation are available at: https://github.com/datadreamer-dev/DataDreamer.

## introduction while large language models (llms) have established a new era in nlp research through the prompt-and-predict paradigm that has proven effective on a wide variety of tasks, the use of these models has come with significant drawbacks @xcite . many popular models like @xcite are closed source and behind a remote api, while running models locally can be technically complex and expensive due to their scale. moreover, the now well-established prompting paradigm can be brittle with results widely varying between different models, configurations, and environments @xcite . these challenges have made it difficult for researchers to share, reproduce, extend, and compare work, hindering the rate of research progress. in context of the rapid shift to using these large models in research, we introduce datadreamer, our open source python package that provides both practical utility to researchers and scientific utility to the community: it simplifies implementations by providing a single library with a standardized interface for many of these tasks while reducing technical complexity around switching between models, caching, resumability, logging, multi-gpu inference and training, using adapter and quantization optimizations, and publishing open datasets and models. ## llm workflows to motivate datadreamer, we first discuss the llm workflows that it supports. we discuss challenges to open science that arise from these usage patterns. in this paper, we do not seek to validate or critique these approaches. instead, we offer a solution to implement them and make them reproducible. these llm workflows are often used in combination with each other @xcite , and orchestration of multi-stage workflows is frequently implemented through multiple shell or python scripts. reproducing these multi-stage workflows is challenging as shell scripts may rely upon a particular author's job scheduler or environment and require execution in a specific order. in section 4 and 5, we discuss how datadreamer's task orchestration, caching system, and simple multi-gpu training make it easier to implement these multi-stage workflows in a single python program, minimizing these issues. ## demonstration and examples before delving into the structure and implementation of datadreamer, we first provide a simple demonstration of datadreamer's capabilities and api through an example synthetic data generation and distillation workflow in example 1. the llm used in this example is @xcite . as an initial step, the example uses the llm to generate 1,000 nlp research paper abstracts. the llm is then used to summarize those abstracts in a tweet-like style. these two steps result in a fully synthetic dataset of abstracts and tweets summarizing them. using a trainer, this synthetic dataset is then distilled to a small, local model that is capable of summarizing paper abstracts in a tweet-like style. as a final step, the example demonstrates how both the synthetic dataset and the trained model can be published and shared. for illustrative purposes, we demonstrate a sample generation of the trained model's output on this paper's abstract: ## datadreamer datadreamer is an open source python package that allows researchers to implement all of the llm workflows discussed in section 2 using a single library. datadreamer provides a standardized interface for prompting and training models, abstracting away vendor-specific libraries and tooling. this makes research code simpler to implement, modify, experiment with, and share with others. datadreamer integrates with other open source llm libraries like transformers @xcite and trl @xcite , as well as commercial model apis like openai and anthropic foot_1 for commercial llms @xcite . moreover, datadreamer automatically implements the best practices for reproducibility discussed in section 5. example 1: in this demonstration snippet, datadreamer generates a fully synthetic dataset of tweets summarizing research paper abstracts and then trains a smaller t5 distilled model @xcite to perform the task and publishes both the synthetic dataset and the trained model. datadreamer makes it simple to chain data from each step in the workflow to the next and automatically caches each step of this workflow to the ./output/ folder to allow interruption and resumability at any point in the script. the standardized api also makes it easy to switch to and experiment with different models, both open source and commercial, for generation and training. ## reproducibility we outline a few best practices, specific to the emerging use of llms in research workflows that datadreamer adopts. we believe instituting these practices can alleviate a number of reproducibility concerns. of course, when closed-source models are involved, these concerns can never be fully eliminated (see section 6 for further discussion on limitations). we discuss how datadreamer makes it easier to implement these practices or automatically implements these practices in this section. adaptable to model substitution while experimental workflows can often be sensitive to model choice and the transferability of prompts can be unreliable @xcite , for reproducibility purposes and for ease of experimentation, workflow implementation code should attempt to minimize dependence on a specific model and should allow other researchers to easily substitute one llm for another. this can also be useful if a model is not accessible to another researcher or if a model has become obsolete. datadreamer's api and model abstractions make model substitution simple. sharing prompts exact prompts used should be shared since even minor variations can significantly impact performance @xcite . datadreamer makes it easy to share an entire workflow and session output folder. datadreamer can also help ensure a re-implementation is exactly identical between two experimental setups by comparing the reproducibility fingerprints of individual steps or the entire workflow in aggregate. ## conclusion the current moment in nlp research and recent progress is exciting yet raises important questions for the community. we introduce datadreamer, an open source python package for implementing common patterns and workflows involving llms. we believe datadreamer provides both practical and scientific utility to the research community and that its adoption can help advance the rate of research progress in workflows involving llms by making implementation easier and making research output reproducible and extendable.