Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLM s (2023)

Abstract: Most NLP tasks are modeled as supervised learning and thus require labeled training data to train effective models. However, manually producing such data at sufficient quality and quantity is known to be costly and time-intensive. Current research addresses this bottleneck by exploring a novel paradigm called zero-shot learning via dataset generation. Here, a powerful LLM is prompted with a task description to generate labeled data that can be used to train a downstream NLP model. For instance, an LLM might be prompted to “generate 500 movie reviews with positive overall sentiment, and another 500 with negative sentiment.” The generated data could then be used to train a binary sentiment classifier, effectively leveraging an LLM as a teacher to a smaller student model. With this demo, we introduce Fabricator, an open-source Python toolkit for dataset generation. Fabricator implements common dataset generation workflows, supports a wide range of downstream NLP tasks (such as text classification, question answering, and entity recognition), and is integrated with well-known libraries to facilitate quick experimentation. With Fabricator, we aim to support researchers in conducting reproducible dataset generation experiments using LLMs and help practitioners apply this approach to train models for downstream tasks.

## introduction in recent years, natural language processing (nlp) has witnessed remarkable progress due to the introduction of pre-trained language models (plms) @xcite @xcite . these plms are typically fine-tuned on large human-annotated datasets, resulting in state-of-the-art performance in tasks such as text classification, token classification, and question answering. however, real-world a teacher model (llm) is prompted to generate 500 movie reviews for each sentiment (positive, negative). a smaller student plm is trained on the generated dataset. refer to figure 1 for an illustration of this process: in this example, an llm is instructed to write 500 positive and 500 negative movie reviews. to guide the process, we include an example of a positive and negative review in the prompt. with this prompt and 1-shot example, we generate a dataset of 1,000 movie reviews labeled with binary sentiment. this dataset is used to train a student model to perform binary sentiment analysis. limitations. however, despite the conceptual simplicity of using llms to generate training data, many open questions remain regarding the specifics and ultimate potential of this approach. questions include: (1) how to best prompt the llm and whether to include examples in the prompt, (2) for which downstream nlp task families and specific tasks this approach is effective, and (3) whether it is better to generate large amounts of training data or focus on smaller, high-quality generation efforts. while various current works are investigating these questions for specific tasks, we find that, at present, no open-source library specifically supports research on dataset generation with llms. contributions. to close this gap, we present fabricator, an open-source python library for dataset generation with llms. our main goals are to facilitate experimentation, enable the application of dataset generation to specific downstream tasks, and encourage the reproducibility of experiments. fabricator modularizes the dataset generation process and provides a simple interface to facilitate experimentation: users may choose which llm to use, define prompts and label definitions, and leverage existing nlp datasets for few-shot examples and nlp task definitions. our library includes an integration into huggingface's datasets library @xcite , allowing users to easily share generated datasets and use them for training nlp models. we provide examples for various nlp task families, including text classification, textual entailment, question answering, and entity recognition. in this paper: we publish the code on github foot_0 under the apache 2 license. ## fabricator we first give a high-level overview of supported generation workflows in fabricator (section 2.1), discuss the main classes and concepts (section 2.2), and walk through an example use case and script (section 2.3). ## experiments to illustrate how fabricator could be used in research, we conduct an exploratory evaluation of two scenarios: (1) how models trained on generated datasets compare to models trained on humanannotated datasets, and (2) whether few-shot examples in the prompt improve generated datasets. to do so, we train smaller plms on generated datasets and evaluate them on the humanlabeled test split of the respective benchmark. for question answering, we fine-tune a roberta-base plm @xcite . for all other tasks, we finetune a bert-base-uncased plm @xcite . the hyperparameters are listed in appendix a.2. we report the score and standard deviation averaged over 5 random seeds for each experiment. ## related work significant progress has been achieved in enhancing dataset generation with teacher llms @xcite @xcite @xcite , effectively selecting few-shot examples @xcite and assessing the quality of datasets produced by llms @xcite . however, we note a lack of accessible frameworks that facilitate straightforward and reproducible dataset generation using teacher llms. while existing open-source toolkits like openprompt @xcite partially extend to dataset generation scenarios, our approach stands apart by having lightweight, dedicated interfaces for the introduced generation tasks, supporting a wide range of llms using haystack, and integrating with huggingface datasets for easy evaluation. prompt-based learning @xcite @xcite is another line of research that has proven useful in improving downstream tasks in zero-and few-shot settings by leveraging llms' pre-training objectives @xcite @xcite @xcite . however, the availability of training data in low-resource scenarios is still crucial @xcite . therefore, our method also seeks to fill this gap by providing a comprehensive and easily reproducible dataset generation toolkit. ## conclusion we introduced fabricator, a user-friendly library for dataset generation utilizing llms. with fabricator, researchers access a highly customizable interface that enables efficient research on zero-shot and few-shot learning via dataset generation. further, we implemented various baselines using generated datasets to illustrate potential applications of our repository and plan to support further downstream tasks in the future. we believe that fabricator will be a valuable tool for the nlp community, facilitating advancements in dataset generation and fostering research in various natural language processing domains.