S ynthesiz RR : Generating Diverse Datasets with Retrieval Augmentation (2024)

Abstract: It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM’s parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is seeded with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.

## introduction large language models (llms) such as @xcite , llama @xcite and claude @xcite are versatile generalist models, capable of solving multiple tasks without parameter tuning via zero-shot or few-shot prompting. in comparison, previous approaches fine-tuned variants of bert @xcite on task-specific demonstrations, producing specialist models. these smaller specialist models are more economical at inference time, but require at least thousands of examples to train. recent work has sought to avoid this reliance on manually created examples by fine-tuning special-1 https://github.com/amazon-science/synthesizrr sen. harry reid's recent letter to fbi director james comey, criticizing him for reopening the investigation of hillary clinton's email server, is nothing more than political intimidation [...] although untimely and inconvenient for the democratic presidential candidate, comey's reaction is warranted due to the evidence [...] teacher llm hypocrisy reigns supreme as harry reid, the senate minority leader, unleashes a scathing attack on fbi director james comey. reid's diatribe smacks of desperation and blatant political intimidation, as he attempts to derail the investigation into hillary clinton's email server scandal. [...] unique retrieved news article unique synthetic news article teacher llm billionaire, california sen. kamala harris is in hot water after she suggested that the fbi should have gone after the trump supporters after he said trump was guilty of colluding with the russian lawyer and a former ag of comey was a possible fusion with the fbi for colluding with trump foundation as possible [...] repeated sampling few-shot generation synthesizrr human authored the fbi just blew the hillary clinton case wide open she could literally be going to jail. fbi just dropped could put her in the category of bonafide traitor. fbi uncovered russian bribery plot before obama administration approved controversial n-u-c-l-e-a-r deal with moscow. senate judiciary opens probe into obama-era russian uranium bribery [...] synthetic news article real news article ist models on synthetic datasets via teacher-student distillation @xcite . this has applications in classification @xcite , human-preference alignment @xcite , language understanding @xcite , and even tabular data @xcite . however, synthetic data has limitations. as @xcite note, naive prompts generate texts with limited diversity and reflecting biases of the teacher llms. figure 1 illustrates the few-shot synthesis approach @xcite , which we refer to as fewgen, for the task of detecting politically-biased articles. with a suitable prompt and in-context examples, sampling contin-generalist llm ## background and task setup in this paper, we focus on generating datasets for challenging text classification tasks. denote an example as consisting of input text x and output @xmath0. our goal is to produce a synthetic dataset and train a specialist language model m s (e.g. a bert-style pre-trained model @xcite ). we create d synth via task inversion: repeatedly prompting a teacher language model m lm to generate synthetic covariates x given corresponding labels y. we denote the student's task (predicting y from x) as τ and the teacher's task (generating x given y) as τ inv . synthesizrr aims to address the lack of diversity by leveraging retrieval during the content sourcing step. we assume the existence of a corpus r where each document may hold task-relevant information. however, documents need not originate from the same distribution as our task covariates; even distantly related documents can yield valuable synthetic examples. for instance, we shows that we can successfully generate reviews and humorous questions from a corpus of product descriptions. we also assume access to a seed set of examples d @xmath1, y 1 ), . . . , (x n , y n )} which is sufficiently large to represent the classes but small enough to be manually compiled by a user in a few hours; in experiments, we use the in-context learning set as d seed . importantly, we assume the seed set is insufficient to train an effective student, and a larger d synth (@xmath2. figure 2 illustrates our method for generating distributionally similar covariates. initially, we retrieve documents based on the examples in d seed , assuming that the corpus contains sufficient domain-similar documents. we then construct a context refinement instruction to perform task inversion on each retrieved document. this approach provides the llm with a unique and grounded prompt for each generated example, thereby circumventing the need for the teacher llm to memorize extensive corpus data within its limited parameters. task inversion may be challenging due to the mismatch between retrieved documents and test examples; to overcome this, we limit our investigation to teacher llms demonstrating strong instruction-following capabilities @xcite @xcite . ## method algorithm 1 shows our dataset generation method. we distill a student model in these steps: step 1. content sourcing using retrieval: synthesizrr uses each in-context covariate x icl as a query for information retrieval, in addition to its subsequently role during in-context learn- ## experimental setup tasks and their difficulty. we perform our main experiments on the first 6 datasets in task inversion for humor (ziser et al., 2020) involves generating humorous ques-corpus domain size doc. tokens realnews/dom us/eu news 30.1m article 27.1b realnews/reg regional news 2.7m article 2.1b realnews/ind indian news 0.9m article 0.6b products e-commerce 15.0m product 2.3b movie summary movies 42k plot 0.02b ## results: intrinsic evaluation in this section, we focus on evaluating intrinsic properties of the generated datasets, including their diversity and entity coverage. we focus on a llama-2 chat 13b teacher llm, retrieving from contriever using corpora per rq: does synthesizrr address entity diversity? popularity bias is a phenomenon wherein llm generations tend to over-represent popular "head" entities. this has been studied for qa tasks @xcite . in figure 4 we see how synthesizrr eliminates popularity bias across entity types. by sourcing from the long-tail of retrieval results (@xmath3, the generated dataset has much higher entity entropy compared to fewgen. this positions synthesizrr closer to gold, which also shows high entity entropy. rq: how is entity similarity in synthetic data affected by grounding to an in-domain corpus? for the category task we generate 8k product reviews and randomly select 8k gold examples. in each fewgen example also has a lower density of entities, as visible in ## results: student distillation we have established that synthesizrr generates more diverse datasets compared to a baseline approach. now, we return to the application of training a specialist model based on these datasets. with icl, 3-shot synthesizrr using the retricl strategy trains better students than 32shot fewgen (↑1.3% for llama and ↑3.2% for claude) and non-retricl. we conclude that naively adding icl examples is not an effective use of the llm's context window. instead, a better content sourcing strategy improves the student distillation, leading to better test performance. . gpt3.5 is text-davinci-003 whereas gpt3.5-t is gpt-3.5-turbo (openai, 2022), llama2 is 13b chat version @xcite , claudev1 is instant-v1.2 version @xcite . accuracy is measured on a distilbert student, where we train 5 student models and report the mean accuracy (std. dev. @xmath4.0 in all cases). within each dataset, we bold the best result. ## comparison to previous work we benchmark synthesizrr against four prior synthesis methods: (1) sungen @xcite uses zerogen to create 200k synthetic rows and employs a custom bi-level optimization algorithm to weight each instance; (2) regen @xcite utilizes two bert models, one for retrieval and one as a classifier, to multi-round filter noisy data; (3) s3 @xcite builds and iteratively enhances a seed dataset by identifying and synthesizing corrections using an llm; (4) attrprompt @xcite improves dataset diversity and unbiasedness by prompting gpt3.5-turbo with varied attributes (derived from a human-in-the-loop analysis of each task). standard zero-shot and few-shot generation baselines were compared in we benchmark three popular tasks: imdb @xcite , sst-2 @xcite and ag news @xcite . previous studies have generated larger datasets ranging from 20k to 200k examples with varying student model hyperparameters, but often lack reports on intrinsic dataset quality, making a fair comparison challenging. therefore, we independently reproduce these results using the synthetic datasets re-leased by the original authors foot_0 . following yu et al. (2023a), we subsample these datasets to 6k rows, keeping a uniform distribution across classes, and generate the same number of synthetic covariates using synthesizrr retricl (algorithm 1). for the content sourcing stage of synthesizrr, we retrieve documents from the cmu movie summary corpus @xcite and realnews/dom(appendix i). we measure accuracy on a distilbert student @xcite @xcite @xcite , fixing hyperparams to @xcite . ## conclusion in this work we describe how a retrieval corpus can be used to aid the synthesis of a text classification data set in specialized domains. we show that the diversity of the generated data is enhanced by including retrieved documents in a generation prompt. compared to few-shot generation, we find that synthesizrr produces more diverse and representative text and leads to better students.