Generating Datasets with Pretrained Language Models (2021)

Abstract: To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs. While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size. In this paper, we show how PLMs can be leveraged to obtain high-quality sentence embeddings without the need for labeled data, finetuning or modifications to the pretraining objective: We utilize the generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch, which we then use for finetuning much smaller and more efficient models. Our fully unsupervised approach outperforms strong baselines on several semantic textual similarity datasets.

## introduction while pretrained language models (plms) achieve strong results for many nlp tasks @xcite @xcite , they do not produce good sentence embeddings out of the box @xcite . recent approaches address this by augmenting or replacing the language modeling objective with likewise unsupervised sentence-level objectives (e.g., @xcite ), but they typically lag behind their supervised counterparts trained on human-annotated sentence pairs. unfortunately, obtaining large amounts of high-quality training data can be both difficult and prohibitively expensive @xcite . furthermore, with larger and larger model sizes @xcite @xcite , it becomes increasingly challenging to finetune plms. 1 our code and datasets are publicly available at https: //github.com/timoschick/dino . task: write two sentences that mean the same thing. sentence 1: "a man is playing a flute." sentence 2: "he's playing a flute." task: write two sentences that are somewhat similar. sentence 1: "a man is playing a flute." sentence 2: "a woman has been playing the violin." task: write two sentences that are on completely different topics. sentence 1: "a man is playing a flute." sentence 2: "a woman is walking down the street." figure 1 : continuations generated by gpt2-xl with dino for three different task descriptions. we investigate two different unsupervised approaches to generating sentence-similarity datasets: (i) the input sentence is given and only the continuation is generated. this requires that an (unlabeled) set of sentences is available. (ii) both input sentence and continuation are generated. this does not rely on the availability of any resources. to alleviate both problems, we explore a novel approach to obtaining high-quality sentence embeddings: we mimic the creation of nli datasets by human crowdworkers @xcite , but replace human annotators with large plms. this allows us to automatically create entire datasets from scratch that can be used for supervised training of much smaller models. not only does this solve the problem of limited training data, it also provides a viable path to leverage big models like gpt-3 @xcite without requiring any updates to their parameters. as illustrated in figure 1 , our approach is based on recent methods for providing instructions to plms (e.g., @xcite @xcite . we use the self-debiasing approach of @xcite to ensure that each generated text pair is not only a good fit for a given similarity label, but also not a good fit for other labels. we refer to our method as datasets from instructions (dino). in summary, our contributions are as follows: ## related work there are many unsupervised approaches to obtaining sentence embeddings, for example by averaging word embeddings @xcite @xcite or with carefully designed sentence-level objectives @xcite . ensembling several methods improves results @xcite . recent work obtains sentence representations by supplementing bert @xcite or other plms with additional unsupervised objectives @xcite @xcite . often, labeled datasets such as paraphrase databases @xcite or natural language inference datasets @xcite @xcite are used for supervised learning. some approaches augment existing datasets with automatically generated examples @xcite @xcite @xcite , but in contrast to our work, all of these approaches require that there already exists a labeled dataset for finetuning the generator. providing plms with task descriptions for zero-or fewshot learning has been studied extensively (e.g., @xcite @xcite @xcite @xcite . however, none of these approaches is suitable for generating sentence embeddings. closely related to our work, efrat and levy (2020) examine the ability of plms to follow natu-task: write two sentences that i y . sentence 1: "x 1 " sentence 2: " figure 2 : instruction template i y (x 1 ) for similarity label y and input sentence x 1 ; i y is described in section 3. see figure 1 for three instantiations of the template. ## datasets from instructions let m be a plm with vocabulary v , @xmath0. our aim is to generate a dataset z ⊂ x × x × y of text pairs (x 1 , x 2 ) with corresponding similarity labels y. for @xmath1, we denote with p m (x | x) the probability that m assigns to x as a continuation of x. we first assume that we already have access to a set x 1 ⊂ x of texts (e.g., a set of sentences that are typical of the domain of interest). this is a realistic setting for many real-world applications, where large amounts of unlabeled text are abundant, but it is difficult to obtain interesting and (for our task) useful text pairs and labels. dino requires a set of instructions @xmath2, given an input x 1 ∈ x 1 , prompts its recipient to generate an appropriate second text x 2 . we use the instruction template in figure 2 and consider three levels of similarity (@xmath3, 0.5, 1}), where @xcite 's five-level similarity scheme. note that for all y, i y ends with an opening quotation mark, which allows us to treat the first quotation mark generated by the plm as a sign that it is done. for a given x 1 ∈ x 1 and @xmath4, we could directly use the instructions i y to obtain x 2 by continuously sampling tokens for settings where no set of unlabeled texts x 1 is available, a straightforward approach would be to use the phrase shown in figure 2 up to and including the first quotation mark as an instruction to let the plm generate both x 1 and x 2 . however, this approach has at least two issues: first, generated texts may not match the required schema (e.g., the model may never produce the string "sentence 2:"). second, the set of texts x 1 should ideally be highly diverse, whereas we want to give the model less leeway when generating x 2 , so we may want to use different sampling strategies for x 1 and x 2 . we solve both problems as follows: we first use i y (figure 2 ) up to and including the first quotation mark (the one right after "sentence 1:") to generate x 1 ; we stop as soon as the model produces a quotation mark. we run this procedure repeatedly until we have a sufficient number of sentences. these are gathered into a set x 1 and then we proceed exactly as in the case where x 1 is already given. ## experiments we evaluate dino on several english semantic textual similarity datasets: the sts tasks 2012-2016 @xcite @xcite @xcite , the sts benchmark (stsb) @xcite , and the sick-relatedness dataset (sick) @xcite . for all tasks, we adopt the unsupervised setting without task-specific training examples. we use dino to generate sts-⊂ x×x×y , a dataset of text pairs with semantic similarity labels. we generate two variants: we use gpt2-xl as plm with a decay constant of λ = 100 and the set of counterlabels cl(y that is, we do not restrict the plm when generating texts for @xmath5, but for @xmath6.5 (@xmath7. we apply top-p @xcite and top-k @xcite sampling with @xmath8.9, @xmath9. for each x 1 ∈ x 1 and @xmath10, we generate up to two corresponding x 2 's. foot_0 for sts--x 1 x 2 , we obtain x 1 by generating 15,000 sentences using only top-p sampling (again with @xmath11.9) and no top-k sampling to ensure more diversity in the generated output. we remove all examples where x 1 = x 2 (as those provide no training signal to the model) and split the datasets 90/10 into training and validation. to assess the quality of the generated datasets, we use them to train sentence-roberta @xcite , a biencoder architecture based on roberta (base) @xcite that measures the similarity of two texts by computing the cosine similarity of their embeddings. as our datasets contain many noisy examples, we use a technique similar to label smoothing @xcite and replace similarity scores of 0 and 1 with 0.1 and 0.9, respectively. additionally, for each x 1 , we sample two x 2 's from other dataset entries and augment the dataset with (x 1 , x 2 , 0). we use the default parameters of @xcite with a batch size of 32 and train for at most one epoch; the exact number of training steps is determined based on spearman's rank correlation on the sts-validation set. ## conclusion we have introduced dino, a method for using large plms to generate entire datasets of labeled sentence pairs from scratch, requiring no labeled data and no parameter updates. this is achieved by providing instructions in natural language, combined with the self-debiasing method of schick for future work, it would be interesting to see whether the noise in datasets generated with dino can further be reduced, e.g., by using different sets of instructions @xcite or by supplementing our pipeline with some additional filtering steps.