Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2024)

Abstract: We present Impossible Distillation, a novel framework for paraphrasing and sentence summarization, that distills a high-quality dataset and model from a low-quality teacher that itself cannot perform these tasks. Unlike prior works that rely on an extreme-scale teacher model (e.g., GPT3) or task-specific architecture, we hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs (e.g., GPT2), where paraphrases occupy a proximal subspace in the LM distribution. By identifying and distilling generations from these subspaces, Impossible Distillation produces a high-quality dataset and model even from GPT2-scale LMs. We evaluate our method on multiple benchmarks spanning unconstrained / syntax-controlled paraphrase generation and sentence summarization. Our model with 770M parameters consistently outperforms strong baselines, including models distilled from ChatGPT, and sometimes, even ChatGPT itself. Also, we find that our distilled dataset from 1.5B LMs exhibits higher diversity and fidelity than up to 13 times larger datasets.

## introduction training a compact, yet performant model is a nontrivial challenge in modern nlp, even for classical tasks such as paraphrase generation and sentence summarization. while large-scale, high-quality data is central to this goal, human supervision is hard to scale; as such, research efforts have focused on training models with an unsupervised, automatically generated dataset. common approaches include back-translation @xcite and auto-encoding @xcite , but are often limited in terms of corpus diversity and noisiness @xcite . alternatively, recent works propose to train a compact task model by distilling knowledge from gigantic language models (llms) @xcite i wish to visit there sometime. i hate a long flight. what about you? i don't want to sit in a plane for a long time. ## paraphrastic proximity we develop impossible distillation based on the observation of paraphrastic proximity -i.e. when the lm decoding space is constrained with sufficiently informative context, the model can produce multiple generations that paraphrase each other. notably, @xcite indirectly leverages paraphrastic proximity in context lms -a set of encoder-decoder transformers pre-trained from scratch using specialized training objectives. we, on the other hand, show that paraphrastic proximity holds for off-the-shelf lms such as gpt2, which we can make use of to distill a high-quality task model and dataset. in this section, we first verify this with gpt2-xl @xcite . while the exact distribution of all possible generations is intractable, we can obtain an approximation by sampling a large number of generations given a contextual constraint. concretely, we first sample 1000 context (each with 1-5 sentences) from news articles in xsum dataset @xcite . then we prompt gpt2 to generate 100 next sentences per each contextual constraint. to evaluate whether these sentences are indeed paraphrastic to each other, we measure their (1) pair-wise semantic equivalence, and (2) surface-form dissimilarity. for semantic equivalence, we employ an off-the-shelf nli model @xcite , and determine a pair (x, y) to be semantically equivalent if entailment holds in both directions. for surface-form dissimilarity, we compute the self-bleu @xcite between sentences, for only the semantically equivalent pairs. the results are presented in fig. 3 . in the left figure, as the context becomes longer (i.e. more informative), the generated sentences are more likely to be semantically equivalent, verifying our assumption. notably, the high semantic equivalence does not come from merely generating sentences with similar surface form -even with longer context, the average pair-wise self-bleu is around 32 foot_0 . the results indicate that gpt2 can generate a large number of paraphrases simply by over-sampling multiple completions to the given context. on the right side of fig 3 , we also gauge the paraphrastic proximity under various decoding temperatures. here, the ratio of semantically equivalent pairs dramatically increases as the temperature decreases. low temperature adjusts the sampling distribution to be more skewed towards regions with high probability mass, hence allowing paraphrases to be more easily sampled from these subspaces. however, it is important that the temperature balance the trade-off between sample efficiency and diversity; when the temperature is too low, the generated sentences are almost identical and hence does not qualify as desirable paraphrase. ## impossible distillation impossible distillation starts from an offthe-shelf teacher lm m t , and distills its knowledge into a student lm m s , yielding a specialized model m task for paraphrasing and sentence summarization. as a byproduct of this process, we also obtain a high-quality dataset d task . below, we detail the process focusing on paraphrase generation as the task of interest, then discuss how this generalizes to sentence summarization. ## conclusion in this work, we propose impossible distilla-tion, a novel framework to distill high-quality paraphrase dataset and model from small, lowquality lms. we show that by leveraging paraphrastic proximity and critic-guided distillation, impossible distillation can empower small lms to outperform competitive counterpartsin both performance and controllability, across domains and tasks, without training on humanauthored paraphrases. also, we find that dim-ple, the natural byproduct of our method, presents higher fidelity and diversity than order of magnitude larger paraphrase datasets. impossible dis-tillation shows a promising direction to rediscover the under-explored capabilities of off-theshelf language models, by accurately identifying their characteristics and amplifying them.