Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models (2021)

Abstract: Publicly available, large pretrained Language Models (LMs) generate text with remarkable quality, but only sequentially from left to right. As a result, they are not immediately applicable to generation tasks that break the unidirectional assumption, such as paraphrasing or text-infilling, necessitating task-specific supervision. In this paper, we present Reflective Decoding, a novel unsupervised algorithm that allows for direct application of unidirectional LMs to non-sequential tasks. Our 2-step approach requires no supervision or even parallel corpora, only two off-the-shelf pretrained LMs in opposite directions: forward and backward. First, in the contextualization step, we use LMs to generate ensembles of past and future contexts which collectively capture the input (e.g. the source sentence for paraphrasing). Second, in the reflection step, we condition on these “context ensembles”, generating outputs that are compatible with them. Comprehensive empirical results demonstrate that Reflective Decoding outperforms strong unsupervised baselines on both paraphrasing and abductive text infilling, significantly narrowing the gap between unsupervised and supervised methods. Reflective Decoding surpasses multiple supervised baselines on various metrics including human evaluation.

## introduction language models (lms) like gpt-2 @xcite , trained over vast unstructured data, can leverage enhanced generation methods @xcite @xcite to give fluent and coherent continuations to given input text-e.g. news articles or stories. 1 further results and resource are available at https://homes.cs.washington.edu/ ˜pawest/ reflectivedecoding.html figure 1 : illustration of reflective decoding applied to paraphrasing and abductive infilling @xcite . only the right-context is shown, although both right-and left-contexts are used in practice. first the contextualization step (1) captures aspects of an input by generating many representative contexts for it. then in the reflection step (2) we sample generations that can replace the input and fit these representative contexts ← -rd. gpt-3 @xcite takes this a step further: given a small number of examples and a well-constructed prompt, it shows remarkable performance on tasks where vast quantities of supervised data and finetuning were thought to be necessary. while this demonstrates the potential for lm-decoding in few-shot or even zero-shot outof-the-box settings, limited access to gpt-3 and immense computational cost keep this from being a widely usable or efficient solution. yet recent work shows that gpt-2 may hold similar capabilities when it is primed correctly. @xcite achieve supervised-level performance in a few-shot setting using smaller, accessible models like gpt-2. they learn a small number of taskspecific vectors as a prefix to the input, without tuning the model itself. off-the-shelf gpt-2 is capable of few-shot learning given the right setup; our work aims to push this concept further, by showing that out-of-the-box lms can solve complex generation problems simply by using the right decoding algorithm. we introduce reflective decoding-a novel decoding method that allows lms to be applied to generation tasks that break the "text continuation" paradigm, such as paraphrasing and textinfilling. reflective decoding requires no supervision, only two complementary off-the-shelf lms-one forward ( -→ lm) and one backward ( ←-lm). that means no per-task finetuning, even on unstructured text in the target domain. inspired by the distributional hypothesis @xcite , reflective decoding works by generating text that might occupy the same contexts as an input. we use two lms ( -→ lm and ←-lm) to generate contexts for a given input, which implicitly capture aspects of its meaning (the contextualization step). then, in the reflection step, we condition on this ensemble of contexts, decoding over the input with generations that are distributionally related to-or replace-the input. paraphrasing is a natural application: a good paraphrase should intuitively be compatible with the same contexts as the original text. reflec-tive decoding shows strong unsupervised paraphrasing performance: on the quora question pair dataset, we find one variant of our model (rd 30 ) outperforms unsupervised baselines on all but one metric, and supervised baselines on both the sari metric and human evaluation. we see the same trends on the twitter url corpus @xcite . reflective decoding can also be applied to tasks that only replace part of the input, or generate within it, like infilling; on αnlg @xcite , we outperform the best unsupervised baseline on overall quality, effectively halving the gap with supervised methods. in both applications, reflective decoding directly applies off-theshelf pretrained models, without finetuning on the task or target domain. this provides evidence that off-the-shelf language models can excel at surprising applications, when paired with decoding algorithms designed to elicit specific kinds of infor-mation. ## results and analysis paraphrasing first, the quora dataset: on automatic metrics from past works (bleu, meteor, ter p ) our lowest-n ovelty model setting (rd t op ) achieves the highest unsupervised scores, and highest overall on bleu. other high scoring rows (source, pg-il) are similarly low-n ovelty. the sari metric explicitly balances n ovelty with similarity to reference. on sari we see such low-n ovelty models perform worse. the best overall model on sari is our medium-n ovelty setting (rd 30 ) which outperforms mt and supervised models. our human evaluation measures what fraction of outputs are found to be fluent, consistent, and novel. as with sari, both our mid and high-n ovelty models perform quite well, again with the medium-n ovelty setting outperforming all baselines. as further validation for sari as a proxy for human, they share the same top-5 models. results on the twitter url corpus largely support those on quora. reflective decoding achieves the best unsupervised scores on noveltyaware metrics ( in sum, reflective decoding is able to compete on previously used quality metrics favoring low-n ovelty, but can produce more varied outputs preferred by humans. rd 45 is among the best models by sari and human on quora despite exceeding the novelty of even the reference. we also include example generations in figure 2 to demonstrate the ability of reflective decod-ing to combine o 1 and o 2 . for example, @xmath0, and while she was on the swing, she fell off and was lying on the ground. incorporates information from both observations. specifically, it takes into account the swing that ray is building for his daughter which is only mentioned in o 1 , and hypothesizes about a potential injury due to ray checking on his daughter in o 2 . see appendix for more generations. overall, the strong performance of reflective decoding on αnlg shows that unsupervised generation with context ensemble applies to infilling in addition to paraphrasing. a useful abstraction for understanding re-flective decoding for current applications is "prompting", i.e., writing a prefix to implicitly or explicitly describe a task for a pretrained model. re-flective decoding generates natural contexts that the desired generation would appear in. this breaks from other methods of automatic prompting, which often forego "natural" prompts @xcite , even making them continuous @xcite @xcite . reflective decoding also notably creates a set of prompts (contexts) for each example, where other methods attempt to learn an overall task prompt. still, all of these are connected by the popular intuition that useful behavior in pretrained models can be induced through contextual input. future applications reflective decoding can extend beyond our experiments here, however. a simple example is in-context paraphrasing, i.e. writing a paraphrase that fits the true context that the original sentence appears in. most existing paraphrasing methods consider only out-of-context sentences, and would require significant changes to consider context as a constraint; for reflective decoding we can simply combine true and generated contexts without with the same algorithm. driving reflective decoding is a notion of context as a representation, with clear poten-tial for future work. pretrained lms capture rich information about text spans, but accessing it without fine-tuning is nontrivial; within the model it is an uninterpretable mass of parameters and activation weights. our work observes that unidirectional lms are only capturing this information to predict adjacent context-this is the sole learning signal-so all of this information is expressed in the model's context prediction. thus, we capture some of this rich information to represent spans, by capturing a finite-sample version of this full predictive distribution in generated contexts. in reflec-tive decoding specifically, we use this form of representation to generate back into the source span-paraphrasing or infilling-but the notion can be applied much more generally. in translation for instance, we might first generate contexts for the source sentence that represent its meaning, noisily translate these contexts, then impose that any translations for the source fit the same contexts under a translation-language lm. constraining translations in this way can add robustness to existing systems by anchoring translations to informative contexts. beyond explicit generation even, we might use a very large lm like gpt-3 to define a strong scoring function or metric as in equation 7, first generating contexts for some target sentence, then scoring candidates by how well they generate these same contexts. as in our work, such a score could indicate how well the option fills the same contextual role as the target, harnessing the strong reasoning of whatever model is used. ## related work distributional intuitions a key aspect of re-flective decoding is using a distributional intuition to represent the meaning of a text through many contexts. @xcite ; @xcite quantify semantic relationships and @xcite identify paraphrastic relationships under similar intuitions. a major point of difference between past work and ours is that we sample explicit contexts, allowing unsupervised generation back from these contexts, while past work typically learns a neural representation based on contexts and conditions on this vector-encoded representation. unsupervised paraphrasing some approaches train neural variational auto-encoders unsupervised to represent source sentences, then decodes from these representations to paraphrase (roy and grangier, 2019b; @xcite . this requires training specialized representations, whereas reflective decoding applies general-purpose lms. we compare to @xcite . paraphrasing by editing the input @xcite has shown promise. like reflective decoding, these approaches can be applied without training specialized models, but are necessarily limited by edit-paths and local minima, as edits are often restricted to single-word replacement, insertion, and deletion. generated paraphrases must follow a continuous local edit path, while reflective decoding can generate new sentences from scratch. reflective decoding and mt-based paraphrasing both pivot through an alternative textual form to paraphrase (context and translation, respectively). but mt paraphrasing systems cycletranslate through a pivot language @xcite , which requires supervised bilingual translation data, with an implicit notion of interlingual paraphrasing. @xcite observe that paraphrases close to the source often win on automatic quality metrics. however, dissimilarity from the source correlates with human notions of paraphrasing @xcite . @xcite increase novelty through their diversity-promoting sampling method. alternative metrics that consider novelty alongside quality have been proposed @xcite . the sari metric @xcite , included here, combines these notions. ## conclusions we present reflective decoding, a novel unsupervised text generation method for tasks that do not fit the text continuation paradigm. it uses just two pretrained language models to generate contexts that capture aspects of input text, generating back into the input from there. it significantly outperforms unsupervised baselines in quality and novelty for paraphrasing. further, in abductive natural language generation it outperforms unsupervised baselines by a significant margin and halves the gap with supervised models. reflective de-coding uses the concept of representing meaning with generated contexts, offering new possibilities for unsupervised conditional text generation. the original release; existing code and hyperparameters were available, and we use a smaller dataset. further, there is no publicly available backward gpt-2 model that we are aware of, so releasing a pair of forward and backward models that were trained on the same data allows for proper comparisons about left-to-right vs. right-to-left processing of english text. we estimate that the potential dangers of releasing this from a malicious generation perspective are low. our forward model is similar to already released gpt-2 models. while the backward model adds new generation potential and scientific novelty, it is unlikely to compare to gpt-3 @xcite which many hobbyists and private companies now have access to. we believe that releasing a pair of forward and backward models will be more useful to researchers who wish to study the symmetries and asymmetries of the linguistic distribution.