R ank G en: Improving Text Generation with Large Ranking Models (2022)

Abstract: Given an input sequence (or prefix), modern language models often assign high probabilities to output sequences that are repetitive, incoherent, or irrelevant to the prefix; as such, model-generated text also contains such artifacts. To address these issues we present RankGen, a 1.2B parameter encoder model for English that scores model generations given a prefix. RankGen can be flexibly incorporated as a scoring function in beam search and used to decode from any pretrained language model. We train RankGen using large-scale contrastive learning to map a prefix close to the ground-truth sequence that follows it and far away from two types of negatives: (1) random sequences from the same document as the prefix, and (2) sequences generated from a large language model conditioned on the prefix. Experiments across four different language models (345M-11B parameters) and two domains show that RankGen significantly outperforms decoding algorithms like nucleus, top-k, and typical sampling on both automatic metrics (85.0 vs 77.3 MAUVE) as well as human evaluations with English writers (74.5% human preference over nucleus sampling). Analysis reveals that RankGen outputs are more relevant to the prefix and improve continuity and coherence compared to baselines. We release our model checkpoints, code, and human preference data with explanations to facilitate future research.

## introduction despite exciting recent progress in large-scale language modeling @xcite ), text generated from these language models (lms) continues to be riddled with artifacts. modern lms suffer from the "likelihood trap" @xcite , in which high likelihood (low perplexity) sequences produced by greedy decoding or beam search tend to be dull and repetitive. while truncated sampling methods such as @xcite , nucleus @xcite , and typical sampling @xcite alleviate these issues, they can also produce text with inconsistencies, hallucinations, factual errors, or commonsense issues @xcite @xcite . part of the problem is that lms are trained using "teacher forcing", where they are always given the ground-truth prefix foot_1 and asked to predict the next token. at test-time, however, the prefix can contain model-generated text, allowing errors to propagate during decoding @xcite . this issue, combined with the observation that lms overly rely on local context @xcite , contributes to the generation of sequences that break coherence or consistency within a larger discourse-level context @xcite . to address this issue we present rankgen, a 1.2 billion parameter english encoder model that maps both human-written prefixes and modelgenerated continuations of those prefixes (generations) to a shared vector space. rankgen efficiently measures the compatibility between a given prefix and generations from any external lm by ranking the generations via their dot product with the prefix (figure 2 ). we train rankgen using large-scale contrastive learning, encouraging prefixes to be closer to their gold continuation and far away from incorrect negatives. since our objective considers two sequences rather than just single token prediction, it encourages rankgen to consider longer-distance relationships between the prefix and continuation rather than just local context. we devise two different strategies (shown in figure 1 ) for selecting challenging negative samples, ## rankgen: a generation ranker rankgen is a deep encoder network that projects prefixes and generations to a shared vector space. given a prefix vector and a generation vector, we compute a score for the generation via the dot product between the two vectors. to ensure that these scores are meaningful, we train rankgen using large-scale contrastive learning @xcite , pushing the prefix vector close to the gold completion and away from the vectors of negative samples (figure 1 ). we use two types of negative samples for learning the metric space: ## conclusion and future work we present rankgen, a large encoder which scores continuations given a prefix and can be plugged into any text generation system. rankgen significantly outperforms popular decoding methods on both automatic and human evaluations. we note several exciting future directions for rankgen, including: