k NN - LM Does Not Improve Open-ended Text Generation (2023)

Abstract: In this paper, we study the generation quality of interpolation-based retrieval-augmented language models (LMs). These methods, best exemplified by the k NN-LM, interpolate the LM’s predicted distribution of the next word with a distribution formed from the most relevant retrievals for a given prefix. While the k NN-LM and related methods yield impressive decreases in perplexity, we discover that they do not exhibit corresponding improvements in open-ended generation quality, as measured by both automatic evaluation metrics (e.g., MAUVE) and human evaluations. Digging deeper, we find that interpolating with a retrieval distribution actually increases perplexity compared to a baseline LM for the majority of tokens in the WikiText-103 test set, even though the overall perplexity is lower due to a smaller number of tokens for which perplexity dramatically decreases after interpolation. However, when decoding a long sequence at inference time, significant improvements on this smaller subset of tokens are washed out by slightly worse predictions on most tokens. Furthermore, we discover that the entropy of the retrieval distribution increases faster than that of the base LM as the generated sequence becomes longer, which indicates that retrieval is less reliable when using model-generated text as queries (i.e., is subject to exposure bias). We hope that our analysis spurs future work on improved decoding algorithms and interpolation strategies for retrieval-augmented language models.

## introduction retrieval-augmented language models, which integrate non-parametric dense retrieval with autoregressive next-token prediction, have been validated with strong empirical performance across a variety of tasks @xcite @xcite in addition to achieving low held-out perplexities on lm benchmarks. in this paper, we study interpolation-based lms, a subtype of retrieval-augmented lms that compute the probability of the next token by interpolating between the softmax distribution of the original lm and a token distribution formed by retrieving over an external datastore. these methods, perhaps best exemplified by the knn-lm @xcite , are particularly attractive because they allow any pretrained lm to be retrofitted with a retrieval module without further training. despite these advantages, there is limited understanding about the text generation quality of interpolation-based lms. in this study, we evaluate the quality of generated text from two such methods, knn-lm and trime @xcite , against the output of baseline lms that do not use retrieval. our evaluations involves open-ended text completions generated using different decoding algorithms on both the wikitext-103 and pg-19 datasets. we discover that interpolation-based lms do not improve the quality of generated text, as measured by both automatic text generation metrics such as mauve @xcite and human evaluation. this result begs the question of why the text generation quality does not improve, as the perplexity of interpolation-based lms is substantially lower than that of the baselines. our analysis of the knn-lm model suggests two potential reasons for this lack of improvement: unlike previous works that rely on perplexity to evaluate language modeling or bleu to evaluate machine translation quality of knn-lm-based models @xcite , our work specifically studies the open-ended text generation capability of knn-lms with a range of automatic evaluation metrics as well as human evaluation. we demonstrate that, though they significantly lower perplexity, retrievers might also impair text generation performance of knn-lms. this finding suggests potential future directions for using retrieval during text generation, such as developing more robust retrieval components or employing retriever mechanisms more selectively during decoding. ## related work we present the most extensive study of open-ended text generation foot_0 from interpolation-based lms such as knn-lm @xcite . our results reveal that although these methods are effective at reducing perplexity, they can also be detrimental to text generation. previous work finds that retrieval lms are improved by selectively incorporating retrieval when conditions are favorable @xcite @xcite , although they only examine the teacher-forced setting or other tasks, e.g. question answering. the knn-mt @xcite explores machine translation, which is a constrained task with short inputs, and thus not a good test of open-ended long-form generation. the knn-lm effectively scales retrieval to billions of tokens using a token-level non-parametric interpolation technique first introduced by @xcite . alternative retrieval-augmented models experiment with training the retriever @xcite @xcite , interpolating vectors instead of token probabilities @xcite , scaling to trillions of tokens @xcite , exploiting retrieval for strong few-shot learning @xcite , and so on @xcite @xcite @xcite @xcite . among these, knn-lm stands out as a relatively simple and fundamental work. our findings indicate important weaknesses of retrieval for text generation. reference-based metrics are not well suited to evaluate open-ended text generation @xcite . instead, effective automated approaches compare the machine generated and human language text distributions using samples @xcite @xcite . human evaluation remains the golden standard for natural language generation @xcite @xcite . ## experimental setup using a variety of commonly used text generation evaluation metrics, we evaluate the text generation capability of interpolation-based lms and compare them to baseline lms (i.e., without k-nearestneighbor retrieval from an external datastore). in this section, we describe our experimental setup, including models, automatic evaluation metrics, data selection, and hyperparameters. ## results we find that despite incorporating the retrieval component and interpolating information from the baseline lm and retrieval, these methods do not yield any significant improvement to text generation performance, and even worsen it by some metrics ( ## 15026 interpolation-based lms do not improve automatic text generation evaluation metrics: we find that none of the three models significantly improve generation quality compared to the baseline lm, as shown by various metrics ( these results hold across different decoding algorithms: the results in these results hold across different datasets: in addition to wikitext-103, we also evaluate the text generation performance of the knn-lm on the pg-19 dataset @xcite , which predominantly comprises fictional books and presents a distinct thematic variation to wikipedia. we construct an evaluation dataset from pg-19 similarly to our constructed evaluation dataset from wikitext-103 in section 3.2. 9 the baseline lm is gpt2 ## why do knn-lms fail to improve text generation quality? our evaluations (both human and automatic) do not show a significant quality increase when interpolating an lm's predicted probability distribution with one formed via retrieval over large external datastores. in this section, we try to understand why we do not observe an improvement by empirically analyzing the knn-lm and find two potential reasons: ## discussion in addition to the limitations of interpolation-based lms described in section 5, we hypothesize that there are other potential factors that contribute to the shortcomings of the knn-lm for text generation. specifically, it is possible that the interpolation may impede the language models' ability for self-recovery, and also that integrating the retrieval distribution can potentially introduce additional burdens related to hyperparameter tuning, which may not be optimized for text generation. we discuss these potential issues here as they are interesting avenues to explore for future work. retrieval interpolation may damage the selfrecovery ability of lms: language models exhibit some degree of self-recovery abilities @xcite , i.e., they can regain fluency and coherence even after previously generating poorquality tokens. this self-recovery capability is attributed to the lm's ability to pay close attention to recent context and ignore the long-range past context. however, we hypothesize that when interpolation-based lms encounter artifacts (e.g., non-factual or disfluent text) in a distorted pre-fix qt , they may be less likely to recover, as the retrievals may further increase the probability of completions that resemble those artifacts. furthermore, as we continuously sample and append tokens to the prefix, which the retriever uses as the query to construct p kn n (w t |q t ), the retriever may encounter additional exposure bias as shown in section 5.2, negatively impacting the quality of p kn n (w t |q t ). thus, even when the baseline lms "recover" from distorted past context by producing a high-quality distribution over the next-token prediction p lm (w t |q t ), the retriever may re-introduce the distortion by interpolating p lm (w t |q t ) with p kn n (w t |q t ). hyperparameters introduced by knn-lm are not optimized for text generation: the knn-lm introduces two important hyperparameters, namely the relative weight between the two distribution λ, as well as softmax temperature for the knn distribution τ kn n . recent work @xcite highlights the significance of tuning τ kn n for achieving optimal knn-lm performance, as measured by perplexity. similarly, we investigate the coefficient parameter λ, which plays a vital role as it controls the relative importance assigned to the knn retriever and baseline lm. existing works tune λ by the perplexity on the validation set. however, from figure 4 , we observe that the λ values that produce the lowest perplexity may not translate to the optimal value for text generation quality (measured by mauve). instead of tuning λ for optimizing perplexity, we may want to consider context-dependent λ as in @xcite for generation (e.g., only use the retrieval distribution when it is very confident). finally, interpolation may warrant the design of new decoding algorithms specialized for retrieval-augmented generation. ## conclusion in this work, we show that despite the significant perplexity improvement brought by interpolationbased retrieval-augmented lms such as knn-lms, such methods fail to improve the lms' text generation performance. the text generation quality between knn-lms and baseline lms without retrieval show no significant difference according to both automatic text generation evaluation metrics and human evaluation. upon closer analysis, we identify flaws in using knn-lms to perform autoregressive text generation: the method only benefits a minority of token predictions, and the retriever's 8 quality deteriorates when generating long-form text. we hope our findings can inspire future research to design better training and inference methods so that the impressive improvement of knn-lms in perplexity can better be translated into gains in text generation quality.