Local Temperature Beam Search: Avoid Neural Text D e G eneration via Enhanced Calibration (2023)

Abstract: Previous studies have constantly observed that a language model repeats itself, creating repetitions in an output sequence. To cope with the issue, stochastic decoding schemes have been the de facto approaches; the strategies add randomness in inference, hence avoiding the “self-loop”. However, the remedy comes at the cost of sacrificing output quality due to the randomness involved. In this work, we introduce a deterministic decoding scheme, local temperature beam search. This inference algorithm is an embarrassingly simple variant of beam search, yet it reduces repetition, whose level is superior to that of a sampling-based decoding algorithm, while maintaining the level of coherence as in beam search. Our idea is rooted in the concept of model calibration; we view a repetition as a casualty from overconfidence in a model. Therefore, our work mitigates the miscalibration present in the course of inference with a post-calibration approach applied in beam-specific manner. Our inference scheme is validated on text completion tasks, in which the repetition problem is seen most clearly, and is exhaustively compared with existing inference schemes.

## introduction neural language models have gained much attention with ground-breaking performances @xcite , and accordingly, decoding algorithms have been studied extensively along with such models @xcite @xcite . an inference algorithm aims to find an optimal hypothesis from a search space, which the level of optimum is commonly approximated and mapped by a language model. a choice of decoding/search algorithm can result in significant differences in model outputs, such as in diversity and coherence @xcite . for this reason, a large body of research has been done to find an optimal search algorithm @xcite @xcite @xcite . in a broad view, there are two branches in the inference algorithm: deterministic and stochastic. deterministic branch includes greedy decoding and beam search which are maximization-based inference strategies; the methods select tokens that maximize a sequence probability, hence generating coherent and high quality sequences. however, it has been constantly reported that the maximizationbased schemes generate highly repetitive outputs @xcite . therefore, stochastic decoding algorithms, such as top-p @xcite and top-k @xcite , have been the de facto options in an environment where a language model is likely to repeat itself, such as text completion. however, with the randomness introduced, the stochastic methods are at risk of generating incoherent sequence @xcite . in this work, we view the repetition problem of a language model from the standpoint of model calibration. our intuition is rooted in two interesting findings: 1) a language model assigns high probabilities to repeating tokens @xcite , but 2) human texts hardly contain repetition within a sequence @xcite . in summary, we hypothesize that a language model is overconfident when repeating itself, assigning spuriously high predictive scores to predictions that are less likely to appear. the calibration of a language model is of importance especially in beam search @xcite ; beam search keeps only a finite number of "likely" beams, whose likelihood is the (log) probabilities mapped by a model in inference. therefore, when a probability in a beam is overestimated due to overconfidence, the search will be biased towards the overconfident beam, leading to the degeneration in text, i.e. repetition. in this light, we propose local temperature beam search which is a deterministic algorithm that mitigates the long-standing repetition problem in a deterministic search algorithm while fully enjoying the strengths of a maximization-based approach. we mitigate the bias in beams, caused by overconfidence, by introducing local temperature scaling, in which a temperature value is decided in a contextspecific manner. accordingly, the repetition problem of a language model with our decoding scheme is reduced as much as a sampling-based scheme, whereas coherence score of generated outputs is as high as that of a maximization-based strategy. we attribute such improvement to its connection to n-gram blocking @xcite ; we illustrate how the proposed beam search is an implicit version of n-gram blocking. in this sense, the proposed idea brings more freedom in a beam search process while reducing repetition as n-gram blocking. the proposed decoding scheme is thoroughly tested in text completion, in which the stochastic decoding schemes have been dominant until now. the contributions are as follows: ## related work recent studies have introduced decoding schemes that mitigate such weakness present in the popular decoding schemes. diverse beam search @xcite adds sequence dissimilarity between the beams in the score function, expecting diversity in the final outputs of b beams. diverse sibling beam search @xcite penalizes beams that have the same root, so that outputs do not overlap with each other. delayed beam search @xcite transits from sampling to maximization, in which the first j time steps are inferred with a sampling-based scheme and the rest of the generation is done with beam search. stochastic beam search @xcite and conditional poisson stochastic beam search @xcite bring randomness to beam search by sampling predictions at each time step. @xcite proposes adding noise to a hidden state of a decoder, exploring the data manifold with the noise. this noise-based decoding has been shown to mitigate the low diversity in generated text and has been referred to as noisy parallel approximate decoding (npad). @xcite introduces repetition penalty, in which the penalty is given to a repeating token by discounting the corresponding logit value while leaving other logit values untouched. lastly, contrastive search @xcite is a recent attempt that utilizes hidden representation in order to avoid repetition. the approach noticeably reduces repetitions, yet the core weaknesses are 1) high computation costs and 2) high dependency on isotropy level of an inference model. another popular branch of inference strategy is sampling. top-k @xcite considers only the k most probable tokens when sampling, truncating the sampling space to k indexes. topp sampling considers the smallest set of indexes whose sum amounts up to the p probability. typical sampling @xcite samples a token based on the expected information content of a token. for more detailed explanation, please refer to appendix a. ## conclusion & future study in this study, we view the repetition problem of a language model as a calibration issue; a language model repeats itself as the model is overconfident in predictions. in this light, we propose a local temper-ature scaling in which the post-calibration method is applied only to the overconfident beams. our local temperature beam search is a deterministic decoding strategy that excels in reducing repetition while maintaining coherence level; we attribute the success to the implicit penalty given in the course of generation. lastly, unlike existing inference strategies, the proposed idea is robust to a variety of choice of temperature and beam width. the objective of this paper is centered around reducing the self-overlap in a sequence, and hence the subject of local temperature scaling is chosen accordingly. however, the local temperature beam search can be utilized in tasks with different aims, as local temperature has a role of implicitly penalizing a beam. for example, a language model can be penalized with the proposed idea when the model generates a sequence with gender bias in it. we believe that ways to utilize our approach can be explored in future research.