Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation (2023)

Abstract: The decoding algorithm is critical for open-ended text generation, transforming latent representations into coherent and meaningful outputs. This paper investigates the self-reinforcement effect in text generation and the effectiveness of a repetition penalty to mitigate it. However, determining the optimal repetition penalty value is challenging. To tackle this, we propose a forgetting mechanism that disregards distant tokens, reducing the burden of penalty selection. In addition, we introduce a length penalty to address overly short sentences caused by excessive penalties. Our penalty decoding approach incorporating three strategies helps resolve issues with sampling methods deviating from factual information. Experimental results demonstrate the efficacy of our approach in generating high-quality sentences resembling human output.

## introduction open-ended text generation tasks involve generating coherent and fluent output with limited input information @xcite . these tasks encompass various applications such as chit-chat dialog @xcite , story generation @xcite , and similar domains. transformer-based models @xcite , which predict the probability of the next token during text decoding, have been widely adopted for such tasks. among them, the gpt series, utilizing a decoder-only auto-regressive model, has demonstrated remarkable performance @xcite . the choice of decoding strategy plays a crucial role in determining the quality of text generation, not only in open-ended text generation but also in other natural language generation tasks. decoding strategies can be categorized into two types. one is deterministic methods, and another to ensure high-quality text generation, penalties can be applied to the reinforced tokens, thereby correcting the distribution and improving the generated output. in this paper, we begin by reviewing and ana-lyzing the self-reinforcement effect proposed by @xcite , which refers to the tendency of maximization-based decoding algorithms to assign higher probabilities to tokens that have already been generated, leading to repetitive text. inspired by the work of @xcite , who observed that the vanilla gpt3 model (brown et al., 2020) often produces irrelevant and repetitive text, we hypothesize that this phenomenon may also exist in large language models. a critical insight of our work is to consider the distribution of a neural lm as an enhanced version. for instance, when presented with a prefix such as "barack obama was born in honolulu, hawaii. he was born in," the model (e.g., gpt2-xl) tends to repeat the previous context. this phenomenon might occur due to the probability distribution of the words "hawaii" and "honolulu" having a notable reinforcing effect within the probability space, as illustrated in the left part of figure 1 . to tackle this problem, we have introduced the repetition penalty proposed by @xcite . this penalty effectively mitigates the self-reinforcement effect and reshapes the token distribution, as demonstrated in the right section of figure 1 , resulting in a higher generation quality. however, tuning the repetition penalty hyperparameter can be challenging. therefore, we propose the forgetting mechanism and length penalty as additional strategies to ensure generation quality. through extensive experiments, we demonstrate the efficacy of our approach in generating sentences that closely resemble human-produced text. the self-reinforcement effect refers to the probability of repetition increasing with the number of historical repetitions @xcite . this effect can be intuitively reflected in the red probability shown in figure 1 , which may cause the model to get stuck in a repetition loop since the selection of tokens is forced to be limited to these enhanced tokens. to quantify this phenomenon, we design the following metrics at various levels. different from the metrics proposed by @xcite , which analyze the self-reinforcement of repeated tokens by different numbers of repeated prefixes, we dynamically do analyzation in the model's natural decoding process to reasonably evaluate the selfreinforcement phenomenon arising from the natural decoding state of the model. given a language the summed probability of the top-k tokens. note: topk in v topk the size of token space with different values of k, while top-k in the legend refers to the sampling method with @xmath0. nucleus level. the nucleus size of top-k highest probabilities in the probability distribution for the tth token is defined as ns(t) = @xmath1, where v topk is top-k highest probability token space according to p θ . if ## method in this section, we will present our penalty decoding approach, which comprises three techniques de-signed to enhance the performance of greedy decoding by efficiently mitigating the self-reinforcement effect to generate high-quality text. ## experiments in this section, we examine the effectiveness of penalty decoding compared to other decoding methods for open-ended text generation tasks. detailed setup for experiments can be found in appendix a.1, and comprehensive ablation studies are shown in appendix b. ## conclusion this paper delves into the self-reinforcement phenomenon in text generation and introduces penalty decoding as a solution. through the integration of a repetition penalty, a forgetting mechanism, and a length penalty, the token distribution is adjusted to enhance diversity and diminish model-induced hallucinations, consequently elevating the quality and coherence of the generated text. our penalty decoding can also be combined with sampling-based methods like top-k and top-p sampling. the findings and evaluations in this study aim to encourage further research in advancing the generation capabilities of neural language models. coherence. this metric @xcite employs the opt-2.7b language model @xcite to assess the coherence between the generated text and a given prefix. the metric is defined as follows: greedy ratio. this metric quantifies the proportion of times the language model selects the token with the highest probability during text generation @xcite .