Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation (2023)

Abstract: Despite the huge progress in myriad generation tasks, pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts with maximization-based decoding algorithms for open-ended generation. We attribute their overestimation of token-level repetition probabilities to the learning bias: LMs capture simple repetitive patterns faster with the MLE loss. We propose self-contrastive training to penalize the output of a premature checkpoint of the same model when it incorrectly predicts repetition, which is shown to mitigate repetition effectively while maintaining fluency on two datasets. Furthermore, we find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops.

## introduction existing lms prefer to generate repetitive texts for open-ended generation with greedy decoding or beam search @xcite . even largescale pretrained lms such as gpt3 @xcite still generate redundant sentences @xcite . despite many solutions proposed from the perspective of both training @xcite and decoding @xcite , the cause of preference for repetition still needs to be clarified. by analyzing the training dynamics of lms regarding (non-)repetitive tokens, we reveal the learning bias towards repetition: lms capture simple repetitive patterns first, which dominate the output distribution throughout the input space, and then learn more non-repetitive patterns during training. we show that the repetition problem can be mitigated by only training more steps (i.e., allowing over-fitting), although the coherence with inputs will be impacted. conversely, when trained insuf-ficiently, lms will overestimate repetition probabilities even for golden prefixes. we propose selfcontrastive training (selfcont), which exploits the contrast with a premature checkpoint of the same model by penalizing its output when it incorrectly predicts repetition. experiments on two datasets show that selfcont effectively alleviates repetition while maintaining fluency by factoring out the undesired repetition behaviors highlighted by the premature checkpoint. besides the above analysis about overestimating token-level repetition probabilities during training, we also find that lms use longer-range dependencies to predict repetitive tokens than non-repetitive ones. it may explain why lms tend to fall into repetition loops @xcite . the problem may be solved by improving the modeling of long-range dependencies (e.g., increasing model sizes), which are left to future work. ## related work regarding the cause of the repetition problem, @xcite theoretically derived bounds of repetition probabilities of the first-order markov lm, although it is difficult to extend the bounds to general lms. another line of works attributed repetition to error accumulation during generation @xcite , while lms still prefer repetition given golden prefixes. we divide recent works that alleviate repetition into training-and decoding-based methods: one straightforward solution to repetition is blocking repetitive n-grams generations @xcite or penalizing probabilities of repetitive candidates @xcite . @xcite selected candidates that maximize the probability difference between different-sized models. sampling-based decoding methods are also shown effective in avoiding repetition, such as temperature sampling @xcite , top-k sampling @xcite , nucleus sampling @xcite , and typical sampling @xcite . although these methods reduce superficial repetition, it is unclear whether they utilize the underlying long-range dependencies to maintain coherence. ## empirical analysis neural networks (nns) are highly expressive to approximate arbitrary input-output mappings. using fourier analysis, @xcite showed the spectral bias of nns: they learn low-frequency components faster during training, which are less complex and vary globally without local fluctuation. our key hypothesis is that simple repetitive patterns may be such low-frequency components and learned by lms early. in this section, we first formulate lms ( §3.1), and then investigate the training dynamics ( §3.2) and the ability to model long-range dependencies ( §3.3) of lms. ## self-contrastive training we denote the premature checkpoint as f θ 0 , which frequently predicts repetitive tokens. formally, the selfcont algorithm is formulated as follows: implementation all baselines are implemented based on gpt2 base . we set the batch size to 16, the learning rate to 1e-4, and λ in eq. 3 to 4.0. for selfcont, we fine-tune gpt2 base for one epoch using mle and take the checkpoint as f θ 0 for both datasets. we use different p for different models based on the performance on the validation set. appendix b shows more details. metrics we use perplexity (ppl) under gpt2 xl to evaluate fluency, mauve @xcite to measure the similarity between golden and generated distributions, the token repetition ratios (r-l) to measure the ratio of tokens that appear in previous l tokens @xcite , and distinct (dn) @xcite to evaluate the n-gram diversity. the closer scores to the ground truth mean better quality for all metrics. results as shown in ## conclusion we present empirical studies on lms' preference for repetition by analyzing the training dynamics, which highlights their learning bias towards simple repetitive patterns. we propose penalizing outputs of a premature checkpoint during training, which effectively mitigates repetition while maintaining fluency. we also provide insight into why lms easily fall into repetition loops by showing their disability to model long-range dependencies. sampling-based decoding reduces repetition through randomness but not utilizing long-range dependencies. we believe that maximization-based decoding can also generate coherent texts without repetition by improving the modeling of long-range dependencies, which is left to future work.