Repetition Neurons: How Do Language Models Produce Repetitions? (2025)

Abstract: This paper introduces repetition neurons, which can be regarded as “skill neurons” responsible for the repetition problem in text generation tasks. These neurons are progressively activated more strongly as repetition continues, indicating that they perceive repetition as a task to copy the previous context repeatedly, similar to in-context learning. We identify these repetition neurons by comparing activation values before and after the onset of repetition in texts generated by recent pre-trained language models. We analyze the repetition neurons in three English and one Japanese pre-trained language models and observe similar patterns across them.

## introduction while text generation with llms such as @xcite has been actively studied, the issue of repetition remains a fundamental challenge @xcite . specifically, repetition is particularly problematic under greedy generation, which is often used when reproducibility must be guaranteed @xcite . many researchers have tackled this problem by analyzing repetition @xcite and developing techniques to mitigate repetitive outputs @xcite @xcite . some works specifically focus on attention heads, such as induction heads, framing repetition as a key mechanism for in-context learning @xcite @xcite . however, the internal mechanisms of generative models that produce repetitive outputs remain insufficiently explored @xcite . we focus on the neurons of transformer language models @xcite @xcite that detect repetition in inputs and trigger repetitive outputs in text generation. we refer to these neurons as "repetition neurons" following @xcite . we hypothesize that the repetition neuron is a "skill neuron" @xcite that prompts the model to generate repetition as a task of copying the previous context, akin to "task vectors" @xcite found in in-context learning @xcite . we propose a method to identify repetition neurons by comparing activation values in the input ranges before and after the onset of repetition ( §3.1). as shown in figure 1 , repetition neurons tend to become progressively more strongly activated as the repetition sequence continues. we inspected repetition neurons in three english and one japanese pre-trained language model. our experimental results show that repetition neurons appear in both intermediate and final layers ( §3.2). furthermore, we demonstrate that deactivating these neurons suppresses the output probabilities of repeated tokens ( §4.1), while activating them increases these probabilities ( §4.2) foot_0 . in addition, we highlight the relationship between repetition neurons and induction heads ( §5). 2 general setting ## intervention to repetition neurons if our hypothesis that the repetition neurons invoke repetition is correct, we should be able to control the repetition problem by intervening with these neurons @xcite . ## conclusion we proposed a method to identify the repetition neurons that contribute to the repetition problem in text generation. these neurons are located in both the intermediate and final layers of the transformer, similar to skill neurons and task vectors. our experimental results show that by intervening in the activity of these repetition neurons, we can control the occurrence of repetitive outputs.