Understanding the Repeat Curse in Large Language Models from a Feature Perspective (2025)

Abstract: Large language models (LLMs) have made remarkable progress in various domains, yet they often suffer from repetitive text generation, a phenomenon we refer to as the ”Repeat Curse”. While previous studies have proposed decoding strategies to mitigate repetition, the underlying mechanism behind this issue remains insufficiently explored. In this work, we investigate the root causes of repetition in LLMs through the lens of mechanistic interpretability. Inspired by recent advances in Sparse Autoencoders (SAEs), which enable monosemantic feature extraction, we propose a novel approach—”Duplicatus Charm”—to induce and analyze the Repeat Curse. Our method systematically identifies “Repetition Features” -the key model activations responsible for generating repetitive outputs. First, we locate the layers most involved in repetition through logit analysis. Next, we extract and stimulate relevant features using SAE-based activation manipulation. To validate our approach, we construct a repetition dataset covering token and paragraph level repetitions and introduce an evaluation pipeline to quantify the influence of identified repetition features. Furthermore, by deactivating these features, we have effectively mitigated the Repeat Curse.

## introduction large language models (llms) have demonstrated remarkable progress across various domains, from machine translation @xcite and open-ended text generation @xcite @xcite to interdisciplinary applications in social science behavior analysis @xcite @xcite and psychological * equal contribution. the order of these two authors follows alphabetical order of their last names. to address the issue, a few works identify the most important component in the network of the repeat curse from the mechanistic interpretability view. @xcite identified specific attention heads and layers that tend to copy the next token by examining the model's attention maps. building upon the layers, @xcite identified the repetition neurons by analyzing the activation outputs in the feed-forward network of each layer. compared to the model neurons, sparse autoencoders (saes) have been used in llms @xcite to achieve monosemantic units of analysis. sae maps the complex superposition of polysemantic neurons into monosemantic features. by regularizing activations, it ensures that only a small set of features are activated for each input, making the resulting features human-interpretable @xcite . due to its advances, many recent studies have leveraged sae to provide the most critical and human-understandable features for different tasks. for example, @xcite ; @xcite identified biologicalrelevant features and further utilized through sae; @xcite identified features related to inappropriate content such as nudity and violence. inspired by these works, we pose the following research question: can we leverage sae to identify the features that cause the repeat curse to give a better understanding? unlike the above-mentioned work, we cannot directly identify specific words or phrases that represent repetition. therefore, to identify these features, we propose the "duplicatus charm" (a magic spell inspired by harry potter) to induce the repeat curse. the main difficulties of our method are locating and identifying the "target of the spell", i.e., the most significant features. to address the first problem, we first analyze the logits to identify the layers that have a significant impact on predicting the next token as a repeat token @xcite . then, we explore the features in those layers by stimulating their activations through sae ( §4.3). for the second challenge, we design a pipeline to evaluate the effectiveness of the magic spell. first, we construct a repetition dataset containing two scenarios ( §4.1) and then select the appropriate repeat score for the task through the dataset ( §4.2). leveraging such a metric enables us to pinpoint the features that are most responsible for inducing repetition. we refer to these "targets of the spell" as "repetition features". finally, we cast a spell on the repetition features and scored them using the repeat score we identified. from a data perspective, this allows us to demonstrate whether our spell is effective while manually reviewing the texts with higher scores. we select three language models with different scales: gpt2-small @xcite , gemma-2-2b (team, 2024), and llama-3.1-8b @xcite . the results show that repetition features are primarily located in all three models' intermediate and final layers, suggesting a consistent pattern across different model architectures and scales. with the same coefficient, we demonstrate that activating these features increases repetition while other standard features do not. moreover, deactivating these features could mitigate the repeat curse, without exerting any detrimental effects on the model's performance. leveraging the human-readable nature of these features, we can also summarize the repetition feature's characteristics. overall, our contributions are as follows: ## related work repetation in language models. repetition in language models refers to the phenomenon where the generated text exhibits undesirable and redundant repetitions at various levels, such as tokenlevel and paragraph-level @xcite . although the cause of repetition in llms is still not fully understood, some scholars have proposed methods to mitigate repetition. @xcite introduced the decoding method of contrastive search, which encourages diversity while maintaining the coherence of the generated text. @xcite demonstrate that penalizing repetitions in the training data significantly alleviates the degeneration problem in neural text generation. @xcite presents a rebalanced encoding approach to address the issue of high inflow, reducing repetitions in both translation and language modeling tasks. however, the internal mechanisms of llms when they produce repetitive outputs remain insufficiently explored @xcite . @xcite proposed a psychological linkage between repetition and mathematics, and we want to find out if the similar repeat pattern mechanisms exist in the llm. ## sparse autoencoders sparse autoencoders (saes) provide us with an approximate decomposition of the model's activations into a linear combination of "feature directions" (sae decoder weights) with coefficients equal to the feature activations. the sparsity penalty ensures that, for any given inputs to the model, only a small fraction of features will have nonzero activations. thus, for any given token in any given context, the model activations are "explained" by a small set of active features (out of a large pool of possible features). here's how we perform this decomposition for activation x: the sum runs over all f features, effectively combining them to form the approximation of the original activation. here x is the reconstructed model activation, b @xmath0, w dec,@xmath1, and f i (x) denotes the activation of the i-th feature, i.e., ## method in the following sections, we introduce the pipeline of casting "duplicatus charm" (duc): ( 1 ) repeat pattern construction; (2) evaluation and selection of metrics; (3) repetition feature identification; (4) feature steering. see figure 2 for the method overview. ## conclusion in this paper, we take a perspective from the feature level and introduce a pipeline named "duplicatus charm" (duc). through this mechanistic interpretability method, we can identify the repetition features. by activating the repetition feature, we can induce the repeat curse, which was then evaluated through repeat scores and validated by humans in our experiment. and by deactivating these features will mitigate the repeat curse. furthermore, we summarize the common characteristics of repetition features across three models.