RAP : A Metric for Balancing Repetition and Performance in Open-Source Large Language Models (2025)

Abstract: Large Language Models (LLMs) have significantly advanced natural language processing, but content repetition in open-source LLMs remains a critical challenge that adversely affects user experience. The repetition penalty parameter (RPP) aims to mitigate this issue by preventing repeated content generation, but excessive use of RPP can compromise the overall quality. In this paper, we propose Repetition-Aware Performance (RAP), a novel evaluation metric that quantifies and integrates repetition penalty into the assessment of model performance, enabling tuning of RPP. We evaluate our approach using twelve open-source LLMs, ranging from 2 billion to 70 billion parameters, tested on question answering and machine translation tasks across three datasets with varying prompting techniques. Experimental results show that RAP effectively tunes RPP, helping to identify a trade-off value that significantly reduces repetition while minimizing performance loss. Upon acceptance, we will release the code and the dataset of generated text, providing a valuable resource for further research on repetition detection and LLMs evaluation.

## introduction the rapid advancement of large language models (llms) has transformed natural language processing, enabling remarkable text generation and comprehension capabilities. however, open-source llms often generate repetitive content, leading to extensive empty lines or recurring sentences, which undermines text quality and fluency. this is particularly problematic in tasks requiring coherent and contextually relevant responses, such as conversational chatbots. to address this, the repetition penalty parameter (rpp) is employed during the sampling process to reduce redundant outputs by penalizing previously seen tokens @xcite . while effective, excessive application of rpp can result in incomplete or fragmented responses, ultimately diminishing user satisfaction. this paper explores the relationship between rpp and its effect on generated text. while parameters like temperature and top-k sampling are optimized using methods like random or grid search, these fail for tuning rpp since traditional metrics, such as precision and bleu scores @xcite , overlook repetition. to address this, we propose a metric called repetition-aware performance (rap), which quantifies repetition and penalizes it in the model's performance evaluation, allowing for the tuning of rpp. figure 1 illustrates tuning rpp for llama-3-8b using the original evaluation metrics (original) and our proposed metric. for the webqsp dataset @xcite , repetition ratio (rr) is reduced by 93.1% but the performance only drops by 3.7%. for the ms marco dataset @xcite , rr drops by 73.9% with only 3.7% of performance drop. tuning rpp using rap can significantly reduce the rr with minimal performance loss. more examples can be found in appendix @xcite ). to compute rap, we introduce an efficient algorithm called reda-repetition detection algorithm-designed to identify all repeating patterns, including non-word character (nwc) and text repetitions, along with their corresponding repetition lengths. additionally, we define the repetition ratio (rr), which measures the proportion of repeated content relative to the total text length. rr is applied to penalize the model's performance using a cubic penalty formula, as detailed in section 3. figure 2 illustrates computing rap with a running example. although the accuracy is 100%, the content contains many repeated elements, resulting in a low rap score of 24.63%. we conduct experiments on three datasets encompassing question answering (including knowledge base question answering-kbqa-and openended qa) and machine translation tasks. we evaluate twelve open-source llms ranging from 2 to 70 billion parameters. we also examine the effect of repetition suppression for different prompting strategies including retrieval-augmented generation (rag) @xcite and non-rag, combining with generic prompt and chat template. our study reveals that increasing rpp typically reduces repetition but often results in performance degradation, highlighting the importance of tuning rpp. the effects of rpp differ across various models, datasets, and prompting techniques. additionally, we find that using chat templates is effective in reducing repetition for qa task. our experimental results demonstrate that rap can effectively tune rpp, helping to identify a trade-off value that significantly reduces repetition while minimizing performance loss. these findings offer insights into optimizing rpp for performance and repetition suppression in open-source llms. this paper presents three key contributions. first, we propose rap, a novel evaluation metric based on the repetition ratio that quantifies and integrates repetition issues into model performance, enabling the tuning of hyperparameters such as rpp. we also develop an efficient repetition detection algorithm (reda) algorithm for detecting both nwc and text repetition, addressing the key challenge in open-source llms. second, we conduct comprehensive experiments on twelve open-source llms ranging from 2 to 70 billion parameters, across two tasks-qa and mt-using three datasets and multiple prompting techniques, including rag and non-rag methods, with the use of generic prompt and chat template. these experiments provide valuable insights into reducing repetition while maintaining performance and demonstrate the effectiveness of tuning rpp using rap. third, we publish all source code and data generated during these experiments at github foot_0 , which includes a significant amount of text produced by open-source llms. this dataset is a valuable resource for future research in developing advanced repetition detection methods, assessing llm capabilities in entity extraction, and evaluating performance in question answering and machine translation tasks. ## the proposed method the core concept of rap is to incorporate a penalty for the model's performance based on the degree of repetition in the generated response. to achieve this, we propose an efficient algorithm, reda, for detecting and quantifying repetition in text. we introduce the notion of the repetition ratio (rr), which measures the proportion of repeated content. additionally, we define repetition-aware performance (rap), a metric that integrates repetition penalties into standard performance evaluations, enabling tuning of the repetition penalty parameter. given a text sequence generated by an llm, we first detect non-word characters (nwc) repetition, that includes white spaces and non-alphanumeric characters such as punctuation and symbols. a sequence has nwc repetition if it contains at least k consecutive identical nwc. after carefully checking the outputs of various llms, we choose @xmath0. we then remove these repeated nwc and detect text repetition. ## conclusion this research tackles the challenge of reducing repeated content generation while preserving overall performance in open-source llms. we propose the repetition-aware performance metric, rap, which quantifies and integrates repetition penalty into model evaluation. conventional metrics often overlook repetition, making it difficult to tune the repetition penalty parameter (rpp). in contrast, rap enables effective tuning of rpp and can be easily applied to other repetition methods as well. we evaluated rap on twelve open-source llms using three datasets across qa and mt tasks, with different prompting strategies. results show that higher rpp values generally reduce repetition but can significantly impact performance. this phenomenon emphasizes the need of tuning rpp. the experiments also demonstrate the capability of using rap for identifying the rpp value that efficiently reduce repetition while minimizing performance loss. we also highlight the severity of the repetition issues in the generated content, calling for greater attention from the research community on this matter. our experiments offer valuable insights into managing repetition in open-source llms, including a recommendation to use chat templates for rag prompting. we will release the source code and all experimental data, offering a large dataset of llm-generated content for future research on repetition issues and llm capabilities in qa and mt tasks.