Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance (2025)

Abstract: Existing benchmarks are becoming saturated and less effective in evaluating model performance due to factors such as data contamination and the advancing capabilities of the Large Language Models (LLMs). This paper introduces EMDM (Enhanced Model Differentiation Metric), a novel weighted metric designed to revitalize existing benchmarks. EMDM implements a weighting schema for samples based on their complexity and requisite knowledge, utilizing the performance of a baseline LLM in two experimental setups: “Unguided”, where the model has no prior exposure to test samples, and “Guided”, where the model has prior knowledge about the desired answer. This schema is leveraged in an optimization objective to assign weights to test samples, distinguishing instances of varying complexity. EMDM accounts for both answer correctness and the depth and accuracy of reasoning, offering a more nuanced evaluation of model performance. By weighting test examples based on their required reasoning and knowledge, EMDM achieves a distinguishing range of evaluation scores of 46% among various LLMs, compared to just 17% with traditional exact match (EM) metrics, thereby highlighting the saturation of current evaluation methods.

## introduction large language models (llms) have made significant strides in nlp tasks @xcite , with benchmarks like mmlu @xcite , arc @xcite , and gsm8k @xcite playing a crucial role in tracking progress. however, model performance on these benchmarks has plateaued @xcite , limiting their usefulness in differentiating models. creating more complex benchmarks is resourceintensive and prone to inaccuracies in synthetic data generation @xcite . we introduce the enhanced model differentiation metric (emdm), a novel approach to re-* work done during internship in servicenow vitalizing existing benchmarks. while chain-of-thought (cot) reasoning @xcite has proven beneficial for fine-tuning @xcite , its potential in evaluation is underutilized. emdm addresses this by combining cot correctness with traditional metrics like exact match (em) and accuracy, emphasizing challenging examples that require deep reasoning. our methodology, illustrated in figure 1 , uses a baseline llm with in-context learning (icl) to generate responses under two setups: a guided prompt (with the test example and the desired response included, simulating contamination) and an unguided prompt (standard icl). if a model fails under the guided setup, it indicates limited reasoning for that sample. emdm assigns higher weights to such challenging instances as they could be more differentiating (red highlights) and lower weights to easier ones (green highlights). weights are calculated once per benchmark using a baseline llm and seamlessly integrated into the evaluation process by updating only the final scoring. our contributions in this paper include: • we introduce emdm, a weighted metric combining answer correctness and cot reasoning to emphasize challenging test instances, enhancing model differentiation. our experiments demonstrate the effectiveness of the optimized weights in increasing the llm performance differentiation. ## background & related work in-context learning (icl) allows llms to adapt to new tasks by showing a few relevant examples within the input prompt, without the need for further training or fine-tuning @xcite . in practice, icl is broadly applied for task learning (acquiring new skills) and task retrieval (activating pre-trained knowledge) @xcite ; lin chain of thought (cot), introduced by @xcite , has improved llm performance through step-by-step reasoning. while cot judgment has been successfully used in llm posttraining methods such as reinforced fine-tuning (reft) @xcite to generate preference data pairs, its role in evaluation remains unexplored. emdm fills this gap by incorporating cot quality into llm evaluation. human annotation is time-consuming and unscalable. llm-based evaluation has been widely adopted @xcite , showing high agreement with human judgment @xcite . moreover, @xcite introduce mr-ben, a high quality benchmark designed to assess the capability of llms to serve as a judge for cot reasoning samples. their analysis evaluates several state-of-the-art llms across diverse domains, including mathematics, physics, coding, and logical reasoning, etc., and reveal that gpt-4o and gpt-4 series models consistently outperform other llms as reliable evaluators of cot correctness. building upon these motivations, we use llm based judges to evaluate the accuracy of cot responses, generated by baseline models. some benchmarks, such as wildbench @xcite and flask @xcite , group dataset samples by difficulty using llms. however, the reliability of llm-based difficulty classification remains largely unexplored. in contrast, emdm categorizes samples based on baseline llm performance accounting for both cot reasoning and answer correctness, without directly classifying difficulty (see appendix a.6 for comparison of flask and emdm). guided prompt: includes the test example and the desired answer in the few-shot prompts, simulating contamination and identifying cases where the model fails even with direct guidance and access to the correct answer. ## emdm formulation all responses generated by the two prompt types require cot. the generated cot and answers are then assessed by (figure 1 -step (2)): exact match: checks whether the model's response matches the correct response, thereby assessing the accuracy of the final answer. llm-judge @xcite : assesses whether the reasoning steps effectively support the conclusion, regardless of the final answer's correctness (see appendix a.4). we use gpt-4 to critique the cot generated by the baseline. using the above process, we generate 4×4 = 16 categories (figure 1 weight optimization: weights, denoted by ŵ, are calculated for each of the 16 categories to maximize the pairwise difference between the llms, considering all llm pairs, a and b, excluding the baseline model r, by solving the following optimization: the first term regularizes the weights toward the lower bound l, while the second term, capturing pairwise score differences, pushes them toward the upper bound u . therefore, the combined objective assigns higher weights to categories with larger score differences, due to their stronger gradients, and lower weights to those with smaller differences. this ensures that the weights highlight categories where the models exhibit the most variation, enhancing the differentiation between them. these weights are calculated once and will be used in llm benchmarking. using the calculated weights for samples in each category ( ŵg k ), emdm is defined as a weighted average of the benchmark sample scores: ## conclusion we introduced the enhanced model differentiation metric (emdm), a novel metric that enhances model differentiation through guided and unguided promptings. emdm combines cot judgment and answers correctness to assess reasoning depth. in the guided icl setup, where the correct answer is included in the prompt, models do not always produce correct responses or reasoning paths (i.e., cot), leading to varied behaviours across samples. emdm leverages these differences to identify challenging instances and assign appropriate weights. tested across benchmarks, emdm improves performance separation, especially in saturated benchmarks. for difficult benchmarks, adjusting weight bounds and baseline model selection can further enhance separation (appendix a.1).