The B i GG en Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025)

Abstract: As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria-like helpfulness and harmlessness-which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on specific capabilities such as instruction following, leading to coverage bias. To overcome these limitations, we introduce the BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation. We apply this benchmark to assess 100 frontier LMs using five evaluator LMs. Our code, data, and evaluation results are all publicly available at https://github.com/prometheus-eval/prometheus-eval.

## introduction how can we systematically determine a language model's (lm) proficiency in a specific capability? accurately gauging these capabilities is crucial for pinpointing limitations and identifying areas for improvement in lms. the predominant approach has been to use summary measures, such as the lm's "helpfulness," as a proxy for all capabilities @xcite @xcite , or to employ easily measurable proxy tasks like multiple-choice questions @xcite @xcite . however, perceptions of what is considered helpful can vary from person to person @xcite @xcite , and high performance on classification tasks does not necessarily indicate that the lm possesses the ability to generate fluent text aligned with desired capabilities @xcite . instead, we are interested in directly assessing free-form outputs from lms with respect to specific capabilities. yet, determining if an output is "good" poses a challenge due to the subjective nature of evaluation. while humans can effortlessly discern key factors such as creativity, tone, and cultural sensitivities depending on the context, systematically evaluating these nuances remains a significant hurdle. inspired by human studies that underscore the importance of precise evaluation criteria in conducting effective interviews @xcite , we introduce the biggen bench, a principled generation benchmark designed to evaluate lms using fine-grained evaluation criteria tailored to each specific instance. this enables capturing subtle nuances and detailed variability in the response. as illustrated in figure 1 , when evaluating a rationale for a math problem, it is more instructive to examine whether the rationale logically addresses variable substitution rather than naively assigning a simplistic helpfulness score. specifically, the biggen bench evaluates 9 core capabilities of lms-namely instruction following, grounding, planning, reasoning, refinement, safety, theory of mind, tool usage, and multilingualism-across 77 tasks and 765 instances. moreover, compared to existing generation benchmarks that primarily inspect a narrow range of capabilities (e.g., instruction following) @xcite @xcite @xcite @xcite , our approach represents one of the first efforts to utilize evaluator lms across a broad spectrum of capabilities in a unified evaluation pipeline. employing 5 different evaluator lms, we evaluate 103 frontier lms ranging from 1 billion parameters to 141 billion parameters, as well as 14 proprietary lms. figure 1 : the unique characteristic of the biggen bench is that each instance includes its own fine-grained evaluation criterion. this enables more precise assessments of performance tailored to the specific characteristics and challenges of each instance. in contrast, coarse-grained evaluation criteria (e.g., helpfulness, harmlessness) @xcite ) may overlook subtle nuances and specific details, while domain-specific criteria @xcite might not account for the variability within individual instances. this paper is mainly divided into three parts. additionally, to offer further insights into advancing the frontier lms, we host two interactive websites: one displays visualizations of outputs from the 103 evaluated lms, complete with scores and detailed verbal feedback highlighting limitations and suggesting areas for improvement foot_0 ; the other features a leaderboard presenting scores across each capability as well as the average scores foot_1 . expanding lm-as-a-judge while existing generation benchmarks often focus narrowly on assessing a single capability, such as instruction following @xcite @xcite @xcite @xcite , some benchmarks evaluate other unique capabilities of lms, including reasoning, safety, tool usage, and multilingual capabilities @xcite @xcite @xcite @xcite @xcite . however, these benchmarks either 1) rely on exact match & similarity-based metrics, which fail to capture the finer details of responses-details that are more effectively recognized when using lms as evaluators-or 2) they require a simulator running in the background, which can be costly to prepare for supporting a wide range of tasks. our work adopts evaluator lms beyond instruction following across a broad spectrum of capabilities in a single evaluation pipeline, providing a detailed and thorough evaluation of lms. fine-grained evaluation of lms to replicate the flexibility and insightfulness inherent in human evaluation, prior works have proposed assessing lms using fine-grained evaluation criteria @xcite @xcite @xcite . the work most closely related to this paper is flask @xcite , which demonstrated that using 12 fine-grained evaluation criteria to assess lms, as opposed to relying on coarse-grained criteria like helpfulness and harmlessness, achieves a higher correlation with human evaluators. however, flask is built in a bottom-up manner; it samples instances from existing benchmarks and applies 12 high-level evaluation criteria to each, making it challenging to capture the intricate details of each instance-resulting in domain-specific evaluation criteria as shown in figure 1 . in contrast, the biggen bench is built through a principled top-down approach; we establish nine key capabilities to assess, organize tasks within each capability group, and assign specific evaluation criteria tailored to each instance, ensuring the evaluation is the most fine-grained at the instance level, as highlighted in figure 1 under instance-specific evaluation criteria. also, it is worth mentioning that while follow bench @xcite , info bench @xcite and ifeval @xcite also employ figure 3 : cross-validation results by human evaluators. to maintain high quality, we exclude instances that both evaluators determine to either not fit the task or have a bad reference answer or score rubrics. for instances that only one annotator marks as low quality, we iteratively revise them. ## biggen bench: the big generation benchmark the biggen bench is built in a principled top-down manner, maintaining a "capability -task -instanceevaluation criteria" hierarchy. in this section, we explain the evaluation protocol and construction process of the biggen bench. an explanation of the 9 capabilities and 77 tasks, along with the role of instance-specific evaluation criteria within each capability, is provided in appendix a. ## main results and analyses we present the evaluation results of 103 lms. detailed capability-wise and average scores for each response lm, assessed by five different evaluator lms, as well as the top scoring lms for each capability, are listed in appendix g. this section highlights key findings derived from these results. we examine the overall performance trends across differently sized pre-trained lms (denoted as "base lms"), post-trained lms (denoted as "chat lms"), the performance differences between corresponding base and chat lms, and the gap between open-source and proprietary lms. with findings from prior works, which suggest that using continuous metrics (e.g., a fine-grained 5-point likert scale score rubric) results in smooth, predictable changes rather than emergent trends @xcite @xcite . specifically, the correlation coefficients of the regression lines are high (0.68), indicating a strong linear relationship. furthermore, an r 2 value of 0.47 indicates that nearly half of the variability in performance improvements for base lms can be explained by model size scaling. notably, as an extension to the conventional understanding that the pre-training stage primarily enables larger base lms to store more knowledge effectively @xcite @xcite @xcite , our findings suggest that larger base lms also address diverse tasks that are not primarily knowledge-intensive with remarkable effectiveness @xcite . specifically, the scalability of model parameter size contributes to qualitative improvements in task performance across varied capabilities, including those that require complex cognitive abilities such as reasoning and tool usage (underlined in performance of chat lms is not only attributed to model size scaling. the performance of 61 chat lms is displayed in figure 5 , and summarized in the lower part of ## conclusion in this work, we presented the biggen bench, a benchmark designed to evaluate nine core capabilities of language models. we evaluated 103 frontier language models and studied how performance trends varied between pre-trained, post-trained, and proprietary models, particularly focusing on which capabilities improved with model scaling or post-training. furthermore, we demonstrated that evaluator lms can reliably assess a broad set of capabilities, as confirmed by significant correlations with humans.