Chatbot Arena Estimate: towards a generalized performance benchmark for LLM capabilities (2025)

Abstract: In industrial LLM development, evaluating large language models (LLMs) is critical for tasks like benchmarking internal models and detecting regressions during fine-tuning, but existing benchmark aggregation methods, such as Elo-based systems, can be resource-intensive, public facing, and time-consuming. Here, we describe Chatbot Arena Estimate (CAE) , a practical framework for aggregating performance across diverse benchmarks. The framework, developed and widely adopted within our organization, addresses the need for quick, accurate, and cost-efficient evaluations of LLMs. CAE generates two primary metrics: a “Goodness” score (answer accuracy) and a “Fastness” score (cost or queries per second, QPS). These metrics allow for model ranking both overall and within specific subdomains, enabling informed decisions during model iteration and deployment. We demonstrate CAE’s effectiveness by comparing it with existing benchmarks, including the full Chatbot Arena and the MMLU leaderboard. Notably, our approach achieves higher Pearson correlation with Chatbot Arena Elo scores than MMLU’s correlation with Chatbot Arena Elo scores, validating its reliability for real-world LLM evaluation.

## introduction the landscape of large language model (llm) evaluation is rich with specialized benchmarks. they target domains such as logic @xcite , math @xcite , law @xcite , linguistic understanding @xcite , factual recall @xcite , and general performance @xcite . however, for many decisionmakers in industry, the proliferation of benchmarks can complicate the model selection process. indeed, there exists a need for a single, unified metric for rankings and comparisons. the chatbot arena elo score @xcite has emerged as the gold industrystandard ranking of quality, but is costly, public facing, and lengthy. why the need for a single quality metric? through developing models in a large tech organization, we have found: (1) high level investment decisions between different models requires single, generalized numbers, (2) a comparison of quality and latency creates a pareto frontier which can guide decision making by elucidating gaps in the fronier, (3) fine tuning smaller models for specific purposes requires generalized quality tests to detect skill regression, (4) technical teams need quick, cheap, and general metrics to quickly iterate on model versions. in this paper, we introduce chatbot arena estimate (cae), a practical aggregation framework originally developed and widely used in a leading tech company to evaluate internal llms. cae produces two numbers: a general model quality score (i.e. "goodness"), and a latency score (i.e. "fastness".) it consists of a sparse aggregation of public benchmarks. as shown in figure 1 , our framework results in a simple trade-off between goodness and fastness, enabling stakeholders to make informed decisions quickly and effectively. to our knowledge, we are the first to attempt to directly estimate chatbot arena by systematically reducing different benchmarks into one interpretable number while also focusing on computational and financial efficiency of evaluation. we evaluate fourteen models considered state of the art, selected for disjointedness, that are currently supported for production on easy to access platforms, explicitly providing the correlation between our metric and chatbot arena elo scores. our metric has a higher correlation than others, including the wellknown mmlu. our target audience includes resource constrained teams -such as those in smaller companies, universities, or startups -that lack access to extensive compute resources, public leaderboards, or large-scale human evaluations. figure 1 : outcome of our chatbot arena estimate benchmark applied to thirteen publicly facing language models. here, the x axis is the "performance" (queries per second), which we express on the log scale, and the y axis is "goodness" (our benchmark's outcome). the error is 95% confidence intervals described in section 3.4. ## related work evaluating large language models (llms) is critical as their applications expand across diverse domains @xcite @xcite . one prominent framework is the chatbot arena, which employs competitive rankings based on pairwise model comparisons. inspired by the elo rating system, this approach dynamically evaluates models by ranking them based on performance in headto-head tasks @xcite . while widely used, elo-based systems have significant critiques @xcite : (1) the breadth of questions is difficult to represent effectively, as different model matchups receive different prompts, creating opaque and non-standard rankings. (2) matchups between models of varying quality can yield misleading results-poor-quality pairings may appear similar to high-quality ones. (3) addressing these limitations often requires extensive computational or human resources, as seen in chatbot arena, which depends on o(10k) votes per top model. (4) elo systems struggle to track a model's evolution over time, making static benchmarks a preferred tool for routine evaluations. despite its challenges, chatbot arena has established itself as a central competitive evaluation method, underpinned by the bradley-terry model @xcite . emerging sparse benchmarks, such as metabench @xcite and tinybench(polo et al., 2024), aim to streamline evaluation by focusing on a smaller subset of tasks. however, these methods fall short in correlating with chatbot arena's comprehensive evaluation approach. for instance, metabench draws from only six benchmarks, while tinybench references just mmlu. our benchmark uniquely provides sparse evaluations while directly estimating chatbot arena performance, incorporating data from 23 benchmarks for broader coverage. another important paradigm is llm-as-a-judge, where llms are used to evaluate the outputs of other models. this approach has been adopted by benchmarks like arena-hard-auto @xcite and al-pacaeval 2.0 @xcite . while promising, this methodology raises concerns about potential biases and objectivity, as llm judges may share the same limitations as the models they assess @xcite . static, ground-truth-based benchmarks remain a cornerstone of llm evaluation. these benchmarks often rely on fixed datasets across domains such as mathematics, science, coding, and reasoning. notable examples include mmlu @xcite , math @xcite , gsm-8k @xcite , humaneval @xcite , big-bench (bench authors, 2023), hellaswag @xcite , and agieval @xcite . comprehensive collections such as helm @xcite provide a broader perspective. despite their strengths, static benchmarks are limited in adaptability and may fail to reflect the dynamic nature of llm performance. finally, dynamic evaluation (dyval 2) introduces a psychometric approach, grouping benchmark questions into distinct cognitive domains while employing heuristics to prevent contamination. techniques such as shuffling multiple-choice answers or introducing incorrect options test whether llms rely on memoriza-tion @xcite . these strategies underscore a shift toward adaptive and nuanced evaluation methods, addressing the challenges of traditional static benchmarks in keeping pace with rapid advancements in llm capabilities. ## model evaluation in order to evaluate models, we used a runpod console to inference six open source models on a100 gpus: yi-1.5-34b-chat, llama-3.1-70b-instruct, quen2-72b-instruct, phi-3-small-8k-instruct, gemma-2-9b-it, gemma-2-27b-it, and qwen2-72b-instruct, and the following eight proprietary models on their own public facing apis: gpt-4o-2024-05-13, gemini 1.5 pro 001 05-24, gemini 1.5 pro 08-27, gemini 1.5 flash 08-27, gpt-4-01-preview (strawberry), mistral-large 2, claude 3.5 sonnet 2024-06-20, and claude 3 opus 2024-02-29. queries-per-second is one good stand-in for latency, and to compare apples-to-apples, a company may use the architecture or ones available to it normalized by price. for demonstration purposes, we present the qps measured across public facing architectures by simply timing the response rate of every prompt that was sent to the external servers for our specific benchmark questions. please note that another set of benchmark questions, including longer and multimodal questions, may have garnered a different qps ordering. ## conclusion in this work, we introduce cae, a benchmarking framework that aggregates a minimal set of benchmarks in order to efficiently generalize an agent's capabilities. our approach prioritizes factual, falsifiable questions, such as "what is the height of the eiffel tower?" over more subjective prompts like "compose a beautiful haiku." we intend our focus on factuality to ensure reproducibility and enable objective, quantifiable evaluation metrics, with an eye towards consistent performance assessments. our target audience includes resource-constrained stakeholders, such as modeling managers at smaller companies or universities, who may lack access to extensive human evaluations, large-scale testing, or public ratings like those solicited in chatbot arena. by providing a lightweight evaluation approach, we enable such users to select models that align with their specific requirements in terms of quality and latency. additionally, this framework serves as a guide for those just starting to work with llms, offering a practical tool for navigating trade-offs between different models. it is out of the scope of our paper to suggest specific directions for the open source community to push model development in, considering the thirteen models we profile, but decision makers could use frameworks like ours to make decisions like this. in addition, we recognize that our framework has several limitations. first, the focus on multiple choice questions appears an idiosyncratic choice given how little they resemble the ways users actually engage with llms. while this limitation is mitigated by the strong correlation we see with chatbot arena, it still raises questions about the generalizability across use cases. furthermore, our benchmark does not include any direct tests of linguistic skills or sentiment analysis. in the future, we aim to extend this benchmark to cover multimodal tasks and more complex linguistic skills, such as text summarization. additionally, we plan to incorporate dynamic, evolving benchmarks to mitigate the risks of dataset contamination, further improving the robustness and relevance of future evaluations.