S im BA : Simplifying Benchmark Analysis Using Performance Matrices Alone (2025)

Abstract: Modern language models are evaluated on large benchmarks, which are difficult to make sense of, especially for model selection.Looking at the raw evaluation numbers themselves using a model-centric lens, we propose SimBA, a three phase framework to Simplify Benchmark Analysis. The three phases of SimBA are: stalk, where we conduct dataset & model comparisons, prowl, where we discover a representative subset, and pounce, where we use the representative subset to predict performance on a held-out set of models. Applying SimBA to three popular LM benchmarks: HELM, MMLU, and BigBenchLite reveals that across all three benchmarks, datasets and models relate strongly to one another (stalk). We develop an representative set discovery algorithm which covers a benchmark using raw evaluation scores alone. Using our algorithm, we find that with 6.25% (1/16), 1.7% (1/58), and 28.4% (21/74) of the datasets for HELM, MMLU, and BigBenchLite respectively, we achieve coverage levels of at least 95% (prowl). Additionally, using just these representative subsets, we can both preserve model ranks and predict performance on a held-out set of models with near zero mean-squared error (pounce). Taken together, SimBA can help model developers improve efficiency during model training and dataset creators validate whether their newly created dataset differs from existing datasets in a benchmark. Our code is open source, available at https://github.com/nishantsubramani/simba.

## introduction the rapid expansion of language model (lm) benchmarks has resulted in an overabundance of evaluation datasets. however, the relationships among these datasets remain poorly understood. current evaluation methods primarily focus on * equal contribution overall model win rates or simple aggregate measures, which fail to provide fine-grained insights into dataset characteristics and model performance trends (liang et al., 2023). one approach that the community has taken to mitigate this problem is to look at instance-level predictions and construct coresets, where each individual dataset in a benchmark is subsampled using various heuristics (rodriguez et al., 2021; perlitz et al., 2024; zouhar et al., 2025). this resulting subset is used as a proxy for the entire benchmark. coreset identification, often done using influence functions (koh and liang, 2017; schioppa et al., 2021), has many drawbacks: integration into an already existing evaluation framework is hard, weak statistical signal hinders generalization, and collecting instance-level predictions across many models may be computationally infeasible (chatterjee and hadi, 1986). our work looks to uncover a more structured and simplified understanding of benchmarks through an analysis of the datasets and models directly from the performance matrix without collecting any instance-level predictions. our framework to simplify benchmark analysis is called simba and has three phases: ## stalk: dataset & model comparison a benchmark is represented as a matrix @xmath0, where m is the number of models and d is the number of datasets. b can have missing values. different datasets evaluate different metrics, which often are scaled differently (e.g. classification vs. generation), so we normalize b. for every dataset d i with random chance performance x random , we modify every observation x 1 , . . . , x m to be: note these new x j values correspond to percent above random chance. we use this normalization for all analysis. @xcite moreover, all values are further normalized to be within the interval [0, 1], where 0 corresponds to random chance and 1 corresponds to the maximum possible performance. ## prowl: representative dataset discovery equipped with an understanding of a benchmark, our next step is to identify a target. more specifically, we want to discover a representative subset of a benchmark s from the performance matrix @xmath1, where m is the number of models and d is the total number of datasets. ## pounce: performance prediction equipped with the results of the first two phases of our analysis pipeline, we predict the performances directly using a representative subset. since the representative subset s is just a subset of the full benchmark b, we evaluate whether different regression based approaches can predict d \ s (i.e., the subset of d that is not in s) from s alone. one general approach for matrix prediction is singular value decomposition (svd; candes and recht (2009)). however, svd works only on a partially observed matrix, where rows (models) and columns (datasets) have at least one observation. in a realistic setting, we want to use our subset s to predict the other dataset performances d \ s on entirely new models, so svd would not be immediately applicable. as a result, we focus on three regressor types: ridge regression, knn regression, and mlp regression. ## analysis our evaluation framework relies on point estimates of the performances of models on individual datasets. as such, robustness to noise is a critical consideration when evaluating different approaches of performance prediction. we evaluate the robustness of performance prediction by perturbing the training set of the benchmark matrix b train as b ′ @xmath2, σ 2 ) where µ is mean and σ is the noise level parameter. for all noise perturbations we use µ = 0 and σ 2 = 0.05 or σ 2 = 0.1. this evaluation framework enables us to quantify the stability of our methods under varying noise conditions. as shown in figure 3 , generally, error rates for all methods increase with greater noise. however, we observe that both ridge and knn regression achieve low error rates under both noise conditions, across all three benchmarks (auc-@xmath3.03). ridge consistently maintains auc-mse values of 0.013 for all benchmarks and noise conditions. predictably, the mlp systems are the least robust, but maintain similar error rates to the no noise version. in other words, the mlp systems just are not great at performance prediction. ## related work the nlp community increasingly evaluates on large benchmarks (srivastava et al., 2022; li et al., 2023; liang et al., 2023). some approaches attempt to make evaluation more efficient by doing instance-level reduction (vivek et al., 2023; polo et al., 2024; perlitz et al., 2024). magnusson et al. (2025) look at a loosely related problem and use small scale experiments for pretraining data selection. they, too, require instance-level information, but are focused on analyzing how training on different subsets of pretraining data affect performance rather than looking at a representative set of pretraining corpora. we differ in that we focus on aggregate metrics and do not need access to any instance-level information for efficiency and performance prediction. as mentioned earlier, most work looks at identifying coresets of benchmarks at the instance-level and there exist numerous methods to do this across many different application areas (lewis and catlett, 1994; killamsetty et al., 2021; paul et al., 2021; moser et al., 2025). ye et al. (2023) is one of the few works that, like us, goes beyond instance-level work and tackles the performance prediction task using simple regressors on bigbench. however, their approach is not as lightweight as ours because they requires features of the model and datasets for accurate prediction. from the field of psychometrics and measurement theory, there exists the idea of convergent and divergent validity (campbell and fiske, 1959). convergent validity suggests that metrics measuring similar underlying constructs should correlate highly with each other. conversely, discriminant validity indicates that metrics capturing fundamentally different aspects should show minimal correlation. xiao et al. (2023) propose metriceval, a framework motivated by measurement theory to conceptualize and evaluate the reliable and valid of natural language generation metrics. ## conclusion we propose a three phase approach to simplify benchmark analysis called simba: stalk (dataset & model comparison), prowl (representative set discovery), and pounce (performance prediction). using our approach, we analyze the helm, mmlu, and bigbenchlite benchmarks. our analysis shows that models and datasets alike correlate well with one another (stalk). additionally, using algorithm 1, we can identify representative subsets with 1, 1, and 21 datasets respectively that achieve a greater than 95% coverage (prowl). these representative sets preserve the original model ranks on the benchmark and can be used to predict performance on held-out models with negligible error (pounce). furthermore, simba can be used by lm practitioners and dataset developers directly to reduce evaluation costs and validate dataset uniqueness.