P redicta B oard: Benchmarking LLM Score Predictability (2025)

Abstract: Despite possessing impressive skills, Large Language Models (LLMs) often fail unpre-dictably, demonstrating inconsistent success in even basic common sense reasoning tasks. This unpredictability poses a significant challenge to ensuring their safe deployment, as identifying and operating within a reliable “safe zone” is essential for mitigating risks. To address this, we present PredictaBoard, a novel collabo-rative benchmarking framework designed to evaluate the ability of score predictors (referred to as assessors) to anticipate LLM errors on specific task instances (i.e., prompts) from existing datasets. PredictaBoard evaluates pairs of LLMs and assessors by considering the rejection rate at different tolerance errors. As such, PredictaBoard stimulates research into developing better assessors and making LLMs more predictable, not only with a higher average performance. We conduct illustrative experiments using baseline assessors and state-of-the-art LLMs. PredictaBoard highlights the critical need to evaluate predictability alongside performance, paving the way for safer AI systems where errors are not only minimised but also anticipated and effectively mitigated. Code for our bench-mark can be found at https://github. com/Kinds-of-Intelligence-CFI/PredictaBoard

## introduction a key component of safety in high-stakes scenarios is knowing the operating conditions of the system in use, namely, the specific task instances in which the system succeeds @xcite @xcite ). consider, for example, two autonomous driving systems. system a always detects pedestrians correctly in day time but has a predictably high failure rate at night, allowing for safety measures (e.g., requiring a human intervention). system b, on the other hand, although generally better performing, has random, unpredictable failures that leave the driver uncertain about when to intervene. even if system b has higher performance across the typical range of conditions encountered by drivers, system a is safer due to its predictability. this is visualised in a q&a domain in figure 1 . this principle extends to frontier ai systems, such as large language models foot_0 (llms), which are rapidly being integrated into high-stakes scenarios @xcite @xcite . their growing adoption introduces the potential for high-consequence harm -significant negative impacts on individuals, organisations, or society as a whole-thus requiring robust safeguards to ensure both high performance and safety @xcite . here as well, a key component to risk mitigation is validity predictability, namely, enabling users to anticipate and reject inputs that cause the ai systems to produce undesirable outputs or outcomes @xcite @xcite ; sec 2 discusses this more in detail. however, humans struggle to predict when llms will be correct @xcite , can be biased by prior interactions @xcite and even struggle to evaluate explanations provided by llms @xcite . additionally, alignment using reinforcement learning from human feedback (rlhf, @xcite and other techniques lead to llms potentially deceiving humans more often @xcite , resulting in unpredictable and unsafe behaviour @xcite . technical solutions, including uncertainty quantification (uq, @xcite or external assessors @xcite have been proposed to predict the success of and llm on individual cases (see figure 2 ). however, uq methods require the input to be passed though the llm, are not always reliable @xcite and are degraded by rlhf @xcite ; on the other hand, assessors' performance is limited by llms' idiosyncrasies, such as prompt perturbation leading to wildly different results @xcite ) and success on difficult task instances not guaranteeing success on easy ones even within the same domain @xcite . most importantly, no standardised framework exists to track performance of validity prediction methods and thus stimulate research into better ones and more predictable llms. to address this, we introduce predictaboard, the first collaborative benchmarking framework that jointly assesses llm performance and the predictability of that performance. the subjects of predictaboard are llm-assessor pairs, with the latter predicting the llm's score on each benchmark instance (i.e., prompt). solely ranking pairs based on assessor's performance would however lead to the llm that always fails (whose score is perfectly predictable) dominating. hence, as metric, predictaboard combines the llm performance and the quality of the score predictions into an accuracy-rejection curve (arc, nadeem et al., 2009) representing the llm's performance on instances for which the assessor predicts probability of success above different threshold values. from this curve, practitioners can determine the threshold for their error tolerance and determine the size of the predictably valid foot_1 operating region (see fig 1 ). alternatively, the area under the arc can be used to combine performance of llm-assessor pairs over all error tolerance levels (sec. 4.2). to compete in predictaboard, we require assessors to not rely on the llm's outputs. while these could be used without comparing them to ground truths, avoiding them entirely eliminates implicit reliance on the latter, makes assessors robust to manipulation by the llms (similarly to how relying on human feedback makes llm outputs more persuasive @xcite and avoids wasted computations on instances where the llm will predictably fail. however, we allow the use of internal llm activations to the input. essentially, this requirement distinguishes assessors from filters or verifiers of llm answers and makes predictaboard representative of the real world, where the ground truth is unknown. the current version of predictaboard focuses on success scores, but the same framework can be applied to safety and alignment benchmarks @xcite . moreover, the predictaboard framework can be applied to any other type of ai system, such as llm agents, embodied agents and vision systems. ultimately, the long-term vision of predictaboard is to shift ai benchmarking to consider predictability alongside performance, aligning it with traditional risk assessment practices that evaluate both factors @xcite @xcite . ## ai predictability and safety in this paper, we focus on instance-level predictability of validity (see app. a for a formal definition). for llms, an instance is a specific input prompt, and validity can refer to any performance indicator (e.g. success or safety scores, or other metrics for biased or unethical outcomes). as shown in figure 1 , for each llm we build one or more assessors 3 @xcite to predict the validity of the llm output on a specific instance. validity predictability contributes to safety in high-stakes environments, filtering out the inputs that lead to unacceptable behaviour can prevent harms-inputs can be rejected, redirected to a more reliable system, or supervised by humans. thus, reliable assessors provide a cost-efficient safety layer of safety that complements built-in model mitigations and post-generation filtering (hernández-3 also known as "rejector" @xcite . @xcite . in terms of developing safer models, scalable oversight methods @xcite @xcite already use weaker ai models for high-quality feedback for training and overseeing complex ai models. here, assessors can signal inputs where supervision may fail, or flag prompts that induce convincing but invalid outputs (e.g., in rlhf settings). finally, while red-teaming efforts (hacking policy council, 2023) are directed at finding harmful inputs, the ability to predict such inputs enables scalable vulnerability detection and design guardrails. while this could also be used by bad actors to exploit system vulnerabilities, the ability of safety researchers to more effectively address those vulnerabilities before deployment likely leads to a positive net effect in reducing risks. on the negative side, pairing ai systems with assessors may lead to over-reliance by conflating increased and absolute safety (automation bias). while this should be considered in system design and mitigated through user education, the safety benefits outlined above outweigh this concern. how human users predict llm performance humans were found to only marginally beat random guess in predicting gpt-4's performance @xcite . relatedly, @xcite showed humans overestimate llm future performance based on prior interactions, especially with larger models in high-stakes contexts. they argue that "the best llm is the one that allows humans to make the most reliable inferences about where it will succeed", closely aligning with predictaboard's motivation. @xcite indicated that human predictions become unreliable as ai systems become more capable @xcite found llm-produced explanations supporting a statement do not lead humans to reliably assess whether that statement is correct, even when the llm's token-level probabilities are calibrated. finally, in a specific use case, @xcite reported that the failures of an llm software engineer agent cannot be reliably anticipated. ## experimental results with baselines and existing llms this section presents the results with our llmassessor baseline pairs. we trained assessors on the training split of the mmlu-pro dataset and the scores of each llm; then, we compute metrics on the test split of mmlu-pro and on bbh. ## conclusion predictaboard creates a unique opportunity to explore advancements in both llms and assessors, offering potential gains on these two fronts. this paper aimed to release an initial version of the benchmark and allow the community to guide its future extensions, that can possibly include developing assessors that function across multiple llms and predicting safety indicators @xcite , rather than performance. additionally, comparing with human performance as assessors, as examined in recent studies @xcite @xcite , could provide insights into the differences in llm predictability from automated and human perspectives.