Using tournaments to calculate AUROC for zero-shot classification with LLM s (2025)

Abstract: Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.

## introduction in zero-shot classification with large language models (llms), a classification task is presented to the llm in a prompt, and a discrete classification decision is extracted from the generated response. these generated responses can be used to compute traditional scoring metrics of accuracy, precision, recall, and f1 score, to compare against other classification metrics. supervised methods, in contrast, typically output a probability distribution over possible outputs. in the binary classification setting, real-world deployments of classifiers benefit from these explicit scores, as it allows downstream users to adjust decision thresholds to adapt classifiers to different use cases, with the same classifier potentially using different thresholds for different tasks. in a screening setting, recall is maximized at the expense of precision, because there may be such a high cost to missing cases that we can accept more false positives (e.g., doing research on out-of-hospital mortality, we want to capture as many potential cases as possible). in a case-finding setting, we may prefer to maximize precision at some expense to recall, for cases where we want our predicted cases to be a pure representation of that phenomenon (e.g., in a clinical trial recruitment setting, we want to err on the side of precision when it comes to satisfying inclusion and exclusion criteria). some commonly used instruments to evaluate a classifier at different decision points are the receiver-operator curve (roc) and the precisionrecall curve (prc). measuring the area under the roc and prc curves (auroc and auprc, respectively) can also be used as a single metric to summarize the amount of signal captured by a given classifier across all thresholds. zero-shot classification with llms does not have a straightforward way to perform the equivalent actions. unlike supervised methods, llms do not have the concept of a decision threshold, so they can only be evaluated at one point, and whether this point represents a bias towards high recall or high precision is not easy to measure. for classification tasks where the class of interest has low prevalence, small changes in the number of positive predictions or in the decision threshold can have outsized effects on the f1 score. in such cases, it may be preferable to use threshold-independent metrics like auroc, especially when comparing between classification paradigms (supervised vs. zero-shot llms). in this paper, we describe a method for estimating auroc or auprc for a zero-shot llm by turning the classification task into a ranking task between pairs of instances. we use the elo rating system @xcite , used for chess and online videogames, to assign scores to instances in a dataset, updating them after "wins" or "losses," and use the ordering this rating induces to compute auroc. since this system requires more llm calls than straightforward classification, we also investigate various tournament scheduling strategies and empirically measure which schedules minimize the number of competitions to reach competitive performance. on three diverse datasets, cola, clinifact and a patient outcome prediction task derived from @xcite ), we find that our proposed method leads to stronger classification performance in terms of f1 scores. random scheduling performs surprisingly well, though the "swiss" system used in chess and a novel graph-based method are also strong in most settings. ## background and related work our method is inspired by ratings systems like elo, developed in chess @xcite , but applied to many other settings, including online video games. this and other related systems (c.f., the bradley-terry model (bradley and @xcite ) are implicitly grounded in probability distributions around player ratings, where differences in distributions can be used to calculate probability of one competitor winning against another. these methods also consist of update methods where, after a competition between player a and player b, respective elo ratings can be updated based on the difference between scores. for the elo system, the updated rating for player a (ê a ) with elo rating e a against player b with elo rating e b is: our method is motivated by the goal of generating roc curves, and allowing practitioners to select appropriate cut-offs for downstream classification applications. however, it shares a similarity with a recent thread of llm-based research on the topic of estimating confidence of llm answers. if certainty could be estimated reliably, directly from llm outputs, then roc curves could be generated directly from those outputs. however, recent work in this area shows mixed results, with some work showing difficulty finding correlations with certainty in zero-shot classification tasks @xcite . other work does find some signal, but with white-box methods showing much greater signal than black-box methods @xcite . work studying calibration claims that llm uncertainties are calibrated for true/false question answering @xcite , but that calibration only holds for the largest model they evaluate. ## evaluation we use llama-3.2-3b-instruct and gpt-4o-mini foot_1 as representative models, ranging from smaller, more open models, to larger, more closed models. we further evaluated our approach on three different instruction-tuned models, qwen 2.5 @xcite , distilled deepseek r1 (deepseek-ai, 2025), and phi-tiny-moe (abdin et al., 2024), and present the results in the appendix b. our code to run these models uses the inspect eval framework (ai safety institute), which runs the llama models using huggingface transformers @xcite . ## discussion and conclusion one major benefit of this approach relative to other methods for estimating uncertainty is that it can be computed from generations only. while closed models like chatgpt make log probabilities available in their api, there is no guarantee that such functionality will always exist. our method only relies on the core functionality of llms, the ability to generate text. a drawback of our approach is its requirement of multiple passes over the dataset. our results show asymptotic behavior after 10-20 rounds, but the smaller dataset does asymptote quickly, raising the possibility that the optimal number of rounds is a function of dataset size. future work should explore this relationship to judge the feasibility of applying this method to larger datasets. the code implementing our methods and generating these results is available at: https://github.com/ machine-learning-for-medical-language/ cnlp_llm .