Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons (2025)

Abstract: Large Language Models (LLMs) have shown to be effective evaluators across various domains such as machine translations or the scientific domain. Current LLM-as-a-Judge approaches rely mostly on individual assessments or a single round of pairwise assessments, preventing the judge LLM from developing a global ranking perspective.To address this, we present Knockout Assessment, an LLM-as-a-Judge method using a knockout tournament system with iterative pairwise comparisons. Experiments across three LLMs on two datasets show that knockout assessment improves scoring accuracy, increasing Pearson correlation with expert evaluations by 0.07 on average for university-level exam scoring and machine translation evaluations, aligning LLM assessments more closely with human scoring.

## introduction across various domains, and especially for scientific research, accurate and consistent evaluations are very crucial for informed decision-making. however, the inherent scale and subjectivity make this task very challenging and time-consuming. in recent years, the methodology of "llm-as-a-judge" @xcite has emerged to tackle this challenge, where instead of humans, large language models (llms) take the role of the expert to evaluate complex tasks. using llms as evaluators allows us to mimic the abilities of human experts, making evaluations cost-effective and scalable. although many approaches to llm-as-a-judge exist, the most common is individual assessment, in which the evaluation prompt consists of only the question and the corresponding answer, which is to be evaluated @xcite . while this approach has already shown to yield good evaluation results next to providing scalability @xcite , it does not consider the relative strength of answers in a set to a given question. the more recent approach of pairwise assessment tries to address this issue by providing two responses to the judge llm each time, however, it still fails to account for a global ranking perspective, as pairwise comparisons do not analyze how all responses compare to each other in the broader sense. in this paper we present an llm-as-a-judge method called knockout assessment to address this challenge, which can be seen as a variation of the tournament system used by @xcite , differing in that it makes use of iterative pairwise comparisons. instead of isolating responses individually or in pairs for evaluation, knockout assessment focuses on an iterative process where responses are compared against one another multiple times in a tournament manner. in each round, stronger responses advance to compete against each other in later rounds, allowing us to refine the scores progressively throughout the tournament. this approach allows the judge llms to develop a global perspective on responses without requiring all replies to be included in a single prompt, which would otherwise result in an impractically long context length. to summarize, our contributions are as follows: various studies have used this method for tasks such as evaluating story generation, scoring quality of different texts according to different criteria, or grading university-level exams @xcite @xcite . pairwise assessment another more recent llmas-a-judge approach is pairwise prompting, in which the llm judge is provided with two responses to the prompt instead of one. the judge llm is then asked to evaluate both responses. this has shown to be an effective llm-as-a-judge method for ranking documents, as it gives the judge llm direct comparison points while making evaluations @xcite . however, this method still does not make the judge llm develop a global ranking perspective. this approach is thus able to make implicit use of a global view over the dataset while assigning scores. ## knockout assessment we propose using multiple iterative pairwise comparisions instead of individual assessment with knockout assessment. in each pairwise assessment, one pairwise ranking prompt similar to the comparative prompt introduced by @xcite is used. in each prompt, one question and two answers to that question are provided to the judge llm, which is asked to evaluate both of those answers. we call this a "question-levelmatch". the exact prompts we used for our experiments can be found in appendix a.2. from the response generated by the judge llm, the score each individual answer got is parsed and saved to the list of scores for that answer, which keeps track of all the scores an answer got throughout all its question-level-matches. the answer which got the higher score advances to the next round to be matched up against another answer. the order of texts in pairwise rankings has shown to be an influential factor in the llms decision making @xcite , thus we also collected the results with using a debiasing methodology similar to the one introduced by @xcite , averaging scores from both possible orderings of each answer pair. debiasing thus results in double the compute-time compared to a regular questionlevel-match. the main methodology behind our appproach is a knockout tournament system that iteratively uses the question-level matches. in each tournament round, the n available answers to a question are randomly assgined to pairs. each pair then enters a question-level match, and the higher scoring response advances to the next round. in the case when n is odd, one answer directly advances to the next round. this continues until we reach a tournament round with a single response. once the tournament ends, the final evaluation score for each answer is computed as the average of all the scores it received throughout the tournament. an example tournament with @xmath0. the full algorithm is given in a.1. human grades and llm grades. each question is labeled by difficulty and language. sciex contains 1120 question-answer pairs in total. the second dataset is taken from the wmt metrics shared task @xcite , which includes a list of source sentences and machine translations, accompanied by a human evaluation on a scale of 0 to 100. we filtered the dataset by languages supported by the llama models (english, german, french, italian, portuguese, hindi, spanish, and thai). we also remove identical translations of the same source sentence, as sometimes the human evaluations differed for the same translation, causing inconsistencies. the final processed wmt dataset includes 2100 source-translation pairs. baselines we compare our approach to 2 baselines: (1) individual assessment, where each answer is scored individually and (2) naive pairwise assessment, where answers are paired up and scored only one time, without multiple knockout rounds. evaluation metrics we use pearson correlation as our primary evaluation metric, which measures the linear relationship between the scores provided by our method and the human-provided scores. additionally, we use pairwise ranking accuracy @xcite , which indicates how often the llm judges select the correct winning answer given a pair of answers. the results for pairwise ranking accuracy can be found in appendix a.3 since they mirror our pearson correlation findings. ## conclusion we address a key limitation of many existing llmas-a-judge methods: not having a global view over the responses while evaluating them. to address this, we proposed an alternative llm-as-a-judge method called knockout assessment and tested it with three different llms on two different datasets. knockout assessment improves pearson correlation to human evaluations by 0.07 over individual assessment on average. the performance increase was more significant in scientific evaluation compared to machine translation evaluation, especially for the larger llm. furthermore, for scientific evaluation, the responses which progressed further in the knockout assessment process had 0.08 better accuracy compared to the responses which got eliminated on the first round, which indicates that knockout assessment results in a performance increase from regular pairwise assessments.