C ourt E val: A Courtroom-Based Multi-Agent Evaluation Framework (2025)

Abstract: Automated evaluation is crucial for assessing the quality of natural language text, especially in open-ended generation tasks, given the costly and time-consuming nature of human evaluation. Existing automatic evaluation metrics like ROUGE and BLEU often show low correlation with human judgments. As large language models (LLMs) continue to evolve, researchers have explored their use as alternatives to human evaluators. Although single-agent approaches have shown potential, results indicate that further progress is required to close the gap between their performance and the quality of human assessments. Acknowledging that human evaluations involve multiple annotators, the multi-agent approach allows LLMs to collaborate, enhancing efficiency and effectiveness in handling complex tasks. In this paper, we present CourtEval, a novel Multi-Agent Evaluation Framework modeled after courtroom dynamics. Each agent takes on a distinct role: the Grader, similar to a judge, assigns an initial score; the Critic, like a prosecutor, challenges this score; and the Defender, akin to a defense attorney, defends it. Based on the input from both the Critic and Defender, the Grader re-evaluates the score, leading to a more balanced and fair final decision through this adversarial process. CourtEval substantially outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat.

## introduction evaluating the quality of text, whether generated by language models or written by humans, has long posed a significant challenge, consistently attracting considerable attention from researchers and practitioners alike @xcite . conventional approaches mainly depend on humanannotated texts @xcite , a method often regarded as excessively time-consuming and expensive. automatic evaluation metrics for natural language generation (nlg) tasks help minimize the reliance on human evaluations, which are often costly and time-consuming to gather (zhu and bhat, 2020). to tackle this issue, evaluation metrics based on n-grams, such as rouge @xcite , bleu @xcite , and me-teor @xcite , have been developed as alternatives to costly human evaluations. however, these approaches have demonstrated a relatively low correlation with human evaluations, especially in tasks involving open-ended generation or those requiring specialized domain knowledge @xcite . while embedding-based similarity from pre-trained models can alleviate some of these shortcomings, @xcite highlighted that such similarity, termed "humanlikeness", and pointed out that generating humanlike text may not always reflect the quality of the output. in view of the impressive text understanding and instruction-following capabilities of recent llms, a body of literature @xcite @xcite @xcite ) has adopted llms as evaluators to assess the quality of responses to open-ended questions or traditional nlg tasks, including dialogue response generation and summarization. research findings show that llms can mimic human behavior and provide evaluations aligned with human judgments, offering a scalable and transparent alternative to expensive, labor-intensive human assessments. in the human evaluation processes, relying on a single perspective can introduce bias and instability in the results @xcite . while a single powerful llm can already tackle various missions, emerging studies suggest that multiple llms can further improve one another through debate and cooperation @xcite . by incorporating multiple llms into an integrated group and designing specific interaction mechanisms, different llms can engage in proposing and deliberating unique responses and thought processes across several rounds. this approach leads to enhanced factuality of generated responses @xcite and improvement in the completion of arduous tasks @xcite . furthermore, the multi-agent group also addresses and mitigates the degeneration-of-thought (dot) problem @xcite . our proposed multi-agent evaluation framework is inspired by the structure of a courtroom, where different roles provide opposing perspectives to arrive at a fair judgment. in this framework, the grader acts as the judge, initially evaluating the text and assigning a score. the critic functions like a prosecutor, rigorously questioning and critiquing the grader's decision. in response, the defender, similar to a defense attorney, counters the critic's arguments by defending the grader's judgment. the grader then reconsiders the feedback from both the critic and defender to form a more balanced final decision. this setup mirrors the adversarial nature of courtroom debates, leveraging multiple perspectives to ensure a robust and well-justified evaluation process. we summarize our contribution below:- ## methodology in this section, we elaborate on the principal components in courteval debater agents, diverse role specification, communication strategy, and provide a detailed overview of each component's role and functionality. debater agents are one of the most significant components in our framework. our proposed debater agents controller, grader, critic and defender is designed to act as an ai assistant, using llms but not requiring human input or code execution. we treat each individual llm as an agent and ask them to generate their response from the given prompt. courteval is a framework that simulates a mock conversation between an llm agents and a user proxy agent. here a user proxy agent is an agent playing the user's role in conversations with the llm agent. the workflow of this framework is presented in figure 2 . the user proxy agent takes a llm evaluation problem to be solved as input and would initiate a conversation with the controller agent. the initial message from the user proxy agent consists of an initial prompt and the problem to be solved. another distinct benefit of this framework is that it enables multi-turn dialogues, which can be particularly useful in addressing complex evaluation that require multi-step reasoning. ## conclusion and future work in conclusion, this work presents courteval, a novel multi-agent evaluation framework inspired by courtroom dynamics to enhance the quality, fairness, and reliability of automated text evaluations. the introduction of distinct roles-grader, critic, and defender enables structured debate that mitigates bias and facilitates more accurate alignment with human judgments. our experimental results show that courteval significantly outperforms traditional evaluation methods and state-ofthe-art frameworks across natural language generation benchmarks, such as summeval and topi-calchat, demonstrating a notable improvement. for future work, we plan to expand the multiagent structure with domain experts, optimize debate iterations with adaptive mechanisms, integrate real-world feedback, enhance multilingual and multimodal capabilities, and apply courteval to tasks like machine translation and conversational agents.