Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM (2024)

Abstract: How can we construct an automated debate judge to evaluate an extensive, vibrant, multi-turn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments.At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate.In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration.To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system’s performance to actual debate outcomes.The findings indicate a notable enhancement over directly using LLMs for debate evaluation.Source code and benchmark data are available at https://github.com/ljcleo/debatrix.

## introduction debating is the formal process of gaining consensus among groups with different opinions. in many cases, such as in competitive debates, only the policy proposed by the winning side will be accepted @xcite . debaters must apply various strategies to convince the audience to support their side in these debates. while systems like project debater @xcite have enabled automatic debating in competitive debates, judging these debates still relies on human annotation. automating debate assessment is helpful to improve debate quality in political, commercial, or educational scenarios and can also accelerate the evolution of debate automatons. recently, large language models (llm) such as chatgpt and gpt-4 (openai, 2023) have motion: thbt php is the best language in the world thank you, alice, but... it's about evolution, my friends... modern languages like javascript and python are frankly more enjoyable to use... plus, let's talk about security vulnerabilities... php's history is riddled with them... evolution does not necessarily mean superiority... let's go, let's go rust, let's... bob... no language is without flaws. php has evolved... php 7 has improved drastically ... ... ... php is popular & easy. sounds reasonable... wait, python is a good counter... php7? not enough... alice seems to use more evidence... but bob is really engaging... they talk too much! i can't @xmath0 alice bob llm the debate judge figure 1 : an llm debate judge judging a debate between alice and bob. the llm needs to understand the arguments and how they counter each other (purple bubble); the llm also needs to evaluate the speeches in multiple dimensions (orange bubble). however, multiround debates are often long, detracting attention or exceeding the context window (light gray bubble). however, judging debates with llms incorporates several issues to be considered, as illustrated in figure 1 . first, evaluating long, multi-turn debates remains challenging, as most current research focuses on short text such as open-question answers and user-request responses @xcite . second, debate judging requires a deep understanding of how arguments are organized and refuted across speeches; not only does it require more professional knowledge, but it also demands a thorough, precise analysis of how argu-ments work in the debate. finally, speech quality is affected by various factors, such as argument strength, evidence reliability, and language style, requiring systematic evaluation across dimensions. to this end, we propose debatrix, a fine-grained framework to assist llms in handling these challenges by breaking down debate judging along both chronological and dimensional axes. 1) iterative chronological analysis: we instruct the llm to analyze the debate speech by speech, maintaining speech and analysis streams with a memory system and providing content analysis of previous speeches when analyzing new speeches. after reviewing all speeches, the llm makes decisions based on all analyses. this iterative approach lets the llm concentrate on one speech at a time and understand the context more effectively. it also produces feedback or decisions for each speech, each debater, and the final winner. 2) multi-dimensional collaboration: debatrix also allows llms to focus on a specific judging dimension, such as argument, language, or clash, during the speech analyzing process. each llm agent can make comments on these specific aspects. for the overall judgment, all these individual analyses are combined into one summary, providing a systematic judgment across multiple dimensions. furthermore, we introduce panelbench, a novel benchmark for evaluating automatic debate judging systems. panelbench consists of two collections of debates with judgments, namely debateart and bp-competition. debateart sources from online platforms that follow competitive debate formats; debates vary in speech count and length and have dimensional voting results. bp-competition includes debates transcribed from world-class competitive debate competitions; these debates follow the british parliamentary (bp) format involving four teams (two on each side), enriching panel-bench with long, complex, high-quality samples. on panelbench, debatrix increased winner prediction accuracy compared to the baseline of directly prompting the llm with raw speeches; perdimension judging is also improved. furthermore, the results have also proved that iterative speechby-speech analysis and splitting dimensions help generate a more accurate final verdict. our contributions are as follows: ## related work argumentation persuasion assessment is the foundation of automatic debate systems, as they must be persuasive enough to argue effectively. previous works have focused on the persuasiveness of arguments, including empirical studies @xcite , machine learning models @xcite @xcite and work covering both @xcite . some argumentbased chatbots also take persuasiveness as a motivating factor @xcite . few works have extended these methods to judging whole debates (potash and rumshisky, 2017), including competitive debates @xcite . while these works mainly involve empirical laws or delicate models, within the bigger context of text evaluation, large language models (llms) have become a new and powerful tool to tackle this task. multiple exploitation methods have been proposed, such as conditional probability @xcite , score prompting @xcite @xcite and pairwise comparison @xcite . several works, such as @xcite , utilize multiple of them and introduce various methods to stabilize the results. regarding the challenges of judging debates with llms, some pioneering works have investigated some of them. for instance, chang (2023), @xcite and @xcite focus on multi-dimensional assessment and propose different strategies to improve accuracy, such as dialetics, peer review and group chat. meanwhile, de wynter and yuan (2023) and @xcite have explored llm's ability to handle argumentation tasks, measuring its capability of argumentation reasoning. these works are partially in line with our work yet do not cover all 3 issues mentioned in section 1. finally, @xcite proposed a language judge memory context language source judge memory context source speech judge debater judge winner judge argument argument judge memory context analysis dimensional speech judgment overall speech judgment dimensional debater judgment overall winner verdict alice: in my opinion, we should ... bob: thank you, alice. yet, i think ... alice: ... bob: ... alice: ... bob: ... dialog evaluation framework that works in a multiturn manner, similar to our chronological design; however, this framework is designed for human annotators instead of llm judges. ## debatrix in this section, we provide a detailed overview of debatrix, our fine-grained debate judging framework. ## panelbench to assess llms and our proposed debatrix framework with real competitive debates, we introduce panelbench, a novel benchmark for automatic debate judging covering various styles, formations, and judging dimensions. we include two debate collections in panelbench: debateart for 1v1 debates with dimensional judgments and bp-competition for high-quality debates with multiple debaters on each side. ## experiments we conducted experiments on panelbench to evaluate the debate-judging performance of llms. we also compare our debatrix framework with judging directly with llms. ## analysis 6.1 multi-turn & long debates as demonstrated in on one hand, some baseline models show partial advantage but fail to cover all scenarios: gpt-4 handles long debates well but degrades when judging debates with many turns; dimensional achieves low rmse on debates with few turns or tokens and noniterative on ones with many turns or tokens, but both become rather worse when switching to the other half of the debates. on the other hand, debatrix maintains a relatively low rmse compared to other models, regardless of the number of speeches or tokens. this shows that debatrix effectively assists the llm in evaluating long, multi-turn debates while extending its advantage to short debates. ## conclusion in this paper, we propose a fine-grained debate judging framework based on llm, debatrix. we decompose the debate judging task into an iterative chronological analysis to tackle multi-turn, long debates and elaborate multiple dimensions to generate systematic judgments. we introduce a novel debate judging benchmark, panelbench, to assess our framework and other automatic debate judging approaches. the benchmark covers multidimensional and multi-debater scenarios. under both settings, debatrix significantly improves chat-gpt, aiding it in judging long debates that exceed the context window and outperforming gpt-4.