Can LLM s Judge Debates? Evaluating Non-Linear Reasoning via Argumentation Theory Semantics (2025)

Abstract: Large Language Models (LLMs) excel at linear reasoning tasks but remain underexplored on non-linear structures such as those found in natural debates, which are best expressed as argument graphs. We evaluate whether LLMs can approximate structured reasoning from Computational Argumentation Theory (CAT). Specifically, we use Quantitative Argumentation Debate (QuAD) semantics, which assigns acceptability scores to arguments based on their attack and support relations. Given only dialogue-formatted debates from two NoDE datasets, models are prompted to rank arguments without access to the underlying graph. We test several LLMs under advanced instruction strategies, including Chain-of-Thought and In-Context Learning. While models show moderate alignment with QuAD rankings, performance degrades with longer inputs or disrupted discourse flow. Advanced prompting helps mitigate these effects by reducing biases related to argument length and position. Our findings highlight both the promise and limitations of LLMs in modeling formal argumentation semantics and motivate future work on graph-aware reasoning.

## introduction evaluating the reasoning capabilities of large language models (llms) has largely focused on tasks involving linear reasoning, such as arithmetic, logical puzzles, and chain-of-thought (cot) explanations @xcite . even complex inference methods-like beam search, best-of-n, or tree-of-thoughts @xcite -are ultimately decomposable into linear sequences of intermediate steps. similarly, research on conversational systems predominantly addresses linear interactions between users and llms. in contrast, natural debates are not purely linear. alongside serial (near-path) exchanges, they frequently exhibit branching interactions, with mul-tiple arguments converging on one claim (fan-in) and single arguments influencing several others (fan-out). we focus on these non-linear settings and use debates that exhibit substantial branching rather than simple chains. computational argumentation theory (cat) formalizes these complex interactions through frameworks like quantitative argumentation debate (quad) semantics @xcite . quad semantics assigns each argument a numerical acceptability degree that reflects its strength within the debate. the emerging paradigm of llm-as-a-judge @xcite highlights the significant potential of llms for nuanced assessment and moderation across various applications, such as content evaluation, alignment, retrieval, and reasoning. given the rapid advancement of multi-agent ai systems engaging in sophisticated argumentative interactions, it is increasingly crucial to understand whether llms can effectively serve as impartial judges capable of handling non-linear argumentation structures. thus, our main research question is to investigate whether contemporary llms can effectively reason over non-linear argumentative structures inherent in natural debates. to answer this question, we design a novel evaluation setting: we transform debate graphs into natural dialogues, creating latent argument graphs-simple argument lists without explicit attack or support relationships. this flattened dialogue format mimics how individuals interact in real-world group discussions, where argument structure is not overtly marked. we instruct the llm to rank arguments by their inferred acceptability degrees. the model-generated rankings are then compared against ground-truth acceptability degrees computed using quad semantics. figure 1 summarizes our evaluation pipeline. the main contributions of this paper are: 1. the first systematic evaluation of llms' abiltop: we transform the argument graph (unweighted) into a natural language dialogue, which is fed to an llm with various instruction strategies (zero-shot, icl, cot). bottom: after uniform weight initialization, the graph is scored using quad semantics, producing final argument strengths w f . these are used to derive a gold ranking, which is compared to rankings predicted by the llm via spearman's ρ and kendall's τ . ## results as mentioned, our experiments evaluate the nonlinear reasoning ability of llms by assessing their capability to produce similar argument rankings as the ones generated by quad semantics. notably, quad semantics has full access to the argument graph structure, whereas the llms operate in a more natural dialogue setting. we evaluate the correlation between the llm rankings and those derived from quad using two metrics: (a) kendall's τ , which measures directional agreement based on pairwise orderings, and (b) spearman's ρ, which captures the monotonic relationship and is more sensitive to the magnitude of rank differences. using both metrics allows us to assess the agreement in order and the severity of misalignment. in addition to ranking, we evaluate whether models can reconstruct the signed, directed interaction graph from dialogue by comparing the predicted edge set ê against the gold edge set e. a prediction is correct only if both endpoints and the relation type match. we report precision, recall, and f1 at the graph level, macro-averaged within each split. our experiments yield the following findings: llms can moderately rank arguments in debate dialogues. tables 1 and 2 summarize the results of our study, showcasing the average performance of four state-of-the-art llms across the debatepedia and 12angrymen datasets, respectively. detailed results per llm are provided in we show that off-the-shelf llms, when used with prompting best practices, generally have a decent ability to rank arguments based on their latent graphs. this is supported by the moderately positive τ and ρ values across the results. for instance, in the debatepedia dataset, the highest average ρ achieved is 0.46 with the cot few-shot technique, and the corresponding τ is 0.42. the 12angry-men dataset results show a similar, albeit lower, performance. now, we address this difference. llms' ability to rank arguments is significantly influenced by input size. ## discussion to better understand llms' limits in modeling argumentation semantics, we further analyze their performance along four key aspects: discourse chronology's impact, argument length's influence, positional bias in llm rankings, and model size. lastly, we test whether specialization changes behavior by comparing our general-purpose models to a dedicated reasoner (deepseek r1). ## conclusion we investigated whether llms can reason over non-linear argumentative structures by ranking arguments in natural debates without access to explicit attack or support relationships. using cat semantics-specifically quad-as a structured reference, we evaluated llm performance across multiple instruction strategies. our findings show that llms can partially approximate structured argumentation reasoning, but their performance is highly sensitive to debate length, argument order, and model size. to our knowledge, this study offers the first systematic analysis of llms' ability to recover gradual bipolar argumentation semantics from dialogue alone, highlighting both their current limitations and potential for future graph-aware reasoning.