C on QR et: A New Benchmark for Fine-Grained Automatic Evaluation of Retrieval Augmented Computational Argumentation (2025)

Abstract: Computational argumentation, which involves generating answers or summaries for controversial topics like abortion bans and vaccination, has become increasingly important in today’s polarized environment. Sophisticated LLM capabilities offer the potential to provide nuanced, evidence-based answers to such questions through Retrieval-Augmented Argumentation (RAArg), leveraging real-world evidence for high-quality, grounded arguments. However, evaluating RAArg remains challenging, as human evaluation is costly and difficult for complex, lengthy answers on complicated topics. At the same time, re-using existing argumentation datasets is no longer sufficient, as they lack long, complex arguments and realistic evidence from potentially misleading sources, limiting holistic evaluation of retrieval effectiveness and argument quality. To address these gaps, we investigate automated evaluation methods using multiple fine-grained LLM judges, providing better and more interpretable assessments than traditional single-score metrics and even previously reported human crowdsourcing. To validate the proposed techniques, we introduce ConQRet, a new benchmark featuring long and complex human-authored arguments on debated topics, grounded in real-world websites, allowing an exhaustive evaluation across retrieval effectiveness, argument quality, and groundedness. We validate our LLM Judges on a prior dataset and the new ConQRet benchmark. Our proposed LLM Judges and the ConQRet benchmark can enable rapid progress in computational argumentation and can be naturally extended to other complex retrieval-augmented generation tasks.

## introduction computational argumentation @xcite , or generating arguments for controversial topics-such as debates over banning abortion or the legalization of mari-juana-inherently involve sensitive discussions and diverse opinions on the appropriate course of action. well-informed argumentation requires retrieving publicly available sources of knowledge, such as blogs and news articles, which are often lengthy and noisy, and using them to create well-grounded and persuasive arguments. large language models (llms) employed in a retrieval augmented generation approach are a natural choice for this task due to their ability to handle large context lengths and strong performance across various tasks. to emphasize the importance of retrieved evidence for effective argumentation we call this task retrieval augmented argumentation, or raarg, which we investigate in this paper. while computational argumentation is a challenging task, even evaluating raarg systems is a daunting challenge in itself. the generated arguments are often long and complex, making human evaluation costly and time-consuming. thus, there is an increasing need for automated evaluation methods that can rigorously assess both the influence of retrieval involved in argument generation and the overall quality of the generated arguments, especially when dealing with a diverse set of controversial topics. while previous research on automated rag evaluation has largely focused on metrics like context relevance, answer relevance, and answer groundedness, often using llm-based evaluation methods, also known as llm-as-a-judge @xcite . in contrast, work in argumentation has concentrated on assessing argument quality without considering the impact of retrieval @xcite @xcite . besides, these evaluations have notable limitations, particularly their reliance on single-score outputs that lack interpretability and provide minimal insights into specific performance dimensions like argument cogency, reasonableness, and performance at the (a) an illustration of the retrieval augmented argumentation (raarg) task for generating pro and con arguments conditioned on retrieved evidence, for an example topic of "mandating vaccinations". specifically, a successful automated evaluation of an argumentation system should test the ability to retrieve relevant context and to generate grounded, and high-quality arguments for a diverse set of controversial topics, while agreeing with human preferences. to address these challenges, we propose a systematic evaluation of llm-based argumentation in a retrieval-augmented setting by validating multiple fine-grained metrics, which we call llm judges, as different variants of llm-based evaluation are needed for each of the metrics. overviews of the task and our llm judges are depicted in figure 1 , where figure 1a illustrates our raarg pipeline for a sample controversial topic, and figure 1b shows our proposed llm judges method for automatically evaluating the retrieved evidence and the generated argument. specifically, our contributions are four-fold: (1) first, we demonstrate the feasibility of using modelbased evaluation for the task of argumentation; to the best of our knowledge, this is the first work that investigates automated evaluation of retrieval augmented computational argumentation in dataset task source of arguments source of evidence length of arguments grounding of arguments stance conditioned arguments arguana counterargs (wachsmuth et al., 2018b) counterargument retrieval idebate.org -short x ✓ touche argument retrieval @xcite argument retrieval ebatewise.org, idebate.org, etc. arguments are assumed to be directly available x ✓ changemyview @xcite counterargument generation (dialog) reddit changemyview -short x n/a argtersely @xcite counterargument generation reddit changemyview -short x n/a npov @xcite neutral point of view generation procon.org -long x n/a dagstuhl-15512 argquality @xcite argument a complex and realistic setting. in the rest of the paper, we first review related work to place our contributions in context ( §2). then, §3 defines the task of retrieval-augmented argumentation and describes a reference implementation using best rag practices. §4 details the construction process of the conqret benchmark. in §5, we define multiple llm judges through novel and adapted variations of prompts, while §6 validates them for both a prior argumentation dataset and over conqret, demonstrating the feasibility of automated fine-grained evaluation of computational argumentation. ## related work we now describe some of the related work in argumentation and llm evaluation. ## retrieval augmented argumentation we now define the task of retrieval augmented argumentation, or raarg, and describe our reference implementation using state-of-the-art models and best practices. ## the conqret benchmark in order to enable research on complex, evidencebased argumentation, we develop and present a novel benchmark, conqret. the benchmark is constructed from a popular website of expertgenerated arguments, which we further augment by retrieving the actual sources selected by the experts to be used as the ground truth evidence for retrieval augmented argument argumentation. the details of the dataset construction follow. ## experiments and analysis in this section, we describe the various llm judges used in our analysis. we first evaluate our llm judges in the non-rag setting and then extend the evaluation to address the rag counterpart. ## results and analysis we now present the results of each of the above evaluations on both the datasets. ## conclusions and future work this paper introduces the retrieval-augmented argumentation task, and presents a first reported study on automatic evaluation procedure and validation for this task, without the need for large-scale scale human-annotation. we demonstrate how argumentation can be effectively evaluated over two datasets, showing that our proposed llm judges are more efficient, interpretable, and align strongly with human evaluations. additionally, we introduce a novel benchmark conqret designed to evaluate both retrievers as well as rag systems in realistic scenarios. our contribution also includes the proposal of new and modified metrics through llm judges for measuring context relevance and answer groundedness, and we describe methods to validate these metrics without requiring large-scale annotation. our work has several crucial applications. our llm judges, in addition to demonstrating high agreement with human evaluations, also showed the capability of llms to compute multiple metrics in a single inference call. this encourages research on a holistic evaluation of retrieval-augmented systems, allowing metrics to complement and inform each other during the evaluation process. furthermore, these fine-grained metrics are promising, as they offer multiple cues to improve the training and alignment of llms, ultimately leading to more accurate and reliable systems.