Dynamic Knowledge Integration for Evidence-Driven Counter-Argument Generation with Large Language Models (2025)

Abstract: This paper investigates the role of dynamic external knowledge integration in improving counter-argument generation using Large Language Models (LLMs). While LLMs have shown promise in argumentative tasks, their tendency to generate lengthy, potentially non-factual responses highlights the need for more controlled and evidence-based approaches. We introduce a reconstructed and manually curated dataset of argument and counter-argument pairs specifically designed to balance argumentative complexity with evaluative feasibility. We also propose a new LLM-as-a-Judge evaluation methodology that shows a stronger correlation with human judgments compared to traditional reference-based metrics. Our experimental results demonstrate that integrating dynamic external knowledge from the web significantly improves the quality of generated counter-arguments, particularly in terms of relatedness, persuasiveness, and factuality. The findings suggest that combining LLMs with real-time external knowledge retrieval offers a promising direction for developing more effective and reliable counter-argumentation systems. Data and code are publicly available: https://github.com/anaryegen/ counter-argument-generation

## introduction argumentation in natural language processing (nlp) is becoming an increasingly active area of research, driven by the natural human tendency to express disagreement with claims or viewpoints expressed by individuals during information exchanges. in fact, it is becoming an indispensable tool in many application domains such as public policy, law, medicine, and education @xcite @xcite @xcite . it is possible to distinguish two main research lines in argumentation in nlp: (i) argument mining, which involves analyzing unstructured texts to automatically identify and extract argumentative elements and their relations @xcite @xcite ; (ii) argument generation, which focuses on generating argumentative texts using external knowledge sources @xcite or summarizing key argumentative points @xcite @xcite . this paper contributes to the second line of research by investigating whether dynamic integration of external knowledge helps llms to improve counter-argument generation. counter-argument generation seeks to develop an effective framework for presenting alternative perspectives to an argument while ensuring the correctness of the message and incorporating factual evidence @xcite . llms have shown promising potential to deal with debates or any disagreements by solely relying on the parametric knowledge encoded in the model @xcite . moreover, most of the llms have security safeguards to avoid harmful interactions by any means @xcite @xcite . while maintaining harmlessness is recommendable, it is also important to be reasonable, persuasive, and grounded with a factual basis when arguing @xcite . previous work on counter-argument generation has focused on specific aspects of the argument @xcite , integrating different external news sources @xcite , or by attacking different evidence of the initial argument @xcite . however, no previous work has considered integrating dynamic knowledge to inform counterargument generation. furthermore, we argue that existing datasets for counter-argument generation consist of excessively long @xcite @xcite or extremely short @xcite arguments, which makes it difficult to accurately evaluate their quality. thus, although modern llms tend to generate rather lengthy essay-style responses that may be highly persuasive, they lack coherence, logic, and factual evidence, restricting them to short single sentences is insufficient to study the complexity and desired pragmatic characteristics of a good counter-argument @xcite @xcite . finally, the manual evaluation of counter-argument quality is a challenging, time-consuming, and highly subjective task that requires knowledge of the subject matter and credible evidence to support or refute it @xcite @xcite . in this sense, we believe that traditionally used @xcite reference-based automatic metrics such as bleu, meteor, or bertscore fail to accurately capture the nuanced relationship between the generated counter-argument and the human judgment. in order to address these issues, we propose using retrieved external knowledge from dynamic sources without any limits to particular outlets or databases, along with llms, to generate efficient, concise, evidence-based counter-arguments. furthermore, we present a new curated dataset of concise and structured human-generated argument and counter-argument pairs in which the length of the counter-arguments is enough to study the main argumentative aspects while facilitating manual and automatic evaluation. while this paper employs both manual and automated evaluation methods, we introduce a novel automated evaluation approach using llms-as-a-judge, designed to optimize correlation with human assessments. using the new dataset and dynamically retrieved external evidence, this work aims to answer the following research questions (rq): figure 1 illustrates our proposed framework. first, we automatically generate queries that challenge the main points of the argument or claim for more fact-based information and feed those queries to the web search. next, related evidence is retrieved, and, lastly, the claim and the retrieved evidence are provided as context to the llm to generate a counter-argument. the main contributions of our paper are the following: the use of external sources has demonstrated its effectiveness in the generation of alternative perspectives. @xcite analyzed the question of retrieving the best counter-arguments when no prior knowledge is available. @xcite introduced a framework that incorporated information retrieval, but their approach was limited to using the wikipedia database as the external source. however, wikipedia primarily contains static factual information, which may not align with the dynamic nature of arguments. to address this limitation, @xcite expanded the scope by indexing all documents from the common crawl database for argument retrieval. @xcite proposed an enhanced framework that leverages databases from news outlets alongside wikipedia to retrieve evidence and improve the quality of the generated counter-arguments. all these efforts mainly refer to static databases as external sources, meaning that all the documents containing the evidence of the argument in question should be parsed in advance. moreover, previous argument generation with external knowledge was proposed before the llms entered the race. in our work, we believe that we should test the capabilities of llms and that, in our ever-changing dynamic world, it is not efficient to rely on a pre-defined set of documents as an external source for factual and persuasive counter-argument generation. instead, we propose to integrate knowledge from the whole internet as a source for finding factual evidence to generate better counter-argumentation. ## data in order to perform our experiments and to guarantee a balance in the length of input argument and output counter-argument in the data, we constructed a new corpus for the evaluation of counterargument generation. previous work often focused on either (too short) sentence-level @xcite or (too long) paragraph-level @xcite generation. given the capabilities of modern llms, it was essential to specify clear input and output data to ensure accurate, robust, and fair comparisons during evaluation. thus, our objective is to create a dataset of argument/counter-argument pairs that would meet specific criteria. we argue that (i) generating single-sentence claim-based counter-arguments is insufficient to accurately assess the quality of counter-arguments produced by llms and (ii), counter-arguments that are too long make it extremely difficult to properly evaluate the argumentative quality of the generated text. our analysis revealed that large language models (llms), when not given explicit length constraints, tend to generate verbose, essay-like responses that frequently deviate from true argumentative form, lack substantiating evidence, and demonstrate poor coherence with the original argument while overemphasizing persuasive elements. to address these limitations, we propose generating counter-arguments with a maximum length of three sentences, focusing on conciseness, factual content, and direct alignment with the input argument. with this aim in mind, we constructed a new dataset of argument and counter-argument pairs using the candela corpus as a basis @xcite . the corpus consists of debates and dis-cussions on various controversial topics from the r/changemyview foot_1 subreddit. the corpus is centred around real-world online debates where users post their opinions and evidence for any controversial topic and expect other users to provide reasoning for an alternative perspective. candela is available in a format split by sentences, tokenized, and lowercased, making the reconstruction of the corpus necessary. to address this, we employed llms to convert the data back into a coherent, human-readable format. once the data was fully reconstructed in a human-readable format, we summarized all arguments and counter-arguments. to avoid any bias, we choose a language model different from those used in our experimentation, namely, llama-3.1-70b-instruct @xcite . the generated summaries were then manually verified against the original data and re-summarized when required to ensure semantic and pragmatic correctness. while the corpus includes data from real-world interactions and reflects arguments from natural exchanges of information, we found that not all topics were equally suitable to evaluate whether external knowledge benefits counter-argument generation. specifically, topics lacking a scientific or factual foundation, namely, expressed on deeply subjective topics, often trigger llms to produce generic, safe responses due to their safety guardrails. moreover, the external knowledge of the topics that lack ground-truth factual backing tends to be ambiguous and/or biased. an example of such arguments can be found in appendix b. to mitigate potential quality issues, we implemented a manual filtering process for the corpus, retaining only those arguments that demonstrated both high quality and direct relevance to the subject matter, specifically selecting instances where the incorporation of factual information would meaningfully contribute to argument generation. finally, we further manually refined the summaries to follow a structured argumentative format, emphasizing components such as the main claim, supporting evidence, and examples where applicable (see examples in appendix c). this process resulted in a dataset of 150 high-quality 3-sentence paragraph-level argument and counter-argument pairs. examples of the final version of the corpus used in the experiments can be seen in appendices a and c. the average distribution of sentences and tokens in the data is shown in ## experimental setup in this section, we describe the methodology for generating counter-arguments with dynamic knowledge integration for evidence-driven counterargument generation. preliminary experiments revealed that relying on static pre-defined document sets (such as wikipedia) for factual evidence retrieval often yields incomplete information regarding specific argument topics, resulting in incoherent and unreliable evidence. we determined that external sources must contain topic-specific content, opinions, and observations directly relevant to the events described in the argument, particularly since claims may reference recent events that post-date the last update of pre-parsed sources. consequently, our experimental design incorporates dynamic webbased data as the primary information source to address these limitations. external knowledge is obtained through the web search tool provided by the cohere api foot_2 . the process involved automatically generating five queries (averaging 67 words each) designed to challenge the validity and veracity of the original argument's claims and premises by questioning key points that required factual substantiation. these queries were sequentially submitted to the web search engine, and the retrieved results, averaging 5,496 words in length, were incorporated as contextual information in the final prompt presented to the language model. the model then generated counterarguments based on both the original argument and the retrieved contextual information. to assess the role of external knowledge in counter-argument generation with llms, we performed a comparative analysis using two system configurations: one incorporating external information and another relying solely on the model's parametric knowledge. the model in the latter configuration received only the original argument as input and was tasked with generating counter-arguments using its internal knowledge base exclusively. this experimental design enabled us to evaluate whether llms display better performance when provided with external evidence for counter-argument generation, as measured by comparing the quality of outputs between the two configurations. the experimentation is performed using two llms with strong results on text generation tasks in two different configurations: (i) the llm using only the claim as a prompt to rely on its parametric knowledge and (ii) the llm with dynamic external knowledge. this results in the following four models: importantly, all experiments are conducted in an inferential setting under default hyperparameters to assess the real capabilities of llms in counterargument generation. this setup ensures a fair and robust evaluation of their performance in generating meaningful, well-reasoned, and factual counterarguments. a list of all the prompts used at each generation step is illustrated in appendix d. it should be noted that the evaluation of generated counter-arguments is inherently subjective. while human evaluation is the gold standard, it is time-intensive, costly, and prone to individual biases @xcite @xcite . to address these limitations, along with human evaluation, we will also provide two types of automatic evaluation. first, using reference-based metrics such as bleu, me-teor, or bertscore previously used in counterargumentation generation @xcite . second, we propose to use the llm-as-a-judge approach. leveraging the consistency, scalability, and efficiency of llms, this method enables rapid and reproducible scoring across the five dimensions. to the best of our knowledge, we are the first to propose an llmbased evaluation for counter-argument generation. ## results we first calculate the correlation of the automatic metrics with human judgments. we then used the best automatic metrics to compute the overall rankings for the gold counter-arguments and those generated by the four models. finally, we analyzed dimension-wise rankings obtained in the manual evaluation. correlation with human judgments. we would like to stress that manual counter-argument evaluation is a highly subjective and tedious process @xcite @xcite . thus, ensuring that our automatic evaluation method correlates with human judgments is crucial. therefore, we computed the correlation between every evaluation metric on the evaluator sample set of 75 examples, including human evaluation, using the method described in section 4. the spearman's rank correlation coefficients (ρ) are reported in figure 2 . first, it can be observed that the three llm-as-a-judge methods obtain the highest correlation with human judgments (row marked in red). thus, while judgelm and prometheus obtain a strong correlation, claude 3.5 sonnet is the best method with a ρ of 0.82 (very strong correlation). reference-based metrics show a poor correlation with both human judgments and llm-as-a-judge evaluation methods. this suggests that referencebased metrics may be inadequate for evaluating the quality of counter-argument generation, as they fail to capture the essential dimensions established for human evaluation and disregard the context of the original argument or claim being countered. among the llm-as-a-judge methods, the lowest correlation was observed between judgelm and prometheus. upon a more detailed analysis, we found that these two judges had the highest level of disagreement in evaluating opposition and persuasiveness. prometheus tends to be stricter on opposition, while judgelm does the same on persuasiveness. thus, each model assigned lower scores than the other for those particular dimensions. overall rank evaluation. figure 3 reports the overall rankings obtained by summing all the scores from each dimension that each evaluator provided (as described in section 4). the first observation is that 2 out of 3 llm-as-a-judge agree that command r+ with external knowledge generates the best counter-arguments. moreover, according to the manual evaluation, both command r+ and mistral-7b-instruct with external knowledge are ranked at the top. interestingly, results with respect to the ranking of other models vary, but all four evaluators agree that the human-generated counterarguments (gold reference) are ranked worst. taking into account the high correlation of llmas-a-judge methods with human preferences, we also evaluated the four models over the full corpus of 150 arguments and counter-arguments pairs. the results from each llm-as-a-judge evaluator can be found in the appendix e (figures 5 , 6 , 7 , 8 ). point-wise evaluation. figure 4 illustrates the average dimension-wise rankings from manual evaluation. we can see that command r+ excels in opposition compared to others, whereas both command r+ and mistral-7b-instruct with external knowledge are valued the best on persuasiveness and relatedness. moreover, mistral-7b-instruct with external knowledge and command r+ obtain the highest factuality scores. overall, we can conclude that when external evidence is provided to the llms, the overall quality of the counterargument improves. nevertheless, the performance of generating counter-arguments based on parametric knowledge is quite high, especially for the larger models (command r+). ## analysis of results to answer rq3, we checked whether the generated counter-arguments indeed used the provided external knowledge or whether the retrieved evidence is already incorporated in the parametric knowledge of llms. we used the sentence transformer model gtebase-en-v1.5' @xcite to calculate sentence-level cosine similarity between the generated outputs and provided external evidence. our methodology involved segmenting both the external information and generated counterarguments into individual sentences for pairwise comparison, followed by ranking the similarity scores in descending order. based on manual verification, we established a threshold of 70% similarity as indicative of successful external knowledge integration. our analysis revealed that command r+ with external knowledge effectively utilized external evidence in 82% of its generated counterarguments, while mistral-7b-instruct with external knowledge demonstrated such integration in 51% of the cases. our manual analysis of similarity scores revealed that, when similarity scores exceed 70%, models directly integrate provided external knowledge, while scores between 65-70% result in partial model bleu rouge meteor bertscore avg ## conclusion in this study, we examined the effect of incorporating externally dynamically retrieved evidence into llms in counter-argument generation using web search as an external knowledge source. our results on a newly created gold dataset show that, while llms with external knowledge improved their counter-argument generation, their reliance on it varies, and in some cases, they generate responses with parametric knowledge that obtained better scores. we propose llm-as-a-judge to automatically evaluate the quality of counter-arguments with better correlation scores with respect to human judgments than previously used reference-based metrics. through qualitative analysis, we found that model behaviour shifts when dealing with sensitive or controversial topics. in these cases, llms tend to provide more generalized responses rather than directly integrating factual evidence. interestingly, such responses were often rated more favourably, suggesting a preference for plausibility and coherence over strict factual accuracy. our findings highlight the complexities of integrating external knowledge into llm outputs. while retrieval-augmented generation (rag) can enhance factual consistency, models may still prioritize linguistic fluency and alignment with social norms. future work should focus on refining strategies to ensure that external knowledge is utilized effectively, particularly in contexts that require precise and evidence-based argumentation.