Can Large Language Models Understand Argument Schemes? (2025)

Abstract: Argument schemes represent stereotypical patterns of reasoning that occur in everyday arguments. However, despite their usefulness, argument scheme classification, that is classifying natural language arguments according to the schemes they are instances of, is an under-explored task in NLP. In this paper we present a systematic evaluation of large language models (LLMs) for classifying argument schemes based on Walton’s taxonomy. We experiment with seven LLMs in zero-shot, few-shot, and chain-of-thought prompting, and explore two strategies to enhance task instructions: employing formal definitions and LLM-generated descriptions. Our analysis on both manually annotated and automatically generated arguments, including enthymemes, indicates that while larger models exhibit satisfactory performance in identifying argument schemes, challenges remain for smaller models. Our work offers the first comprehensive assessment of LLMs in identifying argument schemes, and provides insights for advancing reasoning capabilities in computational argumentation.

## introduction computational argumentation is an area of research that engages natural language processing (nlp) tasks including argument mining, assessment, reasoning, and generation @xcite . despite extensive research efforts, one task that has received limited attention is argument scheme classification. argument schemes provide structured templates capturing stereotypical forms of arguments consisting of inferences from premise(s) to a conclusion. these schemes have traditionally been used in formal logic-based argumentation to support reasoning and deliberation @xcite , with particular emphasis on supporting dialogues for value alignment @xcite , but have also been explored within the nlp field. classifying argument schemes (i.e. classifying natural language arguments according to the schemes they are instances of) enhances understanding of the reasoning process behind arguments and is an important step towards supporting human and ai uses of argument in reasoning and deliberation. additionally, scheme classification helps in assessing the validity of arguments (including identifying fallacious reasoning) and in strengthening their quality and scope. classifying argument schemes can enhance decision-making in ai systems, particularly in domains requiring complex reasoning, such as legal and medical applications. incorporating argument schemes into decision-making processes promotes transparency and explainability, which are essential in highstakes applications, such as healthcare, finance, and governance. various taxonomies have been proposed to systematically categorise these schemes; the most prevalent being the taxonomy developed by @xcite . however, classifying argument schemes is a particularly challenging task @xcite @xcite . indeed, the cognitive load and resources associated with this task is higher compared to other tasks such as identifying distinct components of arguments and their stance towards a topic or other arguments @xcite . large language models (llms) are increasingly being used to automate resource intensive cognitively demanding tasks. in particular, they have shown impressive in-context learning (icl) ability, whereby the model learns to perform tasks from prompts consisting of input-output examples, significantly advancing the nlp field. given their remarkable results in a variety of tasks, recent works have begun evaluating the performance of llms on several computational argumentation tasks, including argument component classification @xcite @xcite , fallacy classification @xcite , argument validity and novelty @xcite , argument quality @xcite , stance detection @xcite , and argument generation @xcite . despite these advances, use of llms for reasoning with and about arguments, in particular via use of argument schemes, remains relatively understudied @xcite @xcite . in this paper, we systematically evaluate whether llms are capable of classifying argument schemes based on the taxonomy developed by @xcite . we examine the performance of llm models for argument scheme classification in zero-shot, few-shot, and chain-of-thought prompting. moreover, we explore two approaches for enhancing the task instructions: using formal definitions and llm-generated descriptions of argument schemes. we evaluate seven open-source and proprietary models that vary in size: llama-3.2 @xcite , @xcite , claude-3.5 sonnet @xcite , gpt-4 @xcite , @xcite , gemini-1.5-pro @xcite , and grok-2 (x.ai, 2024). furthermore, we analyse which argument schemes pose a challenge to the models. our experiments on both manually annotated and automatically generated arguments, as well as enthymemes (i.e. natural language arguments in which premises and/or the conclusion of the argument is left implicit) reveal that claude-3.5 performs consistently better compared to the other models, although there is room for improvement in this task. in summary, our contributions are as follows. ## background argument schemes (as) represent stereotypical patterns of reasoning that occur in everyday arguments @xcite . their origin traces back to aristotle's topics, and have become pivotal in modern-day argumentation theory @xcite . taxonomies for systematically categorising as include the pragmadialectical classification @xcite , the periodic table of arguments @xcite , and walton's taxonomy @xcite . in this paper we focus on the latter, which proposes over 60 argument schemes each of which relate premises to a conclusion, and each of which have associated critical questions that identify how to challenge arguments that are instances of the scheme. enthymemes in natural language arguments, either some of the premises and/or the conclusion are left implicit. such arguments are referred to as enthymemes @xcite . for example, consider the following enthymeme taken from a debate on the topic should the death penalty be abolished?: a: "the death penalty negatively affects both the families of the victims as well as the defendant(s)." the argument's conclusion, "therefore the death penalty should be abolished", is left implicit. it is assumed that a reader will infer the intended conclusion from context (i.e. the debate topic). ## conclusion in this paper, we evaluated the ability of large language models to identify argument schemes. our comprehensive evaluation covered zero-shot, few-shot, and chain-of-thought prompting methods across seven open-source and proprietary models on manually annotated human-made arguments, including enthymemes, as well as automatically generated arguments. furthermore, we explored two approaches for enhancing the task instructions: normative information (i.e. the formal definitions of argument schemes as per walton's taxonomy) and llm-generated descriptions of argument schemes. our analysis revealed that larger models can identify argument schemes satisfactorily in few-shot settings when given descriptions of argument schemes, in contrast to smaller models which struggle more. given the inherent complexity of recognising argument schemes, a task that poses challenges even for human annotators @xcite @xcite , our findings are promising. in future work, we plan to extend our analysis to other schemes and taxonomies for a more comprehensive evaluation. furthermore, we plan to investigate argument generation using argument schemes and their associated critical questions as well as the potential of large language models for appropriate use of these schemes in dialogues.