Harnessing Toulmin’s theory for zero-shot argument explication (2024)

Abstract: To better analyze informal arguments on public forums, we propose the task of argument explication, which makes explicit a text’s argumentative structure and implicit reasoning by outputting triples of propositions ⟨claim, reason warrant⟩. The three slots, or argument components, are derived from the widely known Toulmin (1958) model of argumentation. While prior research applies Toulmin or related theories to annotate datasets and train supervised models, we develop an effective method to prompt generative large language models (LMs) to output explicitly named argument components proposed by Toulmin by prompting with the theory name (e.g., ‘According to Toulmin model’). We evaluate the outputs’ coverage and validity through a human study and automatic evaluation based on prior argumentation datasets and perform robustness checks over alternative LMs, prompts, and argumentation theories. Finally, we conduct a proof-of-concept case study to extract an interpretable argumentation (hyper)graph from a large corpus of critical public comments on whether to allow the COVID-19 vaccine for children, suggesting future directions for corpus analysis and argument visualization.

## introduction advances in computational methods for analyzing arguments have benefited various applications spanning debating technologies @xcite , policymaking @xcite , information retrieval @xcite , essay writing support @xcite and legal decision making @xcite . however, unlike these domains with wellwritten arguments, web discourse on social media and public forums may feature arguments from inexperienced writers, often consisting of unclear argumentative structures and reasoning, making argument analysis quite challenging. manual interpretation of such arguments is especially problematic in erulemaking, where government officials are required to make sense of large amounts of public feedback @xcite . to help automate the analysis of such informal arguments, we propose the task of argument explication, which involves making the structure and implicit reasoning of an argument explicit. in particular, we decompose a natural language argument into its claim and reasons. we further elucidate its reasoning by explicitly stating an implicit warrant that logically links a reason to the claim. argument explication can be useful for many applications. for instance, as shown in figure 1 , it can help lay out the reasoning involved in public comments, enabling quick comprehension of arguments being made. it could help identify fallacious arguments by clearly laying out an argument's logical structure @xcite , or aid theme discovery by improving text representation with implicit content @xcite . it can also assist other nlp tasks (e.g., question-answering), where the explicated output could serve as intermediate reasoning, a method that has been demonstrated to improve downstream lm performance @xcite . traditionally, several argumentation theories (e.g., @xcite @xcite have been proposed to analyze arguments, guiding the development of training datasets and supervised models trained on them @xcite @xcite . recent advances in nlp, driven by large language models @xcite , have led to a new modeling approach using specific keywords or phrases as prompts to guide model responses @xcite , with little or no training data. this approach is especially promising for argument analysis, traditionally dependent on bespoke and smaller datasets @xcite compared to other nlp tasks, as it could enable the analysis of unstructured argumentative texts without requiring extensive domain-specific annotations. however, the pathway from explicit argumentation theories to prompting-based model design in the era of lms is less well-defined. in this work, we harness toulmin's model of argumentation for zero-shot argument explication. toulmin's theory proposes a schema to analyze arguments and has been commonly used to annotate real-world arguments @xcite , suggesting its practical utility. this theory also has a substantial scholarly impact; for example, google scholar citations for toulmin's theory (21,177) are close to @xcite . beyond academic communities, this theory is also widely popular for its pedagogical use @xcite . for instance, in a random sample of 100 documents from c4 foot_2 @xcite mentioning toulmin, we find that 21% contain worked-out examples of toulmin-style argument breakdown, potentially serving as supervised training data in lms' pretraining corpora. motivated by these observations, we investigate the use of toulmin's theory for the zero-shot argument explication task. our major contributions include: our work is related to several areas: argument mining involves claim-reason identification from an input argument and thus focuses on analyzing explicit content @xcite @xcite , while our task requires generating implicit information as well. argument reasoning only focuses on generating implicit information, while assuming a prior knowledge of claim and reason @xcite @xcite , which is not available when analyzing real-world arguments. in contrast, our task requires identifying claim-reason pairs before generating implicit information. @xcite investigate the end-toend task of identifying the structure and reasoning of an argument, however only theoretically. @xcite address some relevant subtasks proposed by @xcite , however, they also assume pre-identified claim-reason pairs. argument synthesis involves generating an argument from scratch @xcite @xcite , while our task involves generating output conditioned on an input argument. argument mapping: while prior work has also explored visualizing arguments as maps, they have mainly focused on visualizing individual arguments @xcite or supporting online collaborative tools, where members of a community work together to manually build an argument map @xcite . in contrast, we aim to automate the construction of a corpus-level argument hypergraph by analyzing arguments within an existing corpus. lms for computational argumentation have just started being explored. @xcite treat argument mining as a classification task, and do not consider generating implicit information. @xcite consider augmenting an argument with implicit information using lms, though only focus on explicating discourse markers. lms for reasoning: several prompting frameworks, such as chain of thought (cot) prompting @xcite and react prompting @xcite , have been proposed to elicit multi-step reasoning chains from lms by using specific phrases like 'let's think step by step.' our work is analogous in that we also propose prompts (e.g., 'according to toulmin model') that elicit the reasoning involved in an argument. however, one significant difference is that the generated toulmin breakdown of an argument has a fixed number of components, in contrast to cot or re-act that generate multi-step reasoning chains of varying lengths and forms. ## argument explication task explicating an argument involves a) identifying its structure: determining its claim and reasons, and b) explaining its reasoning: making explicit any implicit information connecting the reason to the claim. following several argumentation theories, we propose decomposing an argument into three core components: the claim (c) is a normative assertion or point of view put forward by the author for general acceptance. it is also known as conclusion @xcite @xcite . a reason (r i ) is a proposition provided by the author to convince the audience why they should accept the claim. @xcite refers to it as data, and later grounds in @xcite ; others use the term premise @xcite . as explained by toulmin, 'the data represent what we have to go on.' the warrant (w i ) provides a logical link between reason and claim, encoding the author's current presupposed world knowledge that explains why claim (c) follows from the provided reason (r i ). a warrant is a missing piece of information that is taken for granted by the author and is assumed common knowledge, yet if it fails to hold, c cannot be inferred from the r i . as per @xcite : 'data are appealed to explicitly, warrants implicitly.' it is also similar to @xcite 's major premise. the above three core components are conceptually supported across multiple theories, including @xcite , @xcite , and @xcite , though use different terminology. 4 we tend to use terminology closer to toulmin's version. we consider singled-authored arguments proposing a single claim, in line with public comments, where the majority of them express support or objection to a specific policy issue. the author may give one or more reasons to support this claim. for every claim-reason pair, there exists a corresponding warrant. in cases where a reason sufficiently supports the claim, a trivial warrant of the form 'if reason then claim' may suffice. formally, the task input is a textual argument t , and its output is a collection of explication triples, @xmath0, r i , w i 〉} ∀ @xmath1, with the same claim appearing in all triples. figure 2 illustrates two connected triples for an input argument. ## method: a two-stage framework we explicate an argument in two stages. in stage 1, we generate the claim (c) and all the reasons (r i ) from the input argument. in stage 2, we generate the warrant (w i ) between each claim-reason pair identified in stage 1. 5 stage 2 facilitates the generation of warrants for every claim-reason pair, enabling us to fully explicate the argument. in both stages, we prompt lms in a zero-shot setting, prompting with references to toulmin's theory, as elaborated next. ## case study: making sense of public opinion via argument explication we now illustrate the use of argument explication by analyzing public comments to the fda on covid-19 vaccine approval for children. prior studies used clustering (hoyle et al., 2023) and topic modeling (pacheco et al., 2022) to identify the main beliefs (or propositions) held by the public. however, comments are often argumentative, with inferential relations among propositions. knowing how propositions are interconnected in the broader debate can identify not only what people believe, but also why. for example, if the public health policy has to reduce vaccine hesitancy, officials must know how propositions interconnect in a broader discussion to knock down fallacious arguments. method: we use hoyle et al. (2023)'s corpus of 10,000 public comments sourced from regulations.gov, exhibiting a general vaccine hesitancy. we generate explication triples, 〈c, r i , w i 〉, from all comments, via the method outlined in §4.1 using gpt-4, excluding single-sentence comments which are often non-argumentative (refinement of this step left for future work). we cluster embeddings foot_10 of all propositions from the triples, irrespective of their role in triple, using dp-means clustering @xcite , which automatically determines the number of clusters based on a euclidean distance threshold. we use a threshold of 0.5, selected via visual inspection of cluster quality. from 9,187 comments, we obtain 14,137 triples and 308 propositional clusters. to identify interconnections between clusters, we represent a proposition with its cluster id followed by transforming triples of propositions into triples of cluster ids (tids). each tid, comprised of three cluster ids, represents a local argument structure mentioned in one or more comments and reveals inferential relations among the corresponding clusters. overall, we obtain 6,811 unique tids, visualized as a hypergraph, where a tid forms a hyperedge and a node is a propositional cluster. interpretive analysis of the corpus based on the hypergraph: we find several insights from the obtained hypergraph. among all the tids, 1,862 appear in more than one comment, suggesting that people not only share common beliefs but also use similar argument structures to support their beliefs. figure 1 shows a fragment of the larger argument hypergraph around the most common argument, (@xmath2, @xmath3, @xmath4, which occurs 373 times; it opposes vaccine approval (@xmath5. some comments further elaborate on the backing for p2, by citing low mortality rates from covid-19 among children (p8), obtained by citing data from government websites. countering any node in this chain could knock down the entire argument chain. on further exploring the local neighborhood of p1, we find two other frequently mentioned reasons: vaccine side-effects (p7) and lack of long-term testing (p3), consistent with findings from studies of social media discussion on vaccines @xcite , conferring convergent validity to our approach from a different source. explicitly stating warrants also helps reveal the relationship between distinct parts of the hyper-graph. 18 since we cluster all propositions irrespective of their role in a comment, some clusters include both implicit and explicit propositions. for instance, cluster p5 (vaccines are unnecessary for children) includes propositions implied in some comments, while explicit in others. thus, such clusters bridge distinct parts of the hypergraph. overall, we find corpus visualization as a hypergraph promising direction for future work. graph visualization (among concepts, entities, etc) has been proposed for exploratory corpus analysis @xcite . complementary to these efforts, our approach can visualize the structure of interrelated arguments, and help analysts research complex questions concerning cluster relations (e.g., 'why do people think covid-19 does not affect children?'). ## conclusion in this work, we analyze arguments by making explicit their structure and reasoning, employing lms in a zero-shot setting, and using references to toulmin's theory as prompts. we validate our approach via robustness across different references and theories. here, our case study focuses on applying computational argument modeling to analyze public comments, which is useful for civic decision-making. argument explication may also have potential in other formal, argument-rich areas, such as law or peer reviews, where authors often include explicit rebuttals and counter-rebuttals. future work could extend our approach to include these additional components and utilize this explicit argument structure. overall, the visualization and navigation of recurring, proposition-level argument hypergraphs could aid interpretive work and content analysis in the computational social sciences.