Contextualizing Argument Quality Assessment with Relevant Knowledge (2024)

Abstract: Automatic assessment of the quality of arguments has been recognized as a challenging task with significant implications for misinformation and targeted speech. While real-world arguments are tightly anchored in context, existing computational methods analyze their quality in isolation, which affects their accuracy and generalizability. We propose SPARK: a novel method for scoring argument quality based on contextualization via relevant knowledge. We devise four augmentations that leverage large language models to provide feedback, infer hidden assumptions, supply a similar-quality argument, or give a counter-argument. SPARK uses a dual-encoder Transformer architecture to enable the original argument and its augmentation to be considered jointly. Our experiments in both in-domain and zero-shot setups show that SPARK consistently outperforms existing techniques across multiple metrics

## introduction reliable analysis of arguments in natural language holds the promise to support applications such as automated grading @xcite , and tackling misinformation and targeted speech @xcite . computational argument analysis has been relatively popular through tasks like argument extraction @xcite , evidence mining @xcite , relation assignment @xcite , writing support @xcite and claim generation @xcite . a particularly challenging task is argument quality assessment @xcite , which addresses the cogency, effectiveness, and reasonableness of an argument @xcite pertaining to a topic. assessing the quality of the argument involves analyzing the objective evidence, relevant assumptions, and structural soundness, making the overall task difficult. research on argument quality assessment has focused on extracting textual patterns using various learning frameworks and content features @xcite . it has been widely recognized that contextualizing arguments with implicit knowledge, such as extracting claim revisions @xcite and generating explicit conclusions @xcite can be informative for reasoning models. however, we note that: 1) these methods fail to generalize to novel arguments where this information is not available, and 2) no prior work has considered jointly a comprehensive set of such contextualization strategies. to bridge these gaps, we propose a novel framework called spark (scoring the pragmatics of arguments via relevant knowledge), which incorporates augmentation strategies based on a large language model (llm), gpt 3.5 (ope-nai, 2022), and elements from argumentation literature @xcite @xcite , specifically, feedback, assumptions, arguments with similar quality, and counterarguments. spark processes the original argument and topic and its augmentations separately using a dual-encoder transformer architecture with a multi-head cross-attention layer. we demonstrate the effectiveness of spark 's augmentations and architecture using both in-domain and out-of-domain datasets. we make our en-tire code available at https://github.com/ usc-isi-i2/forecast-argument . dual-encoder architecture. to consider the argument together with the augmentations, we employ a dual bert encoder (figure 2 ) as an improvement to the architecture by @xcite . the first encoder embeds the topic and argument, whereas the second embeds the augmentations. the second encoder can store individual augmentations or their concatenation, arbitrarily fixed to similar quality argument [sep] feedback [sep] assumptions [sep] counter-argument. notably, the dual encoder can effectively store all of the augmentation data without truncating information in practically all cases (see subsection a.4 for the augmentation lengths). we use a multi-head crossattention layer @xcite to enable the model to weigh each augmentation according to the argument-topic pair. we pass the attention outputs to a mean pooler, whose output is fed into three separate regressor heads, one per quality metric. ## conclusions this paper enhances argument quality estimation models by providing contextualized feedback, inferred assumptions, similar quality arguments, and counter-arguments. we employ a dual-encoder transformer to compare the argument and additional evidence effectively. experimental results demonstrate that the best performance is achieved with the combination of all augmentations, indicating their complementary insights. our method, spark, outperforms single bert, xlnet, and gpt-3.5, surpassing baselines and alternative augmentations in both in-and out-of-domain scenarios. feedback augmentation is the most effective augmentation strategy for models, despite being scored the least informative by humans, because the model pattern matching differs from human reasoning.