Contextual Interaction for Argument Post Quality Assessment (2023)

Abstract: Recently, there has been an increased emphasis on assessing the quality of natural language arguments. Existing approaches primarily focus on evaluating the quality of individual argument posts. However, they often fall short when it comes to effectively distinguishing arguments that possess a narrow quality margin. To address this limitation, this paper delves into two alternative methods for modeling the relative quality of different arguments. These approaches include: 1) Supervised contrastive learning that captures the intricate interactions between arguments. By incorporating this approach, we aim to enhance the assessment of argument quality by effectively distinguishing between arguments with subtle differences in quality. 2) Large language models (LLMs) with in-context examples that harness the power of LLMs and enrich them with in-context examples. Through extensive evaluation and analysis on the publicly available IBM-Rank-30k dataset, we demonstrate the superiority of our contrastive argument quality assessment approach over state-of-the-art baselines. On the other hand, while LLMs with in-context examples showcase a commendable ability to identify high-quality argument posts, they exhibit relatively limited efficacy in discerning between argument posts with a narrow quality gap.

## introduction given the highly subjective nature of argumentation, arriving at a standard answer for a contentious topic is often challenging, as diverse opinions exist. consequently, assessing the quality of arguments is a complex task, as it necessitates assessing not only the relationship between arguments and the topic at hand, but also the quality of the argument itself @xcite . nowadays, there are many debate websites on the internet, such as quora, kialo, and zhihu. usually, those websites present multiple arguments in a masonry layout. people tend to view those arguments in a very short period of time, then give those arguments an overall estimation of their strength based on their impression of the relative quality of different arguments. in this regard, on the internet, when confronted with a large volume of text, people's evaluations of the quality of arguments often stem from quick impressions rather than careful consideration after thorough reading and thinking. argumentation is frequently perceived as a tool for facilitating various forms of reasoning, including decision-making and persuasion. however, these approaches often assume that the individuals involved will exhibit purely rational behavior. in contrast, human behavior is known to blend rational and emotional elements in guiding their actions. it has been suggested that a substantial link exists between the process of argumentation and the emotions experienced by the participants in that process. according to the study of social science @xcite @xcite , the strength of an argument relies much more on the argument's emotional appeal rather than the argument's logical coherence, which means it is hard to capture the quality of arguments only with their own context. the first impression made by comparing different arguments plays an important role in assessing the argument's quality. however, how to model this comparing procedure remains uncharted. indeed, most current aq assessment approaches @xcite @xcite consider an argument's quality based on its own context under the restricted perspective of logical, rhetorical, or dialectical. only a few works tried to solve the aq assessment problem through a monolithic view @xcite . some tried to capture a feature that may affect the quality of an argument, such as argument structure @xcite and discourse structure @xcite . however, revealing those features requires extra annotation costs under the guidance of linguistic specialists which makes those methods hard to be applied to large-scale datasets. since argument quality is a relative concept, it's hard to distinguish the slight difference between them, especially for arguments with subtle differences. this study presents a novel investigation into measuring the nuanced differences between arguments for quality assessment. specifically, we examine the contextual interaction of content between various argument posts to enhance the quality assessment process. we explore two different approaches to simulate the comparison of different contexts as humans evaluate multiple arguments: 1) supervised contrastive learning for cross-argument interaction that pulls together those arguments with similar quality, while pushing apart other arguments whose quality is at both ends. 2) utilization of large language models (llms) with in-context examples for the aq assessment. extensive evaluation on the standard ibm-rank-30k dataset @xcite demonstrates that contrastive learning surpasses state-of-the-art baselines in terms of overall assessment accuracy across arguments of varying quality ranges, as well as in distinguishing arguments with similar quality. while llms with in-context examples exhibit effectiveness in recognizing arguments at extreme quality ends, they fall short compared to contrastive learning when quantifying argument quality and differentiating between arguments with close quality gaps. major contributions of this paper are: 1) an investigation of modeling contextual interaction for the aq assessment. 2) proposal of a contrastive interaction approach that outperforms state-of-theart baselines on the ibm-rank-30k dataset. 3) an exploration of llms with in-context examples for the aq assessment underscores the limited performance of llms in quantitatively recognizing the subtle difference between arguments. ## related work argument quality (aq) assessment. the problem of creating a convincing argument has its origins in ancient greece, where the persuasiveness of arguments was discussed through dialectic and rhetoric @xcite . based on classical theories on arguments @xcite @xcite , @xcite identify logical, rhetorical, and dialectical as the three aspects of aq. with recent progress in natural language processing, aq has been studied and applied in various domains, including student essays @xcite , news editorials @xcite , and social media discussions @xcite . in the current research on aq assessment, the main focus has been on studying it as a sub-task in argument mining (am). however, due to the extreme subjectivity of aq, there is no clear definition for it. as a result, it is believed that there are various factors that can influence aq. @xcite summarized 15 factors that affect argument quality, categorizing them into logical, rhetorical, and dialectical aspects. @xcite assessed argument quality specifically in terms of sufficiency with human efforts, hypothesizing that the conclusion of a sufficient argument can be derived from its premises. @xcite assessed the persuasiveness of arguments by analyzing their argument structure using a factor graph model. on the other hand, @xcite focused on explicating implicit reasoning (warrants) in arguments with the help of trained experts. @xcite attempted to enhance the aq assessment performance by incorporating knowledge from other dimensions into the prediction process through multi-task learning. these studies consider each perspective of argument quality separately, which limits their holistic view of the concept. additionally, some of these approaches require additional annotations @xcite , bringing difficulty in application to large-scale datasets. cross-document interaction. the idea of modeling cross-document interactions has been widely explored in machine learning and learning to rank tasks. @xcite proposed setrank, which utilizes a self-attention mechanism to capture local context information from cross-document interactions and learn permutation equivalent representations for the input documents. in a related area, @xcite introduced unsupervised contrastive learning, a method for extracting useful representations from high-dimensional data. this technique has been influential in various domains and has shown promising results in learning meaningful representations. building on the concept of contrastive learning, @xcite extended it to the supervised setting, enabling effective utilization of label information. large language models with in-context examples. in-context learning (icl) @xcite has achieved tremendous success on large language models (llms). the main concept behind in-context learning is to leverage analogies for learning. in-context learning uses a small number of examples to create a demonstration context, often expressed through natural language templates @xcite . specifically, a query question and demonstration context are concatenated to form a prompt, which is then inputted into an llm for prediction. besides, to enhance the efficiency and performance of existing llms, especially for non-opensource apis like chatgpt and gpt-4, overprompt @xcite enables parallel processing of multiple inputs within a single prompt using in-context learning. @xcite aim to explore a more effective strategy for carefully selecting in-context examples, which can amplify gpt-3's in-context learning capabilities more efficiently than random sampling. recently, some studies have employed llms for ranking and quality assessment tasks. @xcite examined the use of generative language models like chatgpt and gpt-4 for relevance ranking in information retrieval (ir). @xcite utilized the gpt series models to assess the quality of machine translation and proposed the gemba metric. ## discussion distribution of predicted scores with argument interaction is closer to the real distribution. the histogram presented in figure 4 illustrates that the fine-tuned roberta-base generates quality logits that are more polarized compared to the actual data distribution. in the ground truth, top arguments (@xmath0.9) make up about 34% of the distribution, while in the prediction without argument interaction, they account for nearly 50%. this suggests that the model fails to distinguish between decent and good arguments due to the difference in expression skills and the resulting emotional responses from readers.the histogram of the model with contrastive learning exhibits a distribution closer to the real data. it identifies top arguments at approximately 36%, indicating better differentiation between decent and good arguments compared to the roberta-base model. these findings demonstrate that incorporating contrastive learning enables the model to accurately capture subtle differences in quality and provide more precise assessments. the potential of llms in aq assessment tasks. llms have shown immense potential in argument quality (aq) assessment tasks, particularly in argument comparison. as evident from the results in to evaluate the ability of llms to assess the quality of arguments under the same topic, we conducted experiments on the topic "we should adopt libertarianism". due to the limited number of arguments available for a single topic (approximately 300 to 400), we conducted experiments on the argument quality comparison task by selecting 100 ar-gument pairs for each of the three score differences under 3 examples. despite the relatively limited experimental sample size, which led to some fluctuations in the results, the experimental findings are largely consistent with the conclusions obtained in this observation suggests that llms, despite their proficiency in open-ended text generation @xcite , may lack precision in assigning precise aq scores, leading to relatively lower performance in aq ranking tasks. ## conclusion in this study, we explored two alternative methods for assessing the quality of natural language arguments: supervised contrastive learning and large language models (llms) with in-context examples. comprehensive evaluation highlights the importance of considering contextual interactions for evaluating argument quality. we found that supervised contrastive learning, which captures intricate interactions between arguments, outperformed state-of-the-art baselines in evaluating argument quality. however, llms with in-context examples showed limitations in quantifying argument quality and distinguishing between arguments with a narrow quality gap.