Let’s discuss! Quality Dimensions and Annotated Datasets for Computational Argument Quality Assessment (2024)

Abstract: Research in the computational assessment of Argumentation Quality has gained popularity over the last ten years. Various quality dimensions have been explored through the creation of domain-specific datasets and assessment methods. We survey the related literature (211 publications and 32 datasets), while addressing potential overlaps and blurry boundaries to related domains. This paper provides a representative overview of the state of the art in Computational Argument Quality Assessment with a focus on quality dimensions and annotated datasets. The aim of the survey is to identify research gaps and to aid future discussions and work in the domain.

## introduction argumentation is both a key competence and an important cultural technique in democratic societies @xcite . it serves as a fundamental device for expressing beliefs, perspectives, or justifications around a specific claim. the primary goal of argumentation is to strengthen or weaken the acceptability of a position by presenting supporting or opposing evidence @xcite . in recent years, the field of argument mining (am), i.e., the extraction of arguments from natural language text, has made significant progress (e.g., @xcite @xcite . however, the automatic quality assessment of argumentation (i.e., argumentation quality -aq) is still an open challenge, since defining and measuring the quality of an argument is complex and multifaceted, involving aspects such as logical soundness, persuasiveness, and dialectical reasonableness (e.g., @xcite @xcite . the insights and approaches from am and aq have been applied to various research directions. argument search (as) (e.g., @xcite makes use of the quality of arguments as an additional aid for the ranking of mined arguments @xcite . argument improvement (aimp) focuses on the analysis of the quality of an argument in terms of their improvement from a previous version @xcite @xcite . this paper surveys the literature on computational argument quality assessment. as our main contributions (1) we summarize the development of the field and the quality dimensions overtime, (2) we provide a detailed analysis of the existing annotated datasets in regards to their size, language, quality dimensions, annotation scales, annotation process, and availability, and (3) we identify significant research gaps and propose concrete research directions to guide future work in this domain. ## methodology we defined the scope of our survey as the scope of the view of computer science on aq and selected the digital bibliography and library project (dblp) foot_0 as our initial source of publication on the topic. we queried the bibliography using the search term "argument quality" and collected a total of 80 items (i.e., journal articles, conference and workshop papers, informal and other publications) dated up until the end of march 2024 foot_1 . the search function treats the words "argument" and "quality" as individual substrings and matches them to any of the collected metadata (e.g., author, title, venue, type, access, volume, year, url), thus implicitly expanding the scope of the found publications beyond the strict domain of aq. we excluded duplicated items (e.g., preprints), unavailable entries or those not referring to actual publications, and work not directly concerning the aq domain (see figure 1 ). further, we expanded the scope of our study by applying snowball sampling @xcite in an iterative manner. here, in a first step, we extracted references to prior related work. we then filtered out any publications which were not directly related to the domains argumentation quality, argument mining and argument improvement, such as ones from the philosophical field. at this point we refrained from narrowing down the discovered further references only to the aq domain in order to achieve a more representative overview. in a second step, we extracted references from the newly collected publications and proceeded with the same filtering approach from the first step. this approach was iterated for all publications until no new relevant references were discovered. this resulted in a total number of 211 publications (see appendix a for a complete list). lastly, for the further analysis presented in our paper, we focused particularly on the publications that are relevant to the computational assessment of aq. figure 2 offers an overview of the publications related to the aq domain over the last 20 years. starting from 2003, the blue dashed line describes all publications found in dblp, the solid green line -all publications found in dblp and the ones collected via snowballing, and the red bars indicate the number of datasets released per year. despite the fluctuations in the numbers over the years, we believe that the plots indicate a still rising interest in the field of aq. we consider this collection to be representative of the state of the art in computational argument quality assessment to the best of our knowledge but make no claim to completeness. the following section provides an overview of previous research by examining the proposed quality dimensions. ## dimensions for computational argumentation quality assessment the applications in the nlp community have been explored in various context. in the field of aq, persing and ng first put their main focus on automated essay scoring. they began refining the holistic scoring schemes used by scoring engines at that time by addressing more specific quality dimensions -starting with organization @xcite , thesis clarity @xcite , prompt adherence @xcite , and argument strength @xcite . the latter shifts the focus towards argumentative essays, while the others explore essays in general. nevertheless, slowly but surely the analysis of texts went beyond the assessment of structural aspects (something that still remains the strong focus of am) and began looking for other means to measure the quality of arguments. as more authors took interest in argumentative texts, they began deriving those from reviews, forum posts, etc. @xcite , for instance, explored sentiment, while @xcite took interest in the level of support present in recommendations or opinions. with the increasing number of interactions taking place online, the interest in these interactions remained and a number of publications looked at dimensions such as persuasiveness (e.g., @xcite and convincingness (e.g., @xcite ) as means to assess the quality of arguments. in contrast to prior work on persuasion, el @xcite and @xcite 's work accounts for external factors such as the prior belief of the readers and aims to incorporate and explain subjectivity in the assessment. a particular interest on convincingness and recommendedness is noticeable in work by researchers at ibm (e.g., @xcite , who actively collect the raw data (i.e., unlabeled data) through crowdsourcing, instead of extracting it from forums or similar. in parallel to these developments, which were rather based on intuitive understandings of the targeted quality dimensions, @xcite and @xcite based their work on theoretical frameworks previously proposed in philosophy. following a detailed research on quality dimensions for computational assessment and several major theories for argumentation, @xcite created a new taxonomy. their concept revolves around three high-level quality dimensions defined by blair (2011) -logical, rhetorical, and dialectical -and adds one more layer of sub-dimensions to each of them (e.g., logical: acceptability, relevance, and sufficiency). with a brief delay, this taxonomy became a significant reference and found applications for the creation of new datasets. for instance @xcite aimed at a domain-diverse set of texts, while alhamzeh (2023) assessed conference calls in the financial domain. as such the work in the field of aq once again shifted from an assessment (mostly) of a single quality dimension to the consideration (and thus annotation) of multiple at the same time. enhanced information utilization for aq assessment. prior work @xcite @xcite has ar-gued that it is beneficial to add context to argument via relevant external knowledge in order to better assess its quality. on the contrary, others (e.g. @xcite @xcite ) extract syntactic features (e.g., sentence length, vocabulary richness) from the provided text to aid the quality assessment. further, @xcite take a look at the syntax of arguments (i.e., the arrangement of words) and the coherence between said argument and the topic it relates to. the authors acknowledge that the two aspects may be representative of the cogency, reasonableness and clarity, and demonstrate that "incorporating both syntactic and coherence information can boost the classification performance compared to the models without considering them". thus, in recent years the focus seems to be moving towards an even more targeted fine-grained analysis of the dimensions, allowing not only to rate a single quality aspect (e.g., appropriateness, sentiment), but to consider potentially related factors (e.g., toxic emotions, aggressiveness) @xcite . with the development of a commonly used coding scheme, @xcite address one "key requirement in enhancing reusability" @xcite . the authors state that they do "not propose a specific approach to assess quality; rather [define] a common ground by providing a (...) holistic view" @xcite . argument quality taxonomy. due to the ongoing widespread use of @xcite 's taxonomy in the domain, we base our overview of quality dimensions on their proposed taxonomy and extend it with further dimensions identified in the surveyed papers foot_2 . figure 3 depicts all of the quality dimensions found in the surveyed papers. here, it is essential to point out that it is far beyond the scope of this survey to propose a novel or updated taxonomy. yet, we believe that developments in the field, similarly to such in the field of philosophy, would sooner or later inevitably lead to adaptations to the first taxonomy proposed by @xcite . the taxonomy differentiates between three highlevel quality dimensions. logic / cogency assesses whether the premises within an argument are acceptable, relevant and sufficient to its conclusion. dialectic / reasonableness explores whether an argumentation would be accepted by the target audience, whether it is relevant to the issue and sufficient against counter-arguments. rhetoric / effectiveness addresses the persuasiveness of an argumentation towards a target audience in terms of the credibility of the author, the emotional appeal, the style and the appropriateness of the used language, and the arrangement of the argumentation. the high-level quality dimensions are then split into sub-dimensions, which look at individual quality aspects more precisely. the interested reader may refer to the work of @xcite for the exact definitions of the individual (sub-)dimensions. throughout the creation of this taxonomy, @xcite analyse existing approaches to aq, yet many of them are not included within the final taxonomy. instead the authors leave out some dimensions such as persuasiveness as they are deemed too close to a high-level dimension and add references to further quality dimensions previously addressed by the aq literature. our overview includes the taxonomy by @xcite , all quality dimensions explored throughout the creation of the taxonomy, as well as further dimensions discovered as part of this survey (i.e., sentiment, objectivity, impact, strength). as such we aim to recognize their frequent appearance and relevance in the domain (e.g., @xcite . related fields. in our analysis we categorize work targeting the identification of argument components as am and work targeting the comparison between versions of the same argumentation as aimp. some prior work has built connections between the fields. the work of @xcite and @xcite for instance bridges the focus points of am and aq by examining the relation between the discourse structure of arguments to persuasiveness. others describe the analysis of the structure of the arguments as "the first step in analysing [argument] quality." our overview acknowledges the close connection and oftentimes blurred boundaries between aq, am and aimp, while taking a closer look at the individual dimensions that are relevant to the quality of the argumentation itself. further, it may be argued that @xcite @xcite on student essays is more related to automated essay scoring (aes) than aq, yet it has (also) been widely accepted by prior work as a contribution to the assessment of argumentation quality. the boundary between domains is further blurred when essays are evaluated based on their argumentative structure and am techniques are applied @xcite . the interested reader may refer to @xcite and @xcite for a survey on am and to ramesh and sanampudi (2022) for a survey on aes. for the first survey on aq providing a "holistic view on argumentation quality assessment in natural language" see @xcite . ## annotated datasets we found a total of 57 datasets which either extended existing datasets with further annotations or were created from scratch. the main purposes that the datasets target are argument mining @xcite @xcite , argumentation quality, argument improvement @xcite @xcite , and argument search @xcite @xcite . as our focus lays on the field of aq, we take a closer look at the 32 directly relevant datasets of the total surveyed datasets in regards to their size, language, quality dimensions, annotation scales, annotation process, and identified annotation issues. the selection of the criteria was based on previous studies on annotated datasets from related domains (e.g., @xcite . our overview is based on the datasets, their annotation guidelines and the publications introducing them. the majority of publications uploaded supplemental material or made it easy to discover or request the datasets. in 2 out of the 7 attempts to reach the authors of the respective papers in order to request the datasets remained without a response, and for 1 publication none of the authors' emails are active anymore. ## conclusion while the aq domain is multi-faceted, posing a complex challenge, prior research has tackled it from various perspectives. the interdisciplinary nature of the quality assessment task offers the opportunity to gain knowledge from related research fields, while also being able to contribute back to the related fields e.g., through the creation of annotated datasets and the evaluation of hypotheses about the significance or relations of the individual quality dimensions. we present a survey of existing argumentation quality perspectives for computational assessment and annotated datasets created or suitable for the domain. we outline potential shortcomings and research gaps from prior work, and suggest future work that may be beneficial to the further development of approaches and tools.