Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing (2020)

Abstract: Though preceding work in computational argument quality (AQ) mostly focuses on assessing overall AQ, researchers agree that writers would benefit from feedback targeting individual dimensions of argumentation theory. However, a large-scale theory-based corpus and corresponding computational models are missing. We fill this gap by conducting an extensive analysis covering three diverse domains of online argumentative writing and presenting GAQCorpus: the first large-scale English multi-domain (community Q&A forums, debate forums, review forums) corpus annotated with theory-based AQ scores. We then propose the first computational approaches to theory-based assessment, which can serve as strong baselines for future work. We demonstrate the feasibility of large-scale AQ annotation, show that exploiting relations between dimensions yields performance improvements, and explore the synergies between theory-based prediction and practical AQ assessment.

## introduction providing relevant and sufficient justifications for a claim and using clear language to express reasoning are important features of everyday writing. these are components of argument quality (aq), which has been studied in many domains, such as student essays @xcite , news editorials @xcite , and debate forums @xcite . preceding work in natural language processing (nlp) and computational linguistics (cl) has mostly focused on practical aq assessment foot_0 , considering either the overall quality of arguments @xcite or a single specific conceptualization of aq, e.g., argument strength @xcite , convincingness @xcite , and relevance @xcite . however, @xcite note the need to predict quality in terms of fine-grained aspects. fine-grained prediction enables a deeper understanding of argumentation and offers specific feedback to authors aiming to improve their argumentative writing skills. for instance, authors might want to know whether their premises are sufficient with regard to their claim(s) or whether their language is appropriate. @xcite surveyed and synthesized theory-based dimensions of aq into a taxonomy of three main dimensions: cogency (logic), effectiveness (rhetoric), and reasonableness (dialectic). their initial annotation study showed that assessing these dimensions is challenging, even for experts, but that crowd workers can handle the task comparably well if the guidelines and task are simplified. given the feasibility of annotation and the recognized need for fine-grained dimensions in aq assessment, it is surprising that no further efforts in nlp and cl have been made. there is no large scale annotated corpus and, consequently, no computational model. in this work, we aim to fill this research gap by conducting an in-depth analysis of theory-based aq assessment covering overall aq and the three dimensions (logic, rhetoric, and dialectic) of the wachsmuth et al. taxonomy, and three diverse domains of online argumentative writing (q&a forums, debate forums, and review forums). drawing on existing aq theories, we address five research questions (rqs) to inform and fuel future aq annotation studies and computational aq research: rq1: can we develop a large-scale theory-based aq corpus? we conduct an extensive annotation study with trained linguists and crowd workers on 5, 295 arguments from three domains to create the grammarly argument quality corpus (gaqcorpus), the first large-scale multi-domain english corpus annotated with theory-based aq scores. rq2: are we able to develop computational models that can do theory-based aq assessment in varying domains? based on gaqcorpus, we are the first to propose computational approaches to theory-based aq assessment and show that it is possible to develop models for this task. our models can serve as strong baselines for future research and enable the field to investigate follow-up research questions. rq3: can the interrelations between the different aq dimensions be exploited in a computational setup? inspired by the hierarchical structure of the taxonomy, we explore whether the relationships between dimensions can be computationally exploited. in addition to simple single-task learning approaches, we study the effect of jointly predicting aq dimensions in two variants (flat vs. hierarchical) and find that combining the training signals of all four aspects benefits theory-based aq assessment. rq4: does the corpus support training a single unified model for multi-domain evaluation? when enough data from a single domain is available, training on in-domain data is typically preferred over multi-domain. however, larger amounts of data are especially useful for complex model architectures currently prominent in nlp (e.g., bert @xcite , gpt2 @xcite ). we study these two mutually opposing effects on gaqcorpus and show that our corpus supports training a single unified model across all three domains, with improved performances in individual domains. rq5: can we empirically substantiate the idea that theory-based and practical aq assessment can learn from each other? @xcite suggest that both the practical and the theory-based views can learn from each other, but so far, this has been only tested manually. employing our models, we go one step further and conduct a bi-directional experiment employing a practical aq corpus. we demonstrate two concrete ways how theory-based and practical aq research can profit from their combination. structure. after discussing related work in §2, we describe our annotation study and resulting corpus ( §3). §4 describes the computational approaches which we employ in the experiments ( §5). last, we conclude our work and give potential directions for future work ( §6). ## related work earlier work in computational aq assessment can be divided into practical and theory-based approaches. practical approaches. recently, the field of computational aq research has been mostly driven by practical approaches that each target an individual domain. accordingly, past approaches tackle either overall quality @xcite or specific subqualities of argumentation, such as convincingness @xcite and relevance @xcite . the popularity of practical approaches can partly be attributed to the relative simplicity of crowd-sourcing annotations. much prior work has focused on aspects of student essays, including essay clarity @xcite , organization @xcite , prompt adherence @xcite , and argument strength @xcite . later, @xcite present an approach driven by detecting argumentative units, thereby demonstrating the usefulness of argument mining techniques to the problem. similarly, @xcite predict the absence of opposing arguments and @xcite predict insufficient premise support in arguments. another well-studied domain is web debates. @xcite adapt pagerank to identify argument relevance. pairwise comparison of the convincingness of debate arguments has been conducted @xcite . @xcite additionally predict why an argument receives a low persuasive power score. by explaining flaws in argumentation, they highlight the importance of explainability and specific author feedback. other approaches take into account properties of the source, i.e., the author (durmus and cardie, 2019) or the audience @xcite . in contrast, we assume that a system may not have much knowledge about the authors or audience and thus our models operate solely on the text. @xcite and @xcite present large corpora of crowd-sourced arguments and their quality. these corpora cover a variety of topics, but only within single domains. the authors emphasize overall argument quality cogency effectiveness reasonableness local acceptability local relevance local sufficiency arrangement appropriateness clarity credibility emotional appeal global acceptability global relevance global sufficiency guideline questions annotation aspects figure 1 : taxonomy of theory-based aq @xcite . questions related to each aspect guided annotators in assessing higher level dimensions. title: should 'blogging' be a capital crime? iran is considering it... stance: a government has the right to censor speech (...) text: my government doesn't give me freedom of speech, so i have to argue for this side. freedom of speech is bad because ... um ... then our leader's beliefs could be challenged. no one wants that. i mean, if everyone would just say and believe what our leader says to, we wouldn't need those firing squads altogether! everyone wins. cogency effectiveness reasonableness overall annotator 1 4 1 1 2 annotator 2 4 5 3 4 annotator 3 2 2 2 2 figure 2 : example text from our annotation pilot. linguistic expert annotators highly disagree on scoring the effectiveness dimension. theory-based approaches. rooted in classic argumentation theory, the works can according to @xcite , be categorized based on whether they related to the logical @xcite , rhetorical @xcite , or dialectical @xcite properties of an argument. @xcite were the first to survey and highlight the importance of the theory-based approach to computational aq and synthesized the argumentation-theoretic literature into a taxonomy. @xcite conducted a study in which crowd workers annotated 304 arguments for all 15 quality dimensions following @xcite , and demonstrated that the theory-based and practical aq assessment match to a large extent and that the two views can learn from each other, for instance, when it comes to more practical annotation processes for theory-based aq annotations. however, until now, no further research on computational theory-based aq assessment in nlp has been conducted, no larger-scale annotated corpus has been presented, and thus no computational model that would allow further investigation into the concrete synergies between the two perspectives exists. ## models having developed gaqcorpus to enable computational aq assessment (rq1), we address the remaining research questions by experimenting with aq models. to determine whether we can develop a computational theory-based aq model (rq2), we employ a naive length baseline, three different support vector regression (svr) models, and a bert-based @xcite model. we next investigate whether the interrelations between aq dimensions can be exploited in a computational setup (rq3), employing two multi-task bert-based models. for the bert-based models, we transform each argument into a "bert-compatible" format, i.e., into a sequence of wordpiece @xcite tokens and prepend the sequence with bert's start token @xcite ). the pooled hidden representation of the latter corresponds to the aggregated document representation. the specific details of each model are described below. ## experiments we employ the proposed architectures above to answer research questions rq2-rq5. ## conclusion and future work specific assessment of the rhetorical, logical, and dialectical perspectives on argumentative texts can inform researchers, e.g., about phenomena captured with annotations, and help people improve their writing skills. however, the field of computational aq assessment has been almost exclusively driven by practical approaches. aiming to fill this gap, in this work we advance theory-based computational aq research with the following contributions: we performed a large-scale annotation study on english argumentative texts covering debate forums, q&a forums, and business review forums. we thereby presented gaqcorpus, the largest and first multi-domain corpus annotated with theory-based aq scores (rq1). 10 next, we proposed the first computational theory-based aq models (rq2) and demonstrated that jointly predicting aq scores can improve the performance of the models (rq3) and that in most cases, models benefit from including outof-domain training data (rq4). finally, we investigated concrete synergies between the practical and the theory-based approach to aq assessment in a bi-directional experimental setup (rq5). the theory-based models can help to increase the interpretability of practical approaches, and practical approaches can be employed to increase performance of the theory-based models. in the future, we would like to deploy the models and study to what extent users can actually improve their argumentative writing by getting theory-based aq feedback. further, we will seek to develop ways of adding even finer-grained aspect scores at scale; this remains an open problem.