Creating a Domain-diverse Corpus for Theory-based Argument Quality Assessment (2020)

Abstract: Computational models of argument quality (AQ) have focused primarily on assessing the overall quality or just one specific characteristic of an argument, such as its convincingness or its clarity. However, previous work has claimed that assessment based on theoretical dimensions of argumentation could benefit writers, but developing such models has been limited by the lack of annotated data. In this work, we describe GAQCorpus, the first large, domain-diverse annotated corpus of theory-based AQ. We discuss how we designed the annotation task to reliably collect a large number of judgments with crowdsourcing, formulating theory-based guidelines that helped make subjective judgments of AQ more objective. We demonstrate how to identify arguments and adapt the annotation task for three diverse domains. Our work will inform research on theory-based argumentation annotation and enable the creation of more diverse corpora to support computational AQ assessment.

## introduction the notion of argumentation quality (aq) plays an important role in many existing argument-related downstream applications, such as argumentative writing support @xcite , automatic essay grading @xcite , and debate systems @xcite . for some of these applications, the idea is to automatically give feedback to users to help them improve their writing skills or assess their writing capabilities. for others, assessing aq is an important step in a more complex pipeline for retrieving high-quality arguments. while grading overall aq @xcite or a specific conceptualization of aq, such as prompt adherence @xcite is relatively well explored, researchers have noted the lack of work in so-called theory-based aq foot_0 @xcite , which can be represented with a taxonomy characterizing overall aq into several subdimensions and aspects, for instance, as logic and rhetoric, which therefore provides a more informative and targeted perspective. however, this holistic approach comes with the downside of higher complexity, especially when it comes to annotating textual corpora, which are required for training and developing common computational approaches (see, e.g., @xcite ). in a small study, @xcite demonstrate that theory-based aq annotations can be done both by trained experts and by crowd annotators, though the authors acknowledge the high complexity and subjectivity of the problem and call for the simplification of theory-based aq annotation in order to reliably create larger-scale corpora. to date, no work has tackled this challenge and accordingly, no larger-scale and no domain-diverse corpus of this kind exists. we aim to close this gap by describing our efforts to create grammarly argument quality corpus (gaqcorpus) @xcite , the largest and the only domain-diverse corpus consisting of 5,285 english arguments annotated with theory-based aq scores across four dimensions. building on @xcite , in this work, we modify the annotation task to be suitable for both experts and the crowd while preserving the theoretical basis of the taxonomy. we collect and annotate argumentative texts from web debate forums, as well as community questions and answers forums (cqa), and review forum texts, which are still understudied in computational aq. the latter domains can consist of rather non-canonical arguments in that they exhibit a lack of explicitness of certain argumentative components; are topic-wise more subjective; or consist of longer, more convoluted text. this makes assessing the quality of such arguments even more challenging, but downstream can result in a more robust model of computational aq. given all these challenges, we work closely with trained linguists to adapt the annotation task, iterating over how best to approach these novel domains and simplify the annotation guidelines for crowdsourcing, allowing us to collect a large number of judgments efficiently. we hope that our work fuels further research on theory-based computational aq. our approach to building gaqcorpus can inspire and inform aq annotation in new domains, enriching the domain-diversity of linguistic resources available in this space and consequently expanding computational approaches to aq. structure. we start by surveying previous aq annotation studies ( §2). next, we describe our efforts to adapt and simplify the annotation task ( §3), which is followed by a discussion of the data domains ( §4). overall argument quality cogency effectiveness reasonableness local acceptability local relevance local sufficiency arrangement appropriateness clarity credibility emotional appeal global acceptability global relevance global sufficiency figure 1 : the taxonomy of theory-based argument quality aspects @xcite . most argumentation annotation studies have been conducted on student essays or web debates. student essays have been annotated for thesis clarity @xcite , organization @xcite , and prompt adherence (persing and ng, 2014), and persing and ng (2015) model argument strength rated on a 4-point likert scale. similarly, @xcite annotate the absence of opposing arguments and @xcite predict insufficient premise support in arguments. for web debates, habernal and gurevych (2016) conduct an annotation study in which they present debate arguments pairwise to crowd annotators, who then can choose the more convincing argument. @xcite also annotate the reasons why an argument receives a low persuasive power score. @xcite developed a taxonomy of aq synthesized from traditional works in argumentation theory, such as aristotle (trans 2007). the full taxonomy is depicted in figure 1 , and defines the overall aq to consist of the following three subdimensions, each of which is itself defined by several finer-grained aq aspects: most recently, @xcite and @xcite crowdsourced overall argument quality by presenting pairwise arguments to annotators, who then had to select the argument "they would recommend a friend to use that argument as is in a speech supporting/contesting the topic." this is an extreme simplification of the task, which does not seem to lead to better agreement: the authors @xcite report an average inter-annotator agreement @xmath0.12 and attribute the low score to the high subjectivity of the task. the authors conducted a theory-based annotation study in the spirit of @xcite on a subset of the data (100 arguments) which indicated the highest correlation of the annotations with the effectiveness dimension. later on, @xcite empirically confirmed this observation using computational model predictions across the whole corpus. building on this large body of work, we aim to facilitate the annotation of theory-based aq in diverse domains of real-world argumentative writing and compare expert vs. crowd annotations. our study results in the largest english corpus annotated with theory-based argumentative quality scores. ## annotation study in this section, we detail how we developed and designed our annotation task to enable efficient, reliable collection of theory-based aq judgments with crowdsourcing. we validate @xcite 's hypothesis that crowdsourced annotation of theory-based aq is possible if the task is simplified. ## data domains in this work, we consider three domains: debate forums, cqa forums, and review forums. while debates are generally well-explored in computational aq, we are unaware of any work involving cqa and reviews. for each of these domains, we first identified items likely to be argumentative and then adjusted the guidelines in consultation with expert annotators, as described below. debate forums. of these three domains, debates is the most straightforward to annotate. given a topic or motion, users can define their stance (pro/contra) and write an argument which supports it. we included data from two online debate forums. convinceme (cm) is a subset of the iac, where users share their stance on a topic and discuss their point of view, with replies aiming to change the view of the original poster. change my view (cmv) is a reddit forum in which participants post their opinion on a topic and ask others to post replies to change their mind. we sampled original posts from cmv, skipping any moderator posts, and the first reply to an original post from cm, in order to limit the context that annotators must consider when evaluating arguments. cmv posts always include the author's perspective in the title, while cm posts may or may not include a stance in the title. in the guidelines, we instruct annotators to judge a post by how successfully it justifies the author's claim. cqa. in community questions and answers forums, users post questions or ask for advice, which other users can address. we experimented with arguments from yahoo! answers foot_5 (ya). when posting a question, a user can provide background information for their question (context) and can later indicate which response is the best answer to their question. the forum's looser structure provides for a wide variety of content, which is appealing as a potential source of non-standard arguments, but challenging as many of the posts do not contain any arguments. through manual analysis, we identified three categories that frequently contained controversial topics, hypothesizing they would have a higher incidence of debates: social @xmath1, society & @xmath2, and politics & @xmath3. we empirically selected the category with the highest proportion of arguments in a study on amazon mechanical turk (mturk). qualified annotators foot_6 decided if question and best-answer pairs were argumentative. we collected 10 judgments for 100 pairs from each category and aggregated judgments with a simple majority. law & ethics had the most argumentative posts (70%, compared to sociology with 40% and society & culture with 34%), so we sampled posts from this category to annotate. in the guidelines for this domain, we asked annotators to judge the argumentative strength of an answer with respect to how well it addressed the given question. the guidelines and subdimension questions were altered to encourage this. one obstacle in pilot studies with expert annotators was posts offering, as many users solicited legal advice in the law & ethics forum. we decided to consider advice as argumentative as long as the author supported the advice with justification, which mirrors our general approach to the argumentative dimension. reviews. the third domain consists of restaurant reviews from the yelp-challenge-dataset foot_7 . on yelp, users write reviews of businesses and rate the quality of their experience from 1 (low) to 5 (high) stars. unlike the debate and q&a forums, the format of yelp does not support dialogue between users (i.e., users cannot directly reply to other users or posts), and so it is possible to present each post in isolation as a self-contained argument. as most posts do not explicitly state a claim, we pose the star rating as a claim the user is making about the business, and the review as the argument supporting it. yelp reviews can be highly subjective in that each review is based on a single user's experience. for instance, a user may rate a restaurant as 5-stars and write only the food was delicious in their review. to address this subjectivity, we asked annotators to judge the argumentative quality of each review with respect to how well it supported the rating provided. another challenge was defining what constituted a counterargument, as these have a very different character than counterarguments in debates (for example, everyone says that the pizza crust is too thin here but that's authentic!). in consultation with our experts, experts completed a series of pilots before each domain was presented to the crowd, using the task design described in §3.1. expert agreement on novel domains (ya and yelp) are shown in ## a theory-based aq corpus applying the annotation task design and data selection described above, we created gaqcorpus, containing 5,285 arguments across three domains, annotated for theory-based dimensions. all arguments were limited to have a length between 70 and 200 characters. ratings were provided by the two groups of annotators described above, experts ( §3.1) and the crowd ( §3.2). each group judged 3,000 arguments, with about 1,000 arguments annotated by both groups for comparison. the size of the corpus is described in ## conclusion theory-based aq assessment provides a holistic and targeted perspective on aq, but its high complexity makes annotation difficult. in this work, we describe our efforts to create gaqcorpus, a multi-domain corpus of 5,295 arguments annotated for quality along theory-based aq dimensions. we demonstrate that it is possible to collect complex annotations with crowdsourcing in three domains: debate, cqa, and review forums. drawing from the initial study of @xcite , which suggested the general feasibility, we relied on the intuition of trained linguists to simplify the task and guidelines while preserving the theoretical basis of the task. the agreement between experts and the crowd was higher than the agreement in earlier studies @xcite , validating our approach and indicating that it is possible to collect complex ratings using a crowd. gaqcorpus and the findings of our annotation study will serve as a basis for future corpus development and computational model development in theory-based aq. they are available for download from https://github.com/grammarly/gaqcorpus . debates cogency 2.0 effectiveness 1.7 reasonableness 1.0 overall 1.3 title: should you need to pass an iq test to have kids?-stance: dumb parents lead to more dumb kids. text: i have a strong opinion that before having children, the prospective parents should have to pass a series of background and iq tests. kids being brought into this world need a good foundation to start a successful life with. you may have that limited case where the parents are morons and the kids strive to be different then their failure parents, but in most cases it is an endless line of parasites on our world. we need more smart people. cqa cogency 2.7 effectiveness 2.0 reasonableness 1.7 overall 2.0 question: bounced check? context: does the company holding the bounced check have to send you a certified letter before issuing a warrant for your arrest. i feel almost certain that they do but i am not sure. answer: i always make sure my checks are not printed on rubber. they are just too expensive and not worth it. we all make a mistake from time to time, and usually it is no big deal except for the extreme annoyance and all the bounced check fees. but if you are worried about an arrest warrant then i am sure you are doing this deliberately and trying to defraud the company. you have probably sent them a couple of bad checks already in an attempt to string them along so your guilt is probably pretty well established. you can hope that you do not have to share a jail cell with a gross deviate of some sort. reviews cogency 1.0 effectiveness 1.0 reasonableness 1.0 overall 1.0 title: business review: 2.0 stars. business name: cook out. city: charlotte. categories: restaurants, desserts, food, fast food, american (traditional), hot dogs, burgers review: burgers are good but i like those other 5 guys burgers instead oh and i guess if your not from around here don't even think about going thru the drive thru it's like the biggest most unreadable confusing hurried crazy thing ever if i ever go again hell with drive thru until i've lived here for at least 5 maybe 10 years and can be a veteran drive thru person i'm walking in it's like if i mix up all the letters in this review and give you 1 minute to read it and figure it out then you gotta move on.