Efficient Pairwise Annotation of Argument Quality (2020)

Abstract: We present an efficient annotation framework for argument quality, a feature difficult to be measured reliably as per previous work. A stochastic transitivity model is combined with an effective sampling strategy to infer high-quality labels with low effort from crowdsourced pairwise judgments. The model’s capabilities are showcased by compiling Webis-ArgQuality-20, an argument quality corpus that comprises scores for rhetorical, logical, dialectical, and overall quality inferred from a total of 41,859 pairwise judgments among 1,271 arguments. With up to 93% cost savings, our approach significantly outperforms existing annotation procedures. Furthermore, novel insight into argument quality is provided through statistical analysis, and a new aggregation method to infer overall quality from individual quality dimensions is proposed.

## introduction for a broad variety of tasks, such as argument mining, argument retrieval, argumentation generation, and question answering, compiling labeled data for argument quality remains an important prerequisite, yet, also a difficult problem. most commonly, human assessors have been presented with one argument at a time and then asked to assign labels on a graded quality scale 0, 1, 2 with label descriptions such as (0) "low quality", (1) "medium quality" and ( 2 ) "high quality" for guidance. in previous work, this was usually done concurrently for multiple orthogonal sub-dimensions of argument quality; judging the overall quality of an argument has been deemed complex @xcite . but on closer inspection, even the more specialized quality dimensions considered are difficult to be assessed as evidenced by the low reliability scores reported. especially crowdsourcing suffers from assessors often having different reference frames to base their judgments on and task instructions being nondescript and therefore unhelpful in ensuring consistency. employing experts, however, not only comes at a significantly higher cost per label; despite their expertise, even experts did not achieve more reliable judgments. we pursue an alternative approach: stochastic transitivity modeling based on pairwise judgments of arguments. this enables the employment of laymen; the decisions required from them remain comparably simple and expect neither prior knowledge nor a common reference frame, while the labels that can be derived from their judgments still exhibit a high accuracy and informativeness. though pairwise judgment has already been considered for assessment of argument quality @xcite , its significant cost overhead has hindered widespread application. we explore the lower bound of effort needed to infer labels of sufficient quality. we combine a pairwise model with a highly effective offline sampling strategy to minimize the set of needed pairwise judgments, saving up to 93% of the effort of an exhaustive comparison. as part of this work, we release the webis argument quality corpus 2020, which includes a total of 41,859 pairwise judgments between 1,271 arguments across the three dimensions of rhetorical, logical, and dialectical quality. further, inferred scalar scores for the three dimensions as well as overall quality and topic relevance are provided, alongside a reference implementation of our model. 1 carrying out a first analysis of the statistical properties of the corpus, we validate both the new annotation method and the corpus by drawing comparisons to previous work. since judging overall quality by itself is a difficult task, based on our statistical analysis, we find that euclidean vector length adequately combines scores from the three aforementioned specialized quality dimensions into a single overall quality score. each study operates on rather small amounts of data; they only annotate 320 @xcite and 437 @xcite individual arguments. both setups become nonviable for larger annotation tasks, since the associated labor costs in such (semi-)expert studies are usually high. this is not an issue in crowdsourced settings, where judgments can be collected in abundance for a comparatively cheap price. however, the problem of annotation quality is more severe here: argument quality might be even more difficult to judge without prior domain-specific knowledge, creating the need for annotation frameworks that can still maintain a sufficiently high data quality. judging from the agreement scores given by @xcite , obtaining reliable data using classic graded scales proves infeasible, an effect that should be even more pronounced in a crowdsourced setting. @xcite measure an arguments' quality as the amount of context or inference required for it to be understood, describing an annotation setup where assessors judge seven individual quality dimensions on a 0-1-slider. recruiting assessors on amazon mechanical turk (mturk), they use intra-class correlation to estimate inter-rater agreement, with an average value of 0.42 over all topics, thus also indicating a poor reliability @xcite . they further observe a correlation with sentence length, prompting them to remove all sentences shorter than four words. all three studies indicate that absolute rating (i.e., having assessors label a single argument on a given absolute scale without the context of other arguments) performs unfavorably. this rating method, also known as likert scale or mean opinion score, is known to have two major drawbacks @xcite : (1) absolute rating is often treated as if it produces data on an interval scale. however, assessors rarely perceive labels as equidistant, thus producing only ordinal data. this leads to a misuse of statistical tests and results in low statistical power of subsequent analyses. (2) absolute rating is difficult for assessors without prior domain knowledge, since they may be unsure which label to assign. this results in noisy, inconsistent, and unreliable data. as an alternative, preference rating (i.e., a relative comparison by showing two arguments to an assessor and letting them declare their preference towards one of them) has been considered by @xcite , who compile an exhaustive set of pairwise comparisons to infer labels for argument convincingness. for 1,052 arguments on 32 issues, each of the over 16,000 total comparisons was annotated by five different crowd workers on mturk. while no α statistics are provided, the authors do conclude that preference ratings in a crowdsourced setting are sufficiently accurate, since the best-ranked rater for each pair achieves 0.935 accuracy compared to a gold label. the indicated reliability of pairwise annotation for argument quality is further corroborated by @xcite , who compile a large dataset of about 14,000 annotated argument pairs, and absolute ratings in the 0-1-range for about 6,300 arguments. pairwise annotations were made in regard to the overall quality of arguments, operationalized as "which of the two arguments would have been preferred by most people to support/contest the topic?" using a strict quality control, they show that the annotated relations consistently reproduce the direction implied by absolute ratings. yet, annotating quality as a single feature is problematic, since (1) it is hard to capture the multi-facet nature of argument quality in that way @xcite and the chosen operationalization is similar to the facet of dialectical quality, neglecting the other major two; and (2) scores for individual quality dimensions are warranted for in-depth training and evaluation for a broad range of argumentation technology @xcite . although preference rating seems promising based on the reported reliability, it creates the need for a model that infers score labels from the collected comparison data. habernal and gurevych propose the use of pagerank @xcite . this is problematic, since cycles in the comparison graph may form rank sinks, distorting the latent rankings. habernal and gurevych deal with this problem by constructing a directed acyclic graph (dag) from the collected data prior to applying pagerank, assuming that argument convincingness exhibits the property of total order. however, no prior evidence for this property is apparent. @xcite note further problems with pagerank and propose the use of gaussian process preference learning instead, demonstrating a high scalability. however, for a practical approach, an effective strategy to minimize the number of needed comparisons is warranted, since, to build the dag, exhaustive comparison data is required. this is inefficient; at worst n 2 comparisons have to be obtained for n arguments. also, no data was collected on how the pagerank method performs on incomplete or sparse comparison data. @xcite also propose an online sampling strategy based on the bradley-terry model (bradley and @xcite . they implement an online bayesian updating scheme, which, contrary to previous work such as presented by @xcite , does not require retraining the whole model when new comparisons are added. after each comparison added to the total set of annotated pairs, they identify the next pair to be compared by calculating which comparisons would reduce the overall model uncertainty the most. @xcite opt for a similar active learning approach, but note that that it is prone to overfitting, causing accuracy to decrease. while online learning uses an approximately minimal amount of comparisons, additional drawbacks besides overfitting can be noted: (1) the updating scheme diminishes the reusability of the collected data, since such a specific method of choosing pairs introduces data bias for other applications. (2) online sampling is complicated to implement on a crowdsourcing platform, preventing multiple workers from making judgments in parallel. (3) in the case of @xcite , the model is not equipped to handle comparison ties, i.e., an assessor declaring no preference. yet, ties frequently occur in real-world annotation tasks. overall, the bradley-terry model appears to be a promising candidate for our purposes: its robustness and statistical properties have been studied in great detail @xcite , and it can be efficiently computed @xcite . however, an alternative offline sampling method has to be formulated, which we introduce in the following section. ## pairwise quality annotation in this section, we define a model to aggregate pairwise judgments into scalar ranking scores and combine different sampling strategies to form a highly efficient annotation framework. ## the webis argument quality corpus to maximize its usefulness, the sample of arguments for our argument quality corpus was drawn from the recently published args.me corpus @xcite , a collection of 387,606 arguments crawled from various debate portals. to ensure some topic diversity and relevance of the arguments to the topic, while keeping the amount of judgments within our budget limits, we (1) indexed the args.me corpus using three retrieval models from the terrier information retrieval library @xcite (namely bm25, dph, and dirich-letlm), ( 2 ) retrieved texts for 20 topic queries at a depth of 50 texts per topic per model, ( 3 ) and pooled all 3000 retrieved texts to remove overlap between the different models. in total, 1,610 unique spans of text remained for the 20 topics. using amazon's mechanical turk, in a first preprocessing step, we tasked crowd workers with deciding whether or not a given retrieved item actually contained argumentative text. each text was judged by five crowd workers, using majority vote as a decision rule. to ensure quality, we recruited only workers for the task with an approval rate of at least 95%, like @xcite . of the 1,610 input texts, 339 were flagged as nonarguments. most of these texts are noise resulting from using debate platforms as data source; examples include statements of acceptance for a debate ("i accept this debate on [topic] and will go first [...]"), statements with no argumentative value ("i think [topic] is good."), definitions, and in some cases even jokes or personal attacks. 1,271 arguments remained for quality annotation. in a second step, all remaining arguments were annotated for argument quality via a sample of pairwise judgments on which we applied our model. for each pair of arguments, a crowd worker was tasked to select the one that exhibits a higher quality compared to the other with regard to a given description of the respective quality dimensions. the annotation was repeated separately for each of 20 topics and each of the three aforementioned quality dimensions. to make the task accessible to workers without prior knowledge of argumentation theory, the quality dimensions were operationalized as follows: "which text has the better logical structure?" (logical quality), "which text has the better style of speech?" (rhetorical quality), and "which text would be more useful in a debate?" (dialectical quality). examples were given to annotators as guidance. ## corpus analysis in this section, we carry out a statistical analysis of our new corpus. first, we study the distribution and the correlation effects between the different quality dimensions, and between the quality dimensions and text length, to draw comparisons to the prior work of @xcite . then, we explore the hypothesis of overall quality being a latent variable by analyzing the influence of quality dimensions. ## conclusion a novel approach for annotating argument quality based on stochastic transitivity modeling has been proposed, outperforming existing approaches in terms of annotation effort and annotation detail, while maintaining a high annotation quality. the overall workload in comparison to previous approaches within the same class of approaches was reduced by 93.17% through an efficient sampling method. sampling at even higher rates is possible, resulting in the new framework operating at the same cost as the traditional approach relying on graded scales. the collected data and a reference implementation of our model are made available in form of the webis-argquality-20 corpus, one of the largest and most detailed corpora for pairwise argument quality. the collected corpus can be used for a multitude of purposes-especially in the emerging field of argument retrieval, it is suitable as basis for retrieval evaluation, or to train new learning to rank models. a second field of application is debate systems, where a dataset can be of use for training a system to formulate new arguments. the developed annotation approach is also not only limited to rate argument quality: it can easily be transferred to other questions or criteria that can be rated by comparison. even though the annotation cost can be slightly higher compared to the traditional absolute rating approach, the derived data is much more detailed and allows for conclusions with higher statistical power. insight into argument quality was derived on a larger scale than in previous studies. it has been shown that the three quality dimensions can be successfully annotated by laymen when using the described annotation procedure. the correlation patterns found in previous studies were reproduced, showing the quality dimensions to be equally correlating with each other. this is likely due to them being dependent on a latent overall quality, a hypothesis that was supported using a pca analysis of derived quality vectors. a procedure to derive a scalar value for overall quality was introduced, proposing euclidean vector length to combine the different dimension scores.