Exploring Methodologies for Collecting High-Quality Implicit Reasoning in Arguments (2021)

Abstract: Annotation of implicit reasoning (i.e., warrant) in arguments is a critical resource to train models in gaining deeper understanding and correct interpretation of arguments. However, warrants are usually annotated in unstructured form, having no restriction on their lexical structure which sometimes makes it difficult to interpret how warrants relate to any of the information given in claim and premise. Moreover, assessing and determining better warrants from the large variety of reasoning patterns of unstructured warrants becomes a formidable task. Therefore, in order to annotate warrants in a more interpretative and restrictive way, we propose two methodologies to annotate warrants in a semi-structured form. To the best of our knowledge, we are the first to show how such semi-structured warrants can be annotated on a large scale via crowdsourcing. We demonstrate through extensive quality evaluation that our methodologies enable collecting better quality warrants in comparison to unstructured annotations. To further facilitate research towards the task of explicating warrants in arguments, we release our materials publicly (i.e., crowdsourcing guidelines and collected warrants).

## introduction implicit reasonings, commonly referred to as warrants @xcite , have long been studied to understand the grounds on which a premise lends support to the claim @xcite . in other words, a warrant, when made explicit, clearly shows the inferential link between claim and premise @xcite . as depicted in figure 1 , identification of such warrants by students has been shown to aid them in making better arguments @xcite , as well as improving their critical thinking skills @xcite and argument comprehension process @xcite . while explication of warrants with assistance from teachers has been shown to be useful for improving students' argumentation skills, automating this explication process would not only help students to be less dependent on teachers, but it can also be beneficial for different downstream educational applications such as argument analysis @xcite , enthymeme reconstruction @xcite and essay scoring @xcite . however, building an automated warrant explication system has been a challenge due to the difficulty of collecting warrants in a form that explicitly manifests the way a warrant relates the information between claim and premise. generally, warrants are annotated in an unstructured (i.e., freetext) format which lays no restriction on its lexical structure @xcite . as a result, sometimes the warrant consist of no information that overlaps with claim or premise which makes it difficult to understand how the warrant connects the claim to its premise. furthermore, for a given argument, unstructured warrants can be framed in diverse ways that would have a wide variety of reasoning patterns @xcite (given no restriction on the lexical structure), and identifying the correct ones from this large pool of warrants can be a preposterous task. in order to annotate warrants on a large-scale and in a way that overcomes the aforementioned challenges, we propose two novel warrant annotation methodologies: pre-defined keyword-based warrant (pkw) and user-defined keyword-based warrant (ukw), which restrict a warrant's lexical structure to a semi-structured form. in contrast to approaches that crowdsource unstructured warrants, these methodologies explicate warrant by enforcing it to have the key information (i.e., keywords) from both claim and premise. the intuition behind our semi-structured approach is to restrict the structure of warrants to specific keyword-based reasoning patterns. we hypothesize that leveraging such keyword-based pattern could assist in collecting high quality warrants, where keywords are derived from the original argument (i.e., claim and premise). our assumption follows the formal definition of warrants in the sense that warrants act as a inferential link between the contents of claim and its premise @xcite . many previous work demonstrated different strategies of collecting high quality warrants @xcite @xcite , but did not apply any restriction on the structure of warrants to handle the variety of reasoning patterns in which warrants can be explicated. in contrast, our annotation methodologies are designed to restrict the reasoning pattern and ensure that warrant explicates the reasoning link between claim and premise. in order to evaluate the warrants annotated via our proposed ukw and pkw methodologies, we devise specific guidelines to judge their quality through scoring. we also collect unstructured warrants (i.e., natural language warrants (nlw)) and perform quality evaluation on them in order to compare with our annotated warrants. our results suggest that in comparison to nlw, high quality warrants can be annotated via our proposed ukw methodology. to the best of our knowledge, this is the first study which targets large scale annotation of warrants in semi-structured form. to facilitate further research in warrant explication, we publish our crowdsourcing guidelines and the preliminary corpus of around 1700 warrants that are annotated via ukw methodology, covering over 600 arguments foot_0 . ## related work explication of warrants in arguments has already been approached in many previous researches. in an initial attempt, feng and hirst (2011) proposed leveraging argumentation schemes @xcite as a means to automatically reconstruct warrant, but did not approach the task due to the absence of training datasets. to overcome the unavailability of dataset, @xcite leverage crowdsourcing and ask non-expert workers to annotate all possible variations of warrants for a given claim-premise pair. however, they concluded that the annotation of warrants varied both in number annotated per argument and in content due to no restrictions imposed on in the annotation process. in order to overcome the prior difficulties, recent approaches leverage crowdsourcing to restrict the number of warrants collected per argument and either employ a step by step filtering process to weed out bad warrants @xcite or hire ex-argument warrant 1. claim: we should abolish zoos. premise: zoos are notorious for animal abuse. abolishing zoos leads to animals being in their natural habitat which results in no animal abuse. banning whaling would stop the inhumane methods of stabbing whales which is unacceptable cruelty towards animals. ## warrant desiderata we define warrant that we want to annotate by characterizing its desired properties. properties such as structure of warrant, quality in terms of how well a warrant links claim and premise, and feasibility of annotating warrant for an argument. in this work, we define this feasibility in terms of whether a warrant can be annotated for an argument, regardless of whether it is good or bad. we assume that explication of a warrant might not be possible if the argument is too good (i.e., warrant is explicated in the premise) or if the argument is too bad (i.e., no warrant can explicate the link between claim and premise). structure warrants are implicit reasoning that logically link the contents of claim and premise @xcite . therefore, we hypothesize that a warrant should have a structure that (a) comprises the key information given in claim and premise, and (b) explicates logical connection between the aforementioned key information with some implicit knowledge that is relevant to the argument. we define the warrants framed in such a way as semi-structured warrants. examples of such semi-structured warrants are provided in feasibility warrants may not be explicable for all the arguments. specifically, for an argument with bad premise there may be no feasible way to explicate the logical link between claim and premise. for example, the warrant for the argument "we should introduce multi-party system because it's the right thing to do" is not feasible, since the argument is a fallacy (i.e. begging the question) where the premise: "it's the right thing to do" provides no adequate support to the claim: "we should introduce multi-party system". similarly, for arguments with very good premise, it might not be necessary to explicate the warrant since the warrant might already be explicated in the premise. in contrast, as shown in quality a key factor distinguishing warrants from any other type of implicit knowledge or statement (e.g., commonsense knowledge) is their ability to justify the flow of reasoning between claim and premise. for example, in ## annotation methodologies in this section, we discuss the development and design of our proposed semi-structured annotation methodologies. in particular, we consider two methodologies for annotating warrants: predefined keyword-based warrant and user-defined keyword-based warrants. ## warrant collection procedure our goal is to establish a procedure for collecting semi-structured warrants and their annotations at large-scale. in order to collect such warrants from each methodology, we build a multi-step crowdsourcing process designed for encouraging annotator's creativity, while preventing biases in the annotations. in general, we break the warrant collection procedure into three steps of simple tasks: (i) deriving keywords, (ii) judging feasibility and (iii) framing warrant. in addition, we implement several mechanisms for quality assurance and employ manual checks to ensure annotators understand and perform the final task correctly. for pkw methodology, keywords in this step are already derived as shown in figure 2 . to ensure annotators understand the notion of keywords, we provide sufficient variety of examples for them to get used to this sub-task. auxiliary verification measures for each task, we hold preliminary qualification test that consists of several basic questions to judge the understanding of annotator's reasoning skills. annotators who score more than a pre-defined threshold (≥ 80%) are granted access to the main task. our qualifications are open to annotators from major english speaking countries, namely usa, uk, new zealand and canada. additionally, to address any ethical issues @xcite raised by our task, we actively monitored multiple pilot tests to ensure annotators were satisfied with our task. simultaneously, we corresponded directly to annotators that had questions/comments on our task. annotators were paid in accordance with the minimum wage calculated by conducting many trials and based on average work-time. additional bonus was paid to annotators that provided quality annotations. ## crowdsourcing we choose amazon mechanical turk (amt) 3 as our crowdsourcing platform due to its success in previous argumentation mining tasks @xcite . as an initial step, we only allowed annotators who @xmath2,000 approved human intelligence tasks (hits). each annotator was paid @xmath00.25 per hit. for the task involving ukw annotations, the payment was increased to @xmath10.35 as bonus. source data for the purpose of annotating warrants for a given set of arguments, we utilize a wellknown argumentation dataset, ibm-rank-30k corpus @xcite , which already consists arguments in the form of claim and premise. the dataset contains around 15k crowd-sourced arguments covering 71 topics, annotated for supporting as well as opposing stance. 4 the arguments were collected with strict length limitations and accompanied by extensive quality control measures. our inspection of arguments from ibm-rank-30k revealed that a for large proportion of the arguments we can explicate warrants. to proceed with our warrant annotation, we selected a subset of three 3 well-known debatable topics: we should abolish zoos, we should ban whaling and we should introduce compulsory voting. annotations in addition to warrants annotated via pkw and ukw methodologies, we also annotated unstructured warrants via crowdsourcing to compare the quality of annotated warrants across different methodologies. we followed similar crowdsourcing procedure as employed by @xcite and refer to the warrants annotated via this process as natural language warrants (nlw). for each methodology, we annotate 40 unique claim and premise pairs, randomly chosen from the three pre-selected topics. each argument was annotated by 5 annotators resulting in a total of 200 annotations per methodology and all warrants were limited to have a length between 60 and 200 characters excluding keywords. filtering for each methodology, it is possible to collect at most 200 warrants (given 40 arguments and 5 annotators). however, not all annotators methodology natural language user-defined pre-defined ## results in order to analyze the quality of the annotated warrants, we hired two annotators who are experts in the field of argumentation to score the crowdsourced warrants. to frame quality scoring guidelines for judging warrants, we ran several pilot tests and take expert advice to make our guidelines easier to interpret. our final quality annotation guidelines are shown in as shown in ## conclusion and future work in this work, we tackle the difficult task of explicating warrants in arguments and propose two novel methodologies to annotate warrants in semistructured format. we conduct extensive analysis and perform annotation study for determining the appropriate methodology for collecting warrants and show that user-defined keyword based approach produces the highest quality warrants as compared to pre-defined keyword-based warrant and natural language warrants. in future, we plan to extend the annotation of warrants for more diverse topics. moreover, we plan to cover warrant annotations for claim-premise pairs with premises attacking the claim in addition to premises supporting the original claim. we would also like to test the usefulness of our annotations for constructing a model for automatic warrant explication which can be used to explicate warrant for any given argument. we believe that such a model can be useful in a pedagogical setting to perform downstream tasks such as argument analysis or giving constructive feedback to students.