Developing an argument annotation scheme based on a semantic classification of arguments (2022)

Abstract: Corpora of argumentative discourse are commonly analyzed in terms of argumentative units, consisting of claims and premises. Both argument detection and classification are complex discourse processing tasks. Our paper introduces a semantic classification of arguments that can help to facilitate argument detection. We report on our experiences with corpus annotations using a function-based classification of arguments and a procedure for operationalizing the scheme by using semantic templates.

## introduction the corpus-based analysis of argumentative texts is a widely used discourse processing task needed both for an in-depth understanding of this basic discourse type, and in the field of argument mining. we here present an annotation scheme that has been developed as part of a project for gaining detailed insight into the linguistic features of arguments. these features can be used for machine learning as well as for the task of argument detection in the study of discourse and discourse processing. in contrast to other approaches in the field, our method aims at the identification and classification of arguments, and not at the analysis of an overall argumentation structure (cf., for example, @xcite . we argue that the annotation scheme will facilitate the annotation process in many applications of argument detection, enabling both researchers and annotators to zoom into linguistic characteristics that pertain to a specific class of arguments rather than to the notion of 'argument' as a whole. the approach therefore reduces some of the vagueness of the category of 'argument' and adds to the transparency of annotators' decisions. arguments are used for different purposes, aiming to persuade an addressee to believe, evaluate, or do something (see e.g. eggs 2008; @xcite . we use this functional versatility of arguments as a starting point for our annotation scheme. more precisely, we propose a systematic testing procedure during which annotators use a set of linguistic templates on a given text passage to determine whether it is an argument or not, and, if so, which argumentative function it has. we are currently developing and evaluating this approach with a corpus of covid-19-related news opinion texts from the new york times. this paper is structured as follows. in section 2, we introduce the general idea of a function-based argument classification and briefly describe our corpus. in section 3, we present and evaluate our initial, rather ad hoc annotation efforts. in section 4, we introduce our function-based annotation scheme and report on our progress in terms of workflow and inter-annotator agreements. in section 5, we summarize our insights and provide an outlook. ## the initial annotation process our first set of annotations ('set 1') was created before the introduction of our annotation scheme. four annotators were instructed to apply a basic, simplified notion of 'argument,' consisting of a claim that is either supported or attacked. practical issues of claim detection and annotation (e.g. size of the discourse unit, treatment of quotes within the texts) were discussed at regular meetings, leading the group from an initial, very thorough exemplary discussion of three texts (subset 1-1, 3,653 words) to an annotation of another ten texts in one hit (subset 1-2, 11,646 words). annotations were created in the inception tool @xcite , hosted on a hhu server. as the annotation task is not only a coding but also a unitizing task, we measure the interannotator agreement using krippendorff's unitizing alpha @xcite . this measure works with an arbitrary number of annotators (where not all have to annotate all texts) and determines the degree of observed disagreement in relation to the expected disagreement (assuming random annotations). values range from -1 to 1, with values around zero representing random annotations, positive values representing more agreement among the annotators, and negative values representing more disagreement than expected by chance. the results for both subsets are displayed in while subset 1-1 showed promising inter-annotator agreement scores, subset 1-2 comes with disappointing scores. the good values for subset 1-1 are likely the result of the initial, intensive discussion between and with the annotators, producing biased annotations. comparing this to the weaker values for subset 1-2, it seems obvious that the annotators need more precise guidelines than what was provided in this second annotation round. this is supported by the fact that introducing a systematic annotation scheme has also been shown to improve inter-annotator agreement in previous projects involving argument annotation (see @xcite . therefore, our logical next step was to introduce such a scheme, as described in the next section. ## introducing an annotation scheme our updated annotation process is divided into three major steps (see the similar approaches in e.g. @xcite ): we here focus on step two, the identification of claims and premises. specifically, we describe the approach we apply to identify arguments by systematically categorizing them semantically. for further information on steps one and three see our annotation guidelines @xcite . in order for a pair of text passages to be included in our database as an argument, it must meet the following criteria: the first two criteria represent the standard definition of claim and premise (see above), while the third one guarantees that our resulting database has a homogeneous subject matter (in order to facilitate future experiments involving cross-topic transfer). the final criterion, which distinguishes our approach from other, existing ones, is the obligatory assignment of the claim to one of three semantic categories. in order to test a pair of text passages for these criteria, annotators insert them into linguistic templates (see @xcite in prep for details). for the final, semantic criterion, these templates take the form 'x, [___] y' as presented in the application of these templates is exemplified in (1). there, we see a claim (bold print) and a premise (underlined) from our corpus, inserted in the template which tests for a supported, positive, deontic claim (represented in by systematically applying such templates, our annotation process is now based on principled linguistic judgments rather than on ad hoc decisions. at the point of writing this paper, we have applied our annotation scheme to 12 texts (14,167 words), with promising results: annotators have reported that applying the provided patterns and being obliged to think about a given text passage in functional terms facilitates argument identification from the start. thus, by specifying the internal, semantic structure of the category claim thoroughly, its separation from premises as well as non-argument units becomes clearer. furthermore, discussions about the status of text passages as argumentative discourse units go more smoothly. these impressions are backed up by a clear trend toward increasing inter-annotator agreements, as illustrated in figure 1 . in set 2, annotators reached an agreement of up to a rounded 0.6 (as compared to 0.4 for set 1), with no negative values. however, this difference does not come out as significant, as is shown by an unpaired t-test comparing set 1 (@xmath0.208831, @xmath1.143289) and set 2 (@xmath2.315300, @xmath3.190852); t(23) = 1.5857, @xmath4.1265. the fact that we have not been able to support our intuition statistically is likely due to the small sample size and is currently being tested on more texts as the project progresses. as these examples, contrasting with (1) above, illustrate, the point of the semantic classification and of the corresponding paraphrases is to enable annotators in the early stage of argument detection to make informed, well-founded decisions. previous work with semantic types left the initial argument detection to experts and applied a semantic classification separately (see @xcite , while our approach aims at an improved identification of arguments, which then become available for thorough linguistic investigation. ## conclusion and outlook in this paper, we have sketched an annotation scheme which builds on a function-based classification of arguments. by systematically applying an array of linguistic templates to pairs of text passages, the annotation process is streamlined and facilitated. a first trend for improved inter-annotator agreements, however, has yet to be statistically confirmed. in the long run, we expect significant improvements in annotator recall as well as a less labor-intensive creation of a gold standard (i.e., the curation of the annotated texts by an expert linguist annotator). in order to further improve our results in terms of inter-annotator agreement and annotator recall, we are currently refining our work flow: for the third set of annotations, we have restricted our corpus to editorials, a more homogeneous subgenre of newspaper opinion pieces, and we are limiting text length to between 40 and 70 sentences in order to avoid too much variation in how the texts deal with argumentation in general. in addition, all annotators are actively involved in the text selection process, pre-assessing and potentially rejecting each text according to a growing catalogue of criteria (e.g. too anecdotal, too many direct quotes). in the future, apart from the methodological benefits of applying a semantically-grounded annotation scheme, ultimately we will also be able to investigate the semantic types per se. possible research questions are, for example, which linguistic features annotators and/or machines use to categorize arguments, and how our classification scheme relates to others (e.g. hidey et al. 2017 on interpretation, evaluation, and agreement/disagreement).