Annotating argumentation in S wedish social media (2020)

Abstract: This paper presents a small study of annotating argumentation in Swedish social media. Annotators were asked to annotate spans of argumentation in 9 threads from two discussion forums. At the post level, Cohen’s k and Krippendorff’s alpha 0.48 was achieved. When manually inspecting the annotations the annotators seemed to agree when conditions in the guidelines were explicitly met, but implicit argumentation and opinions, resulting in annotators having to interpret what’s missing in the text, caused disagreements.

## introduction in recent years, argumentation mining has grown into a central research topic within the field of computational linguistics. with the aim of automatically identifying and analyzing argumentation in text, its envisioned applications are many, from more effective document retrieval to learning aids @xcite . there are many different approaches to how argumentation can be modeled and annotated and there are now many data sets of different size and level of annotation, with domains ranging from legal documents to social media. however, in many of the existing data sets, inter-annotator agreement is not very high and it is because annotating argumentation turns out to be a quite challenging task. there is still a need of more annotated data, as well as investigating how to reliably annotate data. it is also important to investigate other languages than english. because of this, we have conducted a small annotation study on swedish social-media data where the focus has been on identifying instances of argumentation but not analyzing them further. 1 this is both to select documents for further analysis of the identified argumentation instances but also in order to investigate how reliably annotators can agree on what is argumentation or not. ## related work annotating with the aim to distinguish what is argumentative from what is not argumentative has not been the most common goal in argumentation mining, although it is necessarily part of studies that annotate components of argumentation, either implicitly or explicitly as a first step in an argumentation mining pipeline. when it comes to documents from the web, the annotation of argumentation is usually done with respect to a topic. for example @xcite , annotated comments and blog posts as argumentative with respect to a topic in order to select documents for further annotation. on this they reach a 0.51 fleiss κ and 0.59 cohen's κ. similarly, @xcite annotated documents from web discourse as 'non-persuasive' and 'on topic persuasive' before moving on to annotate microstructure. they reached fleiss κ of 0.59 on this task. in some studies presence of argumentation has been annotated together with the stance or the type of the argumentation. for example, @xcite annotated sentences from the web for supporting or opposing argument, or not an argument with respect to a topic. they reached cohen's κ 0.72 and an observed agreement between 0.86-0.84. more recently, @xcite annotated sentences from the web with both expert and crowd-sourced annotators. the sentences were annotated with argument spans, and the spans were marked with stance with respect to a topic. the reached 0.6 krippendorff's α u and the crowd-sourced annotators reached 0.71 α u . ## data the data in this study is from two of sweden's largest online discussion forums, familjeliv ("family life" fm), and flashback (fb). familjeliv is generally considered to be more about relationships and family life and flashback more about politics, although both forums cover a broad range of topics. both forums have a simple thread structure, where a thread is started with a post by a user and then other users reply with subsequent posts, shown in chronological order. there is a possibility for the users to cite each other, but there is no visually explicit tree structure as for example on reddit. for this study, nine threads were randomly chosen among the threads which had a length of about 30 posts. these threads are shown in ## conclusions & future directions iaa values such as the ones reported here are not uncommon in argumentation mining tasks. still, both the cohen's κ of 0.48 and the krippendorff's α 0.48 are lower than the previously reported studies, (for example 0.59 cohen's κ in @xcite or 0.71 krippendorff's α u in trautmann et al. ( 2020 )). however, as opposed to those studies, the annotators were not asked to annotate with respect to a topic, so the results are not fully comparable. annotating only 9 threads might have affected the iaa, especially since the iaa varied between the threads. when manually inspecting the annotations, it seemed as when the conditions asked for in the guidelines were very explicitly met, annotators agreed. when the argumentation (or not argumentation) was more implicit the annotators disagreed. this is something which has to be considered when further developing the guidelines. another thing to consider when annotating complex phenomena such as argumentation is that even though the annotators disagree, it might not be the case that one is right and the other is wrong. as shown in for example @xcite there are cases where two different annotations could both be considered correct. if one allows for several annotations to be correct, this would need to be reflected in both the guidelines and evaluation. in the future we plan to test the guidelines in a domain where one can assume that people are more explicit with their argumentation, such as newspapers. we also plan to extend the guidelines to annotate components of argumentation to see how this affects the annotation.