Annotating Topics, Stance, Argumentativeness and Claims in D utch Social Media Comments: A Pilot Study (2020)

Abstract: One of the major challenges currently facing the field of argumentation mining is the lack of consensus on how to analyse argumentative user-generated texts such as online comments. The theoretical motivations underlying the annotation guidelines used to generate labelled corpora rarely include motivation for the use of a particular theoretical basis. This pilot study reports on the annotation of a corpus of 100 Dutch user comments made in response to politically-themed news articles on Facebook. The annotation covers topic and aspect labelling, stance labelling, argumentativeness detection and claim identification. Our IAA study reports substantial agreement scores for argumentativeness detection (0.76 Fleiss’ kappa) and moderate agreement for claim labelling (0.45 Fleiss’ kappa). We provide a clear justification of the theories and definitions underlying the design of our guidelines. Our analysis of the annotations signal the importance of adjusting our guidelines to include allowances for missing context information and defining the concept of argumentativeness in connection with stance. Our annotated corpus and associated guidelines are made publicly available.

## introduction user-generated content (ugc) such as can be found in the comment sections of newspapers and social media sites is a valuable resource for the collection of argumentative texts written in natural language. according to @xcite , user comments offer a "substantial amount of factual information, and [demonstrate] a public process of weighing alternatives via the expression of issue positions and supporting rationales". the field of argumentation mining, which forms a part of natural language processing (nlp) research, uses this type of data as a resource to train and test automatic detection systems for the purpose of extracting the various components making up the argumentation expressed by the users @xcite . training the systems requires annotating the data, for example labelling claims and reasons for those claims in the text. the various annotation tasks required for producing such data have proven to be very difficult for human annotators. defining a good set of annotation guidelines is essential towards advancing the field of argumentation mining on ugc data such as social media comments. currently, the myriad of theoretical perspectives on how to analyse argumentation as well as the unpredictable nature of ugc data have lead to a lack of current consensus on reliable guidelines for the various argumentation annotation tasks. this paper presents a pilot annotation study for the identification of the topics, topic aspects and stance expressed by the comments, as well as the detection of argumentativeness and the main claim or conclusion presented. this study assesses the suitability of our current guidelines by measuring the inter annotator agreement (iaa) for all tasks and analyses some specific cases which proved most challenging to our annotators. our aim is to adjust the guidelines based on these results and analysis @xcite , which will then serve as the basis for an extensive annotation study on a more substantial corpus and including more annotation tasks required for a full analysis of the argumentation presented in the comments (a.o., this will include premise and argumentative relation annotation). in section 2, we briefly discuss some of the relevant research. in section 3, we give an overview of the theoretical frameworks which form the basis of our annotation scheme. in section 4, we describe our pilot corpus and in section 5 we give an overview of the annotation procedure, as well as more information on the rationale underlying our guidelines. in section 6, we first present the results of the iaa study. in section 7, we then present our analysis of the annotations. we end with section 8 on concluding remarks as well as indications for future research. ## related research in the field of argumentation mining, many different problems related to the analysis of argumentation in texts are being treated as different subtasks for automatic detection. many of these tasks relate more generally to the processing of various aspects of texts within the broader field of nlp. for example, as a preliminary step towards the more argumentation-specific detection tasks, the tasks of topic and stance detection are often performed. users, especially when arguing on controversial topics, tend to emphasize specific aspects of the topic, a concept called "framing" @xcite . therefore, both the more general topics and the more fine-grained topic aspects need to be identified. a major challenge still lies in determining how the different aspects relate to each other and to the major topic under discussion @xcite . this challenge relates to the issue of determining how fine-grained the targets of the users' stance needs to be. there is still no consensus on this issue, but the findings have confirmed that the more fine-grained the stance target, the more difficult it is to automatically classify the stance, and stance targets that are defined too broadly may not be specific enough to become associated with each respective side (pro/con) of the debate target @xcite . most authors therefore opt for a predetermined list of topics, in which topics either take the form of single words (more coarse-grained approach) or phrases (fine-grained). the latter approach was used by saint-dizier (2016), who defined controversial issues as evaluative statements (e.g., "climate action is necessary") as this would aid in mining pro/con arguments for these specific controversies. as a preliminary step for the annotation of different argument components @xcite , the text needs to be split into smaller segments. this step is often skipped in argumentation mining research, in favour of departing from pre-segmented text @xcite . the segmentation of user-generated comments is not straightforward, as they contain many irregularities in the use of punctuation, capitalization and other orthographic markers of sentence boundaries. determining the boundaries of argumentative segments is challenging but necessary for argumentation mining, as it forms the preliminary step both towards identifying the claim and premises as well as more fine-grained components of the argument. textual indicators of argumentativeness such as discourse markers are often used as features for the automatic detection of argumentative segments @xcite . an important caveat of using argumentative words or phrases for the task of argumentativeness detection is that they may also occur in non-argumentative text @xcite . argumentation mining research has produced some work on the linguistic cues which may be helpful for the identification of argumentativeness @xcite , especially the detection of discourse markers has been explored to serve as signallers of argumentativeness to aid the automatic detection of arguments @xcite @xcite . the detection of segments representing the claim or conclusion of an argument in texts is an important prerequisite for applications involving fact checking @xcite . it is a very challenging task, especially when applied to an extremely varied resource, like online discourse @xcite . there is no current consensus on what exactly constitutes a claim, leading to many different annotation approaches @xcite . ## theoretical frameworks for the annotation of topics in comments, it is difficult to predetermine a list of possible topics. the comments in our corpus were made in response to newspaper articles shared via facebook, so we can safely assume they will often refer to the newspaper content @xcite . in order to capture all the available topic information, the distinction between structuring and interactional topics is useful. the structuring topics are those found in the surrounding context; the interactional topics are those which form the topics of discussion and are found in the immediate context of interaction @xcite . this distinction has been applied by @xcite , who used the two categories for the topic annotation of user comments to newspaper articles shared via facebook and comments made on the same articles, via the official newspaper website. in their study, the structuring topic is that which is reported on in the article and the interactional topics are present in the individual comments. this approach seems feasible for our corpus, since the data type is so similar. for some interactional topics and aspects in the comment, the user expresses a stance reflecting his personal opinion towards the specific topic or aspect. not all topics and aspects touched upon in the comment will be the target of the users' opinion, as some will only be mentioned in passing, or to set up the context for the argumentation. the typical stance annotation labels include pro, contra and none, where the last one represents a topic or aspect which is not used as a target for the opinion of the user (küc ¸ük and can, 2020). parallel to the distinction between the broader (discussion-wide) structuring topic label and the narrower interactional topic label (based on the comment text itself), the distinction between a debate stance and explicit stance is made by @xcite . the debate stance is the stance towards the target of the whole debate. it is often implicit and inferrable from the explicit stance(s) which rely on textual evidence. this distinction has proven successful for stance annotation on noisy social media data and has even helped model implicit argumentation @xcite . given the difficulty of the automatic segmentation of argumentative texts @xcite and our focus on the segment-level annotation of argumentativeness and claims, we also perform a preliminary manual segmentation step before handing the data to the annotators to avoid the error percolation which an added segmentation annotation task would inevitably introduce. while discourse markers are considered useful indicators of argumentativeness @xcite , @xcite proposed the more specific concept of the argumentative marker which is a lexical item signalling the presence of an argumentative move (e.g., marking a standpoint). the argumentative marker may consist of a single word, phrase or sentence. a distinction is made between those markers which are syntactically part of the main constituents of the phrase and those that are independent on a semantic and syntactic level @xcite . the concept of shell language developed by @xcite is similar to the argumentative markers, in the sense that they may be used as signallers of argumentativeness and may consist of longer sequences of words than is the case for discourse markers. shell language includes organizational phrases, such as ones marking the expression of an opinion (e.g., "i think that"), but also ones marking argumentative structures (such as "after all"). @xcite have used shell language for the task of automatically separating topical contents from organizational language and have demonstrated the usefulness of this task, for instance to improve topic detection. they evaluated their fully unsupervised shell topic model on argumentative ugc sourced from online forums. @xcite contends that argumentativeness is present in an utterance even if it does not contain a linguistic expression it may explicitly be linked to. the first major challenge lies in defining what exactly constitutes argumentativeness. the distinction between the argumentative and informative components of utterances @xcite provides one possible answer. the informative component corresponds to the propositional content of the sentence. the argumentative component signals the utterance's argumentative orientation: whether or not it has the potential of being used as a premise for a given conclusion. this perspective helps to circumvent the difficulty of reconstructing the intentions of the author of an argumentative text, as it emphasizes the text itself as the locus of the author's intention. @xcite 's concept of the claim consists of any utterance conveying subjective information and anticipating the question "why are you telling me that?". @xcite have criticized @xcite annotated dataset of online comments from blog threads for containing noisy sentences annotated with claims. however, we hypothesize this is not necessarily caused by the definition of the claim concept. instead, it is a characteristic of this type of data. our annotation guidelines employ a very similar definition of the claim concept as the one used in @xcite . the current pilot study represents the first phase of our aim to define the annotation guidelines on a corpus of 100 dutch user-generated comments sourced from the social media platform facebook. all comments were made in response to an online newspaper article being shared via the official facebook page of a flemish newspaper. in this first phase, we focus on measuring agreement and finding edge cases for the annotation tasks of topic and stance labelling, segmentation of the text into argumentative units and claim identification. the second phase will consist of an agreement study on the same corpus sample of 100 comments and will focus on the tasks of premise and argumentative relation identification. ## corpus facebook is by far the most popular platform for accessing news @xcite . however, given the difficulty in collecting this data automatically and assumptions about the lack of argumentation in shorter texts @xcite , this platform is rarely used to source data for argumentation mining purposes. our corpus of 100 dutch comments was sourced from the official facebook page of het laatste nieuws (hln), a popular flemish newspaper. as is the case for many news outlets @xcite , the facebook page is used to share news articles with a wider audience, often linking to the official website of the newspaper. the corpus was collected manually and contains comments made on posts published in the second and third weeks of june 2020. we did not predetermine a list of topics to filter the facebook posts. instead, we chose to collect the first ten comments of each most recent post on the page which was topically related to a political or controversial topic. this included news stories related to policy decisions, party politics, but also related to topics like health care, poverty and racism. we filtered out duplicate comments and comments in other languages than dutch, but did not filter out multiple comments made by the same user. aside from the 100 comments, we also collected the 30 associated article texts and gathered 30 screenshots of the facebook posts commented upon. ## annotation six annotators participated in this study. all of them were dutch native speakers and linguists who received the same annotation guidelines @xcite to follow foot_0 . they each annotated a total of 100 dutch user comments, all 100 were the same set for each annotator and were used to measure inter-annotator agreement. we performed two preliminary steps to prepare the corpus for the annotators. first, to prepare for the topic annotations we annotated the structuring topics and aspects contained in the facebook posts and the news article texts. for the topic annotations of the facebook posts, we labelled the topic information as present on screenshots of the post in question, thus including information like the title of the article, the accompanying image and description text in our decision making. for the topic annotations of the article texts, we limited ourselves to the title and the first three paragraphs of each article. the second preparatory step was to manually segment all the comments in order to prepare for the argumentativeness and claim annotation tasks, which were to be performed on a segment-level. we segmented the comments based on the shell language expressions we found @xcite , but only if they were also set apart from other segments in a syntactical way @xcite . if the expression occurred syntactically embedded in the phrase, we considered it a part of the larger segment. this approach allowed us to distinguish organizational segments from content segments. we did not consider markers of opinion (such as "i don't think that") as part of shell language, since we only wanted to focus on separating organizational phrases marking argumentative structures from content segments, leaving markers of subjectivity to remain a part of the content segments. thus, our segmentation approach is more coarse-grained. other researchers like @xcite have used shell-like concepts for the task of detecting argument components in persuasive essays, also allowing shell language (in @xcite terms "argument words") to occur in the argument content (in @xcite terms "domain words"). for the segmentation of comments containing rhetorical questions, we decided to follow the perspective of speech act theory, which contends that this special type of questions only has the surface structure of a question, but realizes the speech act of making a statement @xcite . thus, every rhetorical question which was followed by an explicit answer was considered as one segment together with that answer. our approach resulted in a total number of 504 segments across the whole corpus. the highest number of segments for a single comment is 19; the lowest is 1. the comments in our corpus contained an average number of 5 segments. we divided our annotators into two groups of three. each group was assigned different annotation tasks: we provide an example of an annotated comment from our corpus to help clarify the annotation tasks. we refer the reader to the guidelines for a more detailed description of the labels. for the comment "and in the stores everyone touches all the fruit and vegetables with their hands better to monitor everything that is in the store, people even open packages and eat the product", the topic-aspect combinations corona (measures, spread of disease), enforcement and shops were labelled by one of our annotators. three stance labels were identified: pro towards the topic of enforcement and the aspect of measures and contra towards the aspect of spread of disease. the segmented comment was shown to the second group of annotators as follows: "1[and] 2[in the stores everyone touches all the fruit and vegetables with their hands] 3[better to monitor everything that is in the store,] 4[people even open packages and eat the product]". in a first round, the annotator determined for each segment whether it was argumentatively relevant or not, resulting in the following labels: argumentative (segments 2, 3 and 4); non-argumentative (segment 1). in a second round, the annotator was given the segments prelabelled for argumentativeness. the annotator then had to determine which of the segments labelled as argumentative best represented the central claim/conclusion of the user comment. in this example, segment 3 was labelled as the claim. all other segments marked argumentative in this comment are therefore seen to support or to be otherwise argumentatively relevant leading up to this claim. ## inter-annotator agreement results for the first annotation tasks, viz. assigning topics, aspects and stance, the annotators could assign multiple labels to the same comment. as standard iaa scoring mechanisms (such as cohen's pairwise kappa) assume the assignment of one category label per unit of annotation, these metrics are not suitable for measuring iaa for multiple labels per annotator. therefore other metrics, such as krippendorff's alpha @xcite , have to be used to calculate disagreement (or distance) between sets of assigned labels. krippendorff's alpha considers difference in annotation on all possible annotation units, irrespective of the number of labels and the type of annotation (categorical, numeric, ordinal). to calculate the distance between two sets of annotation labels, we followed the implementation of @xcite , using masi (measuring agreement on set-valued items). the resulting distance is 0 when sets are overlapping, and 1 when sets are disjoint. the first annotation tasks appeared to be challenging, resulting in a total of 88 different topics, 245 aspects and 197 stance labels. as a lot of the disagreement for stance is caused by the choice of different aspects of the same topic, we also calculated the distance when only considering the sets of stance labels disregarding the specific topic or aspect they were attached to. ## analysis we reached moderate agreement for the topic and aspect labelling tasks, considering the krippendorff's distance is rather high between all pairs of annotators. this is partly due to errors percolating from the topic labelling step to the aspect labels. when we consider only the stance labels per comment, without taking into account the topic and aspects they are linked to, we see that on average, almost half the comments show perfect agreement. in addition, annotators sometimes assign similar (sub)topics or additional (sub)topics, resulting in fairly large distance scores: e.g. @xcite vs @xcite results in a distance of 0.55, and [police, politician] vs [politician] results in a distance of 0.665. perfect agreement on the stance labelling task was reached for only 8 comments, all three annotators agreeing on the "none" stance label for those comments. when considering partial agreement (defined here with the following condition satisfied: all three annotators share at least one stance label towards the same topic, e.g. "(politics)contra"), we noted partial agreement for 19 comments. in the following, we briefly discuss some concrete annotation examples for the annotation tasks which were performed on the comment level. when considering the distance metric between the stance labels only, we noted some recurring labelling errors. in some cases, the annotator identified the stance towards a specific topic, while the others identified only the stance towards the broader related topic. in example 1, annotator 1 identified a contra stance towards the mouth mask obligation topic, while annotator 2 identified only a pro stance towards the more general topic of corona measures. annotator 3 did identify both stances. one possible explanation for this confusion is that the annotators may have made a distinction between main topics and more peripheral topics in the comment. in future guidelines, it will be necessary to emphasize the need for identifying both the stance towards the broader topics and more specific ones. comments containing irony such as example 2 seem to complicate the stance annotation in some cases (perfect agreement for stance labelling amounting only to 8% of the total of 25 ironic comments and partial agreement reaching 28% (7/25)). two of the annotators indicated doubts in cases such as these and were subsequently instructed to use the none label in case of doubt. this instruction will also be added to the future guidelines. additionally, we will ask the annotators to mark the instances of irony and context-related understandability issues, a strategy also applied by @xcite in their stance annotation study. considering the difficulty of the annotation task, we reached substantial agreement scores for the task of argumentativeness detection (0.76 fleiss' kappa). for the argumentativeness labelling task, nonargumentative segments were defined as those having primarily an organizational function in the comment (for example, to introduce a particular component of the argumentation). two annotators expressed their doubts as to how to annotate segments which in addition to the organizational function, also seemed to express the stance of the user, such as segments 2 and 6 in example 4. to avoid mislabelling of such segments which are stance-bearing (and should thus be labelled as au), we will add this specification to the guidelines. segments occurring at the very start of the comment caused disagreement for argumentativeness labelling, especially in cases like example 5 where the segment appeared to interact with the surrounding context (for instance, the title or the content of the newspaper article). since the information in these segments often consists of an evaluative comment on the article content, they are considered stance-bearing and should be labelled as au. we found a total of 45 comments in our corpus in which the first segment interacted with the context. in 77.8% of these cases (35/45 comments), perfect agreement was reached across all three annotators for the argumentativeness labelling task. when we compare this to the 52% of comments reaching perfect agreement for this task on the whole corpus, we see that the influence of the lack of context information available to our annotators was moderate for this task on our current corpus. we will ensure such segments are flagged in future annotations. this will help us to determine which types of comments may require context features for the automatic detection system. additionally, it will be useful to ensure our future guidelines are more explicit on the possible guises of argumentativeness (defined in our guidelines as "all information relevant to the support or expression of the author's position"). therefore, we will explicitly state that this then includes background information (e.g., "in 1400, a virus was considered an illness") and (parts of) personal narratives ("just the other day i saw a whole family with no mouth masks on"). in our corpus, we found a total of 44 comments contained at least one of these less obviously argumentative segments (corresponding to 19% or 98/504 total segments in our corpus). since the annotators themselves raised the question of whether or not such segments were to be considered argumentative fairly quickly, we were able to achieve perfect agreement across all three annotators for 80% (79/98) of segments. however, perfect agreement was only reached for 43% (19/44) of all comments containing this type of segment, meaning that there were few comments in which they were all captured. therefore, we will include more examples of what sequences of less obviously argumentative segments may look like (such as the first segment of example (6) in the guidelines). in example (6), while both segments are argumentative (since together they form evidence for the user's claim that the experts are distributing confusing information about the usefulness of mouth masks), we found disagreement on the argumentative relevance of the first segment. maar (english: but) (29 instances) and en (english: and) (23 instances) were the most frequently occurring shell language expressions and proved indicative of argumentativeness (but preceded an au segment for 23/29 instances and and preceded an au segment for 15/23 instances for all annotators). this is not surprising, since maar (english: but) is a connective often used to signal a contrasting reason in argumentation and en (english: and) can be used to signal an additional reason. an important type of shell language expression we found in our corpus (i shouldn't be saying this; there is not much more to say about this) corresponds to the so-called "discourse markers of standpoint continuity", identified by @xcite as a common characteristic of group discussions on controversial issues. they are commonly used to specify argumentative standpoints as well as to avoid disagreement while saving face @xcite . however, a clear distinction should be introduced in the guidelines between such markers and segments like i do understand his point or i get that, which are indicative of the stance of the author and should be annotated as au (all three annotators currently annotated these segments as nau). the presence of the verbs of understanding as well as the first person singular may help distinguish such segments. we reached moderate agreement for the claim labelling task (0.45 fleiss' kappa). for 33 comments full agreement was reached on the claim annotation task. many of the claims identified are elliptical, e.g.: belgium doomed, mayor not capable. this is typical of the nature of our data. from @xcite analysis of claim segments in various argumentation mining corpora, the presence of policy claims @xcite emerged as a common characteristic in multiple corpora (e.g., of the wikipedia talk page corpus @xcite and the microtext corpus of @xcite ). in our corpus, such policy claims, which are often characterized by the presence of the modal verb "should" @xcite , were present in 32 comments, for example: everyone should just decide for themselves what they find important, taking care of yourself or blending in with the crowd. since the policy claim expresses a wish for things to be done differently, it may be expressed in the form of an advice (we can stop driveling about the mouth masks now), or as a strong imperative (if your heart isn't in it, don't do it), the latter of which seems to be particularly indicative of claim segments. perfect agreement for the claim labeling task was reached on 44% of comments (14/32) (corresponding to 22 segments containing at least one segment with "should" or an imperative form. aside from the occurrence of policy claims, when comparing the claims annotated in our corpus to those found in the web discourse corpus @xcite , our claims are more often anaphoric in the sense that they express the stance of the author, but without specific lexical reference to the given topic ("i personally find it really sad it's not obligatory,"). many claims contain expressions signalling beliefs (such as "in my opinion", "personally", "i find"), which according to @xcite analysis, is characteristic of the claims found in persuasive student essays (such as the corpus of @xcite ). in general, the claim of an argument is more likely to carry stance-taking words toward the topic. this aspect has been identifed as a useful feature for the automatic detection of claims @xcite . the fact that we can find many correspondences with existing corpora of different genres and domains, strengthens our assumption that general markers of claim presence may still be found across genres. adding these markers in the guidelines for the annotation can help create a more unified approach towards the annotation of claims, which appeared to be absent in the field @xcite . we will investigate the presence and usefulness of such general markers of argumentativity in our more extended corpus, including more domains and platforms to source ugc data from. ## conclusion and future research the insights we gained from this pilot annotation study will be used to improve our annotation guidelines. in this study, we pre-segmented all the comments in a preliminary step. this was done to avoid too much error percolation from the annotation results of the segmentation into the claim annotation task, which would make calculating the agreement on claim identification a difficult matter. however, we realize this pre-segmentation may introduce more bias when considering the annotation of other argumentative components like premises and relations between segments. since the automatic segmentation of texts into argumentative and non-argumentative segments is still very challenging and advances in the field are still being developed, we will use @xcite rule-based algorithm for the automatic pre-segmentation of the corpus as a step prior to the manual annotation. in this way, the annotators will be asked to correct the automatic segmentation by merging incorrectly split segments. we believe the low agreement results for the first set of annotation tasks (topic, aspect and stance detection) may be improved by reducing the number of possible structuring topic labels for the annotators to choose from. this will require pruning the list of structuring topics and aspects we identified to include only the most frequently occurring ones. since we are aware of the risk of bias entering our annotation process in providing the structuring topic and aspect labels for the annotators to choose from when deciding on the interactional labels, we will include an evaluation step (removing duplicate labels and becoming aware of ambiguities in certain labels) in our revised annotation guidelines to be performed by a separate annotator. from our analysis of the results for the argumentativeness annotation task, we conclude that our guidelines need more incorporation of stance expressions as an important indicator of argumentativeness. since most of our annotators indicated that they had trouble annotating comments due to missing context, we want to explore the impact of context on the various annotation tasks we have performed in this study. first of all, our new guidelines will have to supply more examples of information that is considered argumentatively relevant, e.g., background or context setting information. in particular, we want to investigate whether there are "triggering devices" which are used to evoke context @xcite in the comment. since the function of context is often defined as narrowing down the range of possible understandings of a text or utterance @xcite , we are interested in studying how supplying the annotators with various degrees of context information will for instance impact their understanding of the argumentativeness and the claim of a comment.