Annotating Attitude in S wedish Political Tweets (2025)

Abstract: There is a lack of Swedish datasets annotated for emotional and argumentative language. This work therefore presents an annotation procedure and a dataset of Swedish political tweets. The tweets are annotated for positive and negative attitude. Challenges with this type of annotation is identified and described. The evaluation shows that the annotators do not agree on where to annotate spans, but that they agree on labels. This is demonstrated with a new implementation of the agreement coefficient Krippendorff’s unitized alpha.

## introduction automatic computational analysis of emotional and argumentative language (sentiment, attitude, emotion, argumentation, etc.) has progressed considerably over recent years, but annotated datasets are still lacking for all but a few languages. at the same time, such datasets are necessary at least as evaluation data, for instance for evaluating approaches that attempt to alleviate the lack of language-specific training data by involving machine translation or multilingual llms. in addition to the fact that careful annotation of large volumes of text for emotion and argumentation is labor-and time-consuming, like most nlp annotation tasks, it is clear from the literature that this particular annotation task is inherently difficult, due to its complexity and subjectivity (see for example @xcite ). here, we present a new dataset consisting of swedish political tweets manually annotated for positive and negative attitude, more specifically what the attitude is towards. we have chosen to call the annotation in this dataset for attitude, but it could also be called stance, or argumentation, as these concepts often overlap. we also address some of the complexities encountered in assessing the quality of the annotations, in particular how to calculate inter-annotator agreement in a reasonable way. for this purpose, we have reimplemented krippendorff's unitized alpha measure @xcite in python, thereby hopefully making it more accessible to the nlp community. ## related work previous work on emotional and argumentive language in swedish are few, and work focusing on annotation of these concepts are fewer. there are however exceptions. for example, some work has focused on sentiment, such as creating a sentiment lexicon @xcite . there are also works describing argumentation annotation. most similar to the task presented here is the aspect based sentiment analysis (absa) corpus @xcite . the corpus consists of editorials, opinion pieces and posts from online forum annotated with sentiments and the aspect of the sentiment (source, target and expression). the agreement was reported as krippendorff's α of 0.34 for documents and 0.44 for paragraphs. beyond the swedish language there are others who have presented similar annotation tasks. for example, @xcite present a dataset of 3883 tweets annotated for argumentation, where a tweet containing an opinion is considered argumentative. the tweets were selected among current popular discussion topics, such as politics. they reach a krippendorff's α 0.74 on a subset of the dataset. another similar task is presented in trautmann (2020), where aspects (defined as "the main point the argument is adressing") are added to previously annotated spans. these spans were annotated for expressing negative or positive stance or argumentation on a topic. instead of asking the annotator to annotate freely, they were 106 shown a set of candidates and asked to choose the appropriate one. the agreement was 0.87 cohen's κ. @xcite also present a corpus of tweets, which consist of 1200 german tweets related to climate change. while the unit of annotation is the same as here, spans, their annotation scheme differs. they annotate different kinds of claims and evidence as well as sarcasm and toxic language. for these categories they reach between 0.41-0.83 krippendorff's α. an analysis of the agreement of the annotations of this dataset was previously presented in lindahl (2024), which discusses disagreement in argumentation annotation. compared to this, in this paper we present the dataset, the annotation procedure and add additional analysis. ## data the tweets in this dataset were collected from the period between february 2018 to september 2022. this period roughly represents the time period (term of office) between two swedish general elections, held in september 2018 and 2022. the tweets were taken from the official accounts of the political parties represented in the swedish parliament as well as from the official accounts of the political party leaders at the time and the official account of the prime minister , in total 19 users 1 . only original tweets were collected, not retweets. from this collection, around 4500 tweets were randomly selected, see ## annotation the annotation was carried out by four annotators with linguistic background. before the main annotation started, a test round was carried out were 1 not all parties or party leaders had an official account. for each tweet there was also the option to ignore the tweet (if there was something wrong with the tweet) or to flag it as "very difficult to annotate". during the test round, the annotators were also asked to write a comment about the tweets that were difficult to annotate and why. a meeting was also held with the annotator between the test and main round in order to discuss difficult examples. after the feedback from the test round the guidelines were updated, see the next section. ## annotation evaluation as previously mentioned, a thorough analysis of the agreement and disagreement in this dataset was done in @xcite . it is reported that even though agreement is low, there are cases in which the annotators partly agree. there are also cases where multiple interpretations are possible. here we will summarize some of the agreement and add new, additional analysis. a new example of a how a tweet has been annotated by three of the annotators is seen below, bold is again negative and italics is positive. a. the elderly should not have to suffer due to understaffing. female-dominated professions must be revalued and appreciated so that more people want to stay in their jobs -it's about the care of our loved ones! b. the elderly should not have to suffer due to understaffing. female-dominated professions must be revalued and appreciated so that more people want to stay in their jobs -it's about the care of our loved ones! c. the elderly should not have to suffer due to understaffing. female-dominated professions must be revalued and appreciated so that more people want to stay in their jobs -it's about the care of our loved ones! we can see that the annotators both agree and don't agree. they all agree that understaffing is negative, but they disagree on how much of the context should be included. annotator a has also included a span which the others have not marked. this is in line with the reported difficulties about determining what to annotate. ## discussion & summary in this paper a dataset of annotated political tweets, with the accompanying annotation procedure, was presented. the agreement (normal krippendorff's α) for our dataset was similar to the ones reported in @xcite , but lower than that in @xcite or @xcite . during the annotation process, based on the annotators feedback, we identified several challenges in annotating attitudes. the most prominent one was what to consider an attitude. due to ambiguity, implicitly and sometimes phrasing, the annotators reported difficulties determining what to include. while we tried to solve this by only annotation explicit attitudes, it remained a problem. by using our new implementation of unitized alpha ( u α), we can confirm this problem. the annotators differ in where they have annotated the spans, resulting in general u α of 0.34. however, at the places where they have annotated the same spans, the agreement ( cu α) is 0.87. this highlights the need to not only report one agreement number, but to look at annotations from several angles. a future annotation task of this kind could probably benefit from annotation predefined spans, or annotating in several steps, as in for example trautmann (2020). another factor to consider in this, previously shown by @xcite , is that there can be several possible interpretations, naturally leading to lower agreement.