Annotation and Detection of Arguments in Tweets (2020)

Abstract: Notwithstanding the increasing role Twitter plays in modern political and social discourse, resources built for conducting argument mining on tweets remain limited. In this paper, we present a new corpus of German tweets annotated for argument components. To the best of our knowledge, this is the first corpus containing not only annotated full tweets but also argumentative spans within tweets. We further report first promising results using supervised classification (F1: 0.82) and sequence labeling (F1: 0.72) approaches.

## introduction in recent years the field of argument mining, which focuses on the automatic identification of argument components and their relations in text, has developed substantially @xcite . however, while the majority of research concentrates on well-structured documents @xcite , less work has been done on user-generated web content @xcite . this shortcoming poses a problem as systems trained on formal and edited texts tend to be inapt of extracting patterns from the more informal user-generated content @xcite . in this paper we focus on tweets, which are of great interest for the argument mining community due to the increasing use of the microblogging service twitter 1 in political online discourse. while some first work on argument mining in tweets exists @xcite , only a small number of available annotated corpora have been created that can be utilized for training tweet-specific argument mining systems @xcite . to improve on this point, we present a new corpus of german tweets annotated for claim and evidence 2 . to the best of our knowledge, this is the first argument tweet corpus not exclusively annotated with the full tweet as the unit of annotation. instead, argumentative spans within tweets, henceforth called argumentative discourse units (adu) @xcite , have been annotated as well. they render the corpus suitable not only for supervised classification but also for sequence labeling approaches. we also present first promising experimental results using this corpus. this paper is structured as follows: section 2 gives a short overview of the relevant social media and twitter-related literature on argument mining. section 3 describes the corpus, the annotation scheme and the annotation procedure. in section 4 we present first classification and sequence labeling results using the annotated data. section 5 discusses our results and gives a brief outlook. ## corpus annotation our complete initial corpus consists of 77,100 tweets collected in 2019 via the twitter api using the python library tweepy foot_0 . all tweets contain the keyword klima ("climate") and mainly concentrate on the topic of climate change, which was intensely discussed by german media and politics during that time. we conducted the following preprocessing steps. first, we removed all retweets and excluded non-german tweets using the language identification tool langid @xcite . these steps led to a subset of 29,525 tweets. in the following, we grouped the tweets into pairs, consisting of a tweet, henceforth called context tweet, and the tweet to be annotated, which is a reply to the context tweet and, for this reason, is called reply tweet. this approach is motivated by the assumption that tweets in a reply relation are more likely to contain argumentation @xcite . moreover, given the short nature of tweets, providing a context is supposed to help interpreting the reply tweet's content. all tweets that were no replies were removed and missing context tweets were collected in an additional step. finally, we removed all @-mentions at the beginning of a tweet, as these mainly point to the tweet's recipients. the final corpus consists of 12,296 context and reply tweet pairs. for the present study, a subset of 300 tweet pairs was annotated. foot_1 ## experiments and results in this section, we present first experimental results based on the annotated corpus. we apply two different approaches: for the tweet-level annotations we trained supervised classification models. this is comparable to the prior studies of @xcite and @xcite . in addition, we use the adu-level annotations for running a sequence labeling approach similar to @xcite . we experimented with different combinations of feature sets, preprocessing steps and models. however, we only present the best results here. tweet level. classification models were trained on different combinations of n-grams and on pretrained bert-based document embeddings @xcite . the latter were created using flair, an nlp framework that contains a unified interface for employing different types of text embeddings @xcite . all shown classification results are yielded using extreme gradient boosting (xgboost) @xcite , which is a variant of the gradient boosting approach introduced by @xcite . we implemented three different classification tasks based on the respective binary target sets: argumentative vs non-argumentative, claim vs no claim or evidence vs no evidence. all results are 10-fold cross-validated. adu level. sequence labeling models were trained on the following features: 1) unigrams, 2) a combination of linguistic (e.g., n-grams, pos tags) and twitter-related (e.g., hashtags, @-mentions) features, 3) pretrained bert-based word embeddings, which were again created using flair. we chose a conditional random fields approach @xcite , using the sklearn-crfsuite 5 . again, all results are from 10-fold cross-validation. in the sequence labeling approach bert-based models perform best for all three labeling tasks. using a set of linguistic and twitter-related features improves the f1 scores compared to the simple unigram models in the argument (0.7 vs 0.69) and claim (0.56 vs 0.53) tasks. however, no improvement is achieved in the evidence task. interestingly, scores are highest for the evidence task whereas the results for the claim task are considerably lower. this pattern contrasts with the tweet-level classification results. ## discussion and outlook in this paper we presented a new corpus of german tweets annotated for claim and evidence. while a few previous studies on tweet corpus creation for argument mining exist @xcite , to the best of our knowledge our corpus is the first tweet dataset with adu annotations. it is also the first german tweet dataset generally annotated for argumentation. although we showed that due to the subtlety of the task relatively low iaa scores were achieved, classification and sequence labeling results based on the dataset are promising. classifying argument and claim components led to robust f1 scores around 0.8. solely evidence units posed somewhat of a challenge for the classifier. however, sequence labeling models performed best for evidence units. with both approaches we surpassed the results presented by @xcite . given that the iaa scores for evidence annotations were reduced as well, we conclude that evidence units pose an especially hard problem to solve. recalling our definitions of claim and evidence, this seems intuitive. as evidence units are only defined with respect to claims, a decision has to be made about the exact boundary between both components. moreover, since tweets tend to contain a high degree of implicitness, it can be demanding to judge if a sequence in fact is relating to a claim. we plan to take this issue into account by refining our annotation scheme further. another interesting path of future work will be the continuing development of the argument detector. following @xcite , one possible way of enhancing results could be building a pipeline based on both classification and sequence labeling approaches. more specifically, a classifier customized for identifying argumentative tweets could function as a filter, thereby allowing to train a sequence labeling model on a purely argumentative tweet set. this could increase the model's precision. to this end, we intend to enlarge the number of annotated data.