Investigating label suggestions for opinion mining in G erman Covid-19 social media (2021)

Abstract: This work investigates the use of interactively updated label suggestions to improve upon the efficiency of gathering annotations on the task of opinion mining in German Covid-19 social media data. We develop guidelines to conduct a controlled annotation study with social science students and find that suggestions from a model trained on a small, expert-annotated dataset already lead to a substantial improvement – in terms of inter-annotator agreement (+.14 Fleiss’ κ) and annotation quality – compared to students that do not receive any label suggestions. We further find that label suggestions from interactively trained models do not lead to an improvement over suggestions from a static model. Nonetheless, our analysis of suggestion bias shows that annotators remain capable of reflecting upon the suggested label in general. Finally, we confirm the quality of the annotated data in transfer learning experiments between different annotator groups. To facilitate further research in opinion mining on social media data, we release our collected data consisting of 200 expert and 2,785 student annotations.

## introduction the impact analysis of major events like the covid-19 pandemic is fundamental to research in social sciences. to enable more socially sensitive public decision making, researchers need to reliably monitor how various social groups (e.g., political actors, news media, citizens) communicate about political decisions @xcite . the increasing use of social media especially allows social science researchers to conduct opinion analysis on a larger scale than with traditional methods, e.g. interviews or questionnaires. however, the publication of research results is often delayed or temporally transient due to limitations of traditional social science research, i.e. prolonged data gathering processes or opinion surveys being subject to reactivity. given the increasing performance of language models trained on large amounts of data in a self-supervised manner @xcite , one fundamental question that arises is how nlp systems can contribute to alleviate existing difficulties in studies for digital humanities and social sciences @xcite . one important approach to make data annotation more efficient is the use of automated label suggestions. in contrast to active learning, that aims to identify a subset of annotated data which leads to optimal model training, label suggestions alleviate the annotation process by providing annotators with pre-annotations (i.e., predictions) from a model @xcite . to enable the annotation of large amounts of data which are used for quantitative analysis by disciplines such as social sciences, label suggestions are a more viable solution than active learning. one major difficulty with label suggestions is the danger of biasing annotators towards (possibly erroneous) suggestions. so far, researchers have investigated automated label suggestions for tasks that require domain-specific knowledge @xcite @xcite ; and have shown that domain experts successfully identify erroneous suggestions and are more robust to potential biases. however, the limited availability of such expert annotators restricts the use of label suggestions to small, focused annotation studies. for tasks that do not require domain-specific knowledge and can be conducted with non-expert annotators -such as crowd workers or citizen science volunteers -on a large scale, label suggestions have not been considered yet. this leads to two open questions. first, if non-expert annotators that do not receive any training besides annotation guidelines benefit from label suggestions at all. second, if existing biases are amplified especially when including interactively updated suggestions that have been shown to be advantageous over static ones @xcite . we tackle these challenges by conducting a comparative annotation study with social science students using a recent state-of-the-art model to generate label suggestions @xcite . our results show that a small set of expert-labeled data is sufficient to improve annotation quality for non-expert annotators. in contrast to @xcite , we show that although interactive and non-interactive label suggestions substantially improve the agreement, we do not observe significant differences between both approaches. we further confirm this observation with experiments using models trained on (and transferred to) individual annotator groups. our contributions are: c1: an evaluation of label suggestions in terms of annotation quality for non-expert annotators. c2: an investigation of label suggestion bias for both static and interactively updated suggestions. ## related work label suggestions. in an early work, @xcite study the effects of label suggestions on the task of word sense disambiguation and observe a positive effect on annotation quality. with the introduction of annotation tools such as brat @xcite , webanno @xcite , or inception @xcite , the use of label suggestions became more feasible; leading to an increased investigation of label suggestions in the context of nlp. for instance, @xcite investigate label suggestions for amharic pos tagging and german named entity recognition and show with expert annotators that label suggestions significantly reduce the annotation time. other works further investigate interactively updated label suggestions and come to a similar conclusion @xcite . label suggestions have also been shown to be effective in non-nlp annotation tasks that require domain-specific knowledge such as in medical @xcite or educational @xcite use cases. ## annotation task our task is inspired by social science research on analyzing public opinion using social media @xcite . the goal is to identify opinions in german-speaking countries about governmental measures established to contain the spread of the corona virus. we use twitter due to its international and widespread usage that ensures a sufficient database and the several challenges for the automatic identification of opinions and stance it poses from an nlp perspective @xcite @xcite . for example, the use of language varies from colloquial expressions to well-formed arguments and newsspreading statements due to its heterogeneous user base. additionally, hashtags are used directly as part of text but also to embed the tweet itself in the broader discussion on the platform. finally, the classification of a tweet is particularly challenging given the character limitation of the platform, i.e., at the date of writing twitter allows for 280 characters per tweet. data collection. initially, we collected tweets from december 2019 to the end of april 2020. using a manually chosen set of search queries ('corona', 'pandemie', 'covid', 'socialdistance'), we made use of the twitter streaming api and gathered only those tweets which were classified as german by the twitter language identifier. this resulted in a set of approximately 16.5 million tweets. we retained only tweets that contain key terms referring to measures related to the covid-19 pandemic and removed all duplicates, retweets and all tweets with text length less than 30 characters. after filtering, 237,616 tweets remained and their daily temporal distribution is visualized in figure 1 . we sample uniformly at random from the remaining tweets for all subsequent annotation tasks. 2 annotation scheme. we developed annotation guidelines together with three german-speaking researchers from social sciences and iteratively refined them in three successive rounds. our goal from a social science perspective is to analyze the public perception of measures taken by the government. therefore, the resulting dataset should help in (1) identifying relevant tweets for governmental measures and if relevant, ( 2 ) detecting what stance is expressed. we follow recent works on stance detection and twitter data @xcite @xcite and use four distinct categories for our annotation. they are defined as follows: unrelated: no measures related to the containment of the pandemic are mentioned comment: measures are mentioned, but not assessed or neutral support: measures are assessed positively refute: measures are assessed negatively the four label annotation scheme allows us to distinguish texts that are related to the pandemic but do not talk about measures (i.e., unrelated). ## study setup our goal is to study the effects of interactively updated and static label suggestions in non-expert annotation scenarios. non-experts such as crowd workers or student volunteers have no prior experience in annotating comparable tasks and only receive annotation guidelines for preparation. 3 our secondary goal is to collect a novel dataset that can be used by social science researchers to study the effects of governmental measures for preventing the spread of covid-19 on the public opinion. to train a model that provides label suggestions to our non-expert annotators, we first collect a small set of 200 expert-annotated instances. we then split our non-expert annotators into three different groups that receive (g1) no label suggestions, (g2) suggestions from a model trained on expert annotations, and (g3) suggestions from a model that is retrained interactively using both expert-annotated and interactively annotated data. ## study evaluation outliers. a fine-grained analysis of annotation time is not possible due to online annotations at home. however, one student in g3 had, on average, spent less than a second for each annotation and accepted almost all suggested labels. this student's annotations were removed from the final dataset and assumed as faulty labels considering the short amount of time spent on this task in comparison to the minimum amount of seven seconds per tweet and annotation for all other students. ## conclusion in this work, we analysed the usefulness of providing label suggestions for untrained annotators to identify opinions in a challenging text domain (i.e., twitter). we generated suggestions using expert-labeled training data as well as interactively training models using data annotated by untrained students. our results show that label suggestions from a state-of-the-art sentence classification model trained on a small set of expert annotations help improving annotation quality for untrained annotators. in terms of potential biases that may occur with untrained annotators we observe that the students retained their capability to reflect on the suggested label. we furthermore do not observe a general amplification in terms of bias with interactively updated suggestions; however, we find that such effects are very specific to individual annotators. we hence conclude that interactively updated label suggestions need to be considered carefully when applied to non-expert annotation scenarios. for future work, we plan to leverage our setup to annotate tweets from a larger time span. in germany, the measures taken by the government have been met with divided public reaction -starting with reactions of solidarity and changing towards a more critical public opinion @xcite . in particular, we are interested if our label suggestion model is robust enough to account for such a shift in label distribution. research project. thereby, we adhere to the twitter developer policy foot_7 and give users full control of their privacy and data as they can delete or privatize tweets so that they cannot be collected. we asked student annotators for voluntary participation in the annotation study. all students have been informed about the goal of the conducted research and the purpose of the collected annotations. during annotation no information about the tweet's author or any other additional metadata was made available to the annotators. we did not collect any personal data from the students before, after, or during the annotation task. data usage. this work presents an investigation of efficient data annotation methods in a case study on social media data. the results of this work allow social science researchers to apply their analysis on a larger scale. in the case of analyzing public opinion on governmental measures, the resulting analysis allows politicians to make more socially sensitive public decisions. this information is useful in aggregated form, without the need for information about individual users. however, we want to point out that users of social media (particularly twitter) do not constitute a representative sample of the general population, especially in germany @xcite . therefore, our goal is not to foster public decision-making solely based upon analysis of twitter but to provide an additional supporting tool. dual use. further, we acknowledge the potential of misuse of our dataset: the annotated data allows anyone, including both individuals and organizations, for training models to identify individuals expressing their consent or dissent with governmental actions. to this end, we follow the argumentation by @xcite that in general we cannot prevent publicly available data from being misused but we want to make both researchers and the general public aware of the possible malicious use. during our preliminary studies, we identified several ambiguities regarding the stance annotation which is in the nature of the source of the texts (twitter) and the subject of the annotation (measures regarding the covid-19 pandemic): these decisions are reflected at the corresponding positions in the annotation guidelines, along with several example tweets. in the end we provide a note that twitter posts may contain malicious, suggestive, offensive, or potentially sensitive content and that the annotation can be paused and resumed at any time.