Lessons Learned from a Citizen Science Project for Natural Language Processing (2023)

Abstract: Many Natural Language Processing (NLP) systems use annotated corpora for training and evaluation. However, labeled data is often costly to obtain and scaling annotation projects is difficult, which is why annotation tasks are often outsourced to paid crowdworkers. Citizen Science is an alternative to crowdsourcing that is relatively unexplored in the context of NLP. To investigate whether and how well Citizen Science can be applied in this setting, we conduct an exploratory study into engaging different groups of volunteers in Citizen Science for NLP by re-annotating parts of a pre-existing crowdsourced dataset. Our results show that this can yield high-quality annotations and at- tract motivated volunteers, but also requires considering factors such as scalability, participation over time, and legal and ethical issues. We summarize lessons learned in the form of guidelines and provide our code and data to aid future work on Citizen Science.

## introduction data labeling or annotation is often a difficult, timeconsuming, and therefore expensive task. annotations are typically drawn from domain experts or are crowdsourced. while experts can produce high-quality annotated data, they are expensive and do not scale well due to their relatively low number @xcite . in contrast, crowdsourcing can be relatively cheap, fast, and scalable, but is potentially less suited for more complicated annotation tasks @xcite ). another approach is using citizen science, which describes the participation and collaboration of volunteers from the general public with researchers to conduct science @xcite . over the past decade, citizen science platforms, which rely on unpaid volunteers to solve scientific problems, have been used for a wide variety of tasks requiring human annotation @xcite , e.g., classifying images of galaxies @xcite or for weather observation @xcite . while citizen science has been shown to produce high-quality annotations in ecological or environmental projects @xcite , its potential has so far not been investigated in depth for natural language processing (nlp). our goal in this work is to assess the practicality of undertaking annotation campaigns for nlp via citizen science. we analyze whether volunteers actually react to our calls and participate, how the resulting quality is compared to crowdsourcing, what the benefits and shortcomings are and what needs to be taken into account when conducting such a project. we especially are interested in differences between annotators recruited via different channels, which we investigate by advertising to different social media platforms, nlp-related mailing lists, and university courses. to explore this possibility, we use the perspectrum dataset @xcite , cc-by-sa) that focuses on the task of stance detection and can be motivated by fighting misinformation and promoting accurate debate in internet discussions. we replicated a portion of the annotations in this dataset using citizen scientists instead of crowdworkers. to accomplish this goal, we designed an annotation workflow that is suitable for citizen science and allows us to recruit volunteers across a variety of platforms. our contributions are the following: our results show that using citizen science for linguistic annotation can result in high-quality annotations, but that attracting and motivating people is critical for its success, especially in the longterm. we were able to attract 98 volunteers when conducting our citizen science project which resulted in 1,481 annotations over 2 months, thereby re-annotating around 10% of the original dataset. we find that annotations obtained through mailing lists and university students were of high quality when comparing them to the original, adjudicated crowdsourced data. we thus conclude that citizen science projects have the potential to be applied to nlp annotation if they are conceptualized well, but are best suited for creating smaller datasets. ## background prior work has developed various means and strategies for annotating large datasets. so far, annotation studies in nlp mainly use domain-experts or crowdworkers, or a mix of both @xcite . crowdsourcing in particular has received increasing attention over the past decade @xcite . paid experts recruiting domain experts (e.g., linguists) for annotation studies has been a widely accepted method to generate linguistically annotated corpora. famous examples are the brown corpus @xcite or the penn treebank @xcite . while the resulting datasets are of the highest quality, domain experts are often few, and such annotation studies tend to be slow and expensive @xcite . although many researchers moved on to annotation studies that recruit crowdworkers, expert annotations are still necessary in various fields, e.g., biomedical annotations @xcite . crowdsourcing to accelerate the annotation process and reduce costs, researchers have utilized crowdsourcing as a means to annotate large corpora @xcite . the main idea behind crowdsourcing is that annotation tasks that do not require expert knowledge can be assigned to a large group of paid non-expert annotators. this is commonly done via crowdsourcing platforms such as amazon mechanical turk (amt) or upwork and has been successfully used to annotate various datasets across different tasks and domains @xcite . previous work compared the quality between crowdsourcing and expert annotations, showing that many tasks can be given to crowdworkers without major impact on the quality of annotation @xcite @xcite . although crowdworkers can substantially accelerate annotation, crowdsourcing requires careful task design and is not always guaranteed to result in high quality data @xcite . moreover, as annotators are compensated not by the time they spend but rather by the number of annotated instances, they are compelled to work fast to maximize their monetary gain-which can negatively affect annotation quality @xcite or even result in spamming @xcite . it can also be difficult to find crowdworkers for the task at hand, for instance due to small worker pools for languages other than english @xcite or because the task requires special qualifications @xcite . finally, the deployment of crowdsourcing remains ethically questionable due to undervalued payment @xcite , privacy breaches, or even psychological harm on crowdworkers @xcite . games with a purpose a related but different way to collect annotations from volunteers is games with a purpose, i.e., devising a game in which participants annotate data @xcite . works propose games for different purposes and languages. for instance, anaphora annotation (phrasedetectives, @xcite , dependency syntax annotation @xcite , or collecting idioms @xcite . it has been shown that if a task lends itself to being gamified, then it can attract a wide audience of participants and can be used to create large-scale datasets @xcite . finally, @xcite investigate games with a purpose in the context of (second) language learning to simultaneously crowdsource annotaions from learners as well as teachers. one such example is substituto, a turn-based, teachermoderated game for learning verb-particle constructions @xcite . we do not consider gamification in this work, as enriching tasks with game-like elements requires considerable effort and cannot be applied to every task. citizen science citizen science broadly describes participation and collaboration of the general public (the citizens) with researchers to conduct science @xcite . citizen science is a popular alternative approach for dataset collection efforts, and has been successfully applied in cases of weather observation @xcite , counting butterflies @xcite or birds (national audubon society, 2020), classifying images of galaxies @xcite or monitoring water quality @xcite . newlyemerging technologies and platforms further allow researchers to conduct increasingly innovative citizen science projects, such as the prediction of influenza-like outbreaks @xcite or the classification of animals from the serengeti national park @xcite . languagearc is a citizen science platform for developing language resources @xcite . it is however not open yet to the public to create projects and does not easily allow conducting a citizen science meta-study as we do in this work. one work using languagearc is by @xcite (ld) who collected resources to evaluate bias in language models. they did not investigate the impact of using different recruitment channels which we do. other projects using languagearc are still running and it is too early to derive recommendations from. compared to crowdsourcing, citizen science participants are volunteers that do not work for monetary gain. instead, they are often motivated intrinsically. for instance, they may have a personal interest on positively impacting the environment @xcite , or in altruism @xcite . asking for unpaid work also entails various issues like finding good ways of how to attract volunteers, and ethical considerations @xcite ) that need to be addressed (cf. §5). intrinsic motivation also has the potential of resulting in higher-quality annotations compared to crowdsourcing. for instance, @xcite find in their evaluation study with citizen scientists that their participants may have been willing to take more time annotating for the sake of higher annotation accuracy. however, as their main goal was to conduct an evaluation study for their specific setup, this finding cannot be generalized to other citizen science scenarios. so far, only @xcite provide a direct comparison between crowdsourcing and citizen science and show that volunteers can achieve similar performance in mining medical entities in scientific texts. they recruit participants through different channels such as newspapers, twitter, etc., but do not compute channel-specific performance, making it difficult to assess whether the quality of the resulting annotation depends on the recurrent channel. in contrast, in the present work, we explicitly consider the recruitment channel in our evaluation and furthermore provide a discussion and guidelines for future citizen science practitioners. also, it attracts intrinsically (not only fiscally) motivated volunteers that are often skilled in the task and can provide high-quality annotations, thus potentially combining the advantages of expert annotations and crowdsourcing. relying on unpaid annotators entails several issues, including attracting volunteers and ethical considerations @xcite ) that need to be taken into account (see §5). ## study design to study the feasibility of citizen science for nlp annotation, we asked volunteers recruited via various channels to re-annotate an existing, crowdsourced dataset. the general setup is described in fig. 1 . to conduct a systematic study, we identified the following four necessary steps: 1) identifying a suitable dataset ( §3.1); 2) selecting suitable recruitment channels to advertise our project on ( §3.2); 3) building a landing page for onboarding participants that asks for informed consent and the channel from which they originated ( §3.3); 4) setting up the annotation editor to which participants are forwarded after the onboarding ( §3.4). ## results we conducted our study between january and march 2022 and promoted the task in successive rounds across all recruitment channels. in total, we were able to recruit 98 participants who provided 1481 annotations resulting in 906 fully annotated instances. each instance with at least one annotation has received on average 1.63 annotations. detailed statistics are provided in the appendix. ## discussion and takeaways here we present lessons learned, discuss legal challenges and ethical considerations, as well as provide guidelines for future citizen science projects. channel-dependent differences our results clearly differ across recruiting channels. we find that overall, facebook and linkedin have the lowest turnout and accuracy when compared to the gold labels, followed by twitter. our assumption for the overall low participation is that our network for these channels was not large enough. advertising our study to nlp-related and universityinternal mailing lists and university courses yielded the highest number of participants who also provided the most and best-quality annotations. although our results show that students may outperform participants from other channels, we also acknowledge that this may not always be a viable option to recruit citizen scientists. overall, our findings indicate that it is important to address the respective target groups that may be interested in a specific study. however, we also note that continuously advertising citizen science studies to the same channels may have a negative impact, as it can cause participation fatigue and lead to fewer volunteers participating. one possible solution could be the use of languagearc @xcite from the ldc and centralize calls for participation. ## conclusion in this work, we presented an exploratory annotation study for utilizing citizen science for nlp annotation. we developed an onboarding process that can easily be adapted to similar projects and evaluated citizen science annotations for reannotating an existing dataset. furthermore, we extended the inception platform, a well-known open-source semantic annotation platform, with a dynamic workload manager and functionality for granting access to external users without registration. this enables its usage for citizen science projects. we advertised the study via twitter, facebook, linkedin, mailing lists, and university courses and found that participants from mailing lists and university courses are especially capable of providing high-quality annotations. we further discuss legal and ethical challenges that need to be addressed when conducting citizen science projects and provide general guidelines for conducting future projects that we would like to have known before starting. overall, we conclude that citizen science can be a viable and affordable alternative to crowdsourcing, but is limited by successfully keeping annotators motivated. we will make our code and data publicly available to foster more research on citizen science for nlp and other disciplines. future work we see the following directions for further research and evaluation to better understand in which settings citizen science can be applicable and how to use it best. here, we used perspec-trum as the dataset to annotate and mentioned in the participation calls that it benefits the social good. therefore, it would be interesting to conduct more projects and see which datasets are suitable as well as whether volunteers participate, even if there is no extrinsic motivation. then, it can also be tested how annotator retention develops, especially when project are running longer. the call for participation itself could also be investigated for the impact it has on turnout, motivation and quality.