Rethinking Annotation: Can Language Learners Contribute? (2023)

Abstract: Researchers have traditionally recruited native speakers to provide annotations for the widely used benchmark datasets. But there are languages for which recruiting native speakers is difficult, and it would help to get learners of those languages to annotate the data. In this paper, we investigate whether language learners can contribute annotations to the benchmark datasets. In a carefully controlled annotation experiment, we recruit 36 language learners, provide two types of additional resources (dictionaries and machine-translated sentences), and perform mini-tests to measure their language proficiency. We target three languages, English, Korean, and Indonesian, and four NLP tasks, sentiment analysis, natural language inference, named entity recognition, and machine reading comprehension. We find that language learners, especially those with intermediate or advanced language proficiency, are able to provide fairly accurate labels with the help of additional resources. Moreover, we show that data annotation improves learners’ language proficiency in terms of vocabulary and grammar. The implication of our findings is that broadening the annotation task to include language learners can open up the opportunity to build benchmark datasets for languages for which it is difficult to recruit native speakers.

## introduction data annotation is important, and in nlp, it has been customary to recruit native speakers of the target languages, even though it is difficult to recruit native speakers for many languages. meanwhile, there are many people learning another language, for instance, duolingo claims that 1.8 billion people are learning a foreign language using their app. 1 in this paper, we examine whether language learners can annotate data as well as native speak-1 https://www.duolingo.com/ ers and whether their annotations can be used to train language models. we explore this question with five control variables that may affect the quality of language learner annotations. these are the language, task, learners' language proficiency, difficulty of the annotation questions, and additional resources that learners can consult. we recruited learners at various levels of proficiency in english (high-resource), korean (mid-resource), and indonesian (low-resource). they annotated data on four tasks, sentiment analysis (sa), natural language inference (nli), named entity recognition (ner), and machine reading comprehension (mrc). we ask three levels of learners to complete multiple sessions of data annotation given with the help of a dictionary or machine-translated texts. our major findings, both in terms of the quality and learning effect of learners' annotations, are summarized as follows: we measure the degree of inter-annotator agreement between learners and ground truth labels, and show that language learners can annotate data at a fairly accurate level, especially for the simpler tasks of sa and ner, and for easy-to medium-level questions. language learners consulting dictionaries generate more accurate labels than learners consulting machine-translated sentences. language models trained on data generated from the distribution of the learners' annotations achieved performance comparable to those of models trained on ground truth labels, demonstrating the efficacy of learner-annotated data. we also observe that learners' language proficiency in vocabulary and grammar tends to improve as they carry out the annotation tasks. we measure their proficiency by conducting pre-and post-tests before and after the annotation. learners perceive that their language proficiency improved during data annotation, and most were willing to re-participate in the process. we hope this paper allows researchers to question the necessity of recruiting native speakers for data annotation and call on other nlp researchers carefully to consider the criteria by which to recruit crowdworkers for data annotation carefully. ## related work we can group annotators of nlp datasets into language learners, non-speakers, and non-experts. language learners are people who are learning the target language, while non-speakers are those who have never learned the target language. nonexperts are people who have no expertise in nlp tasks or data annotations. we look at previous work with these three annotator groups. ## study design this section describes how we carefully design our controlled experiments with diverse factors that may affect the quality of learners' annotations and the learning effect. ## experimental results we discuss the results of our experiments with respect to two research questions: all findings we discuss in this section were shown to be statistically significant at p level @xmath0.05 using anova. specifically, comparisons for annotation accuracy, annotation time, and survey responses were analyzed with four-way anova over the four between-subject factors of task, language, additional resources, and learner level. comparisons between pre-test and post-test results were done with a mixed two-way anova with learner level and additional resources as between-subject factors. pairwise t-tests were conducted for all factors with bonferroni corrections. ## conclusion this study provides interesting results both for the actual dataset annotation as well as understanding the non-native speakers' annotation capabilities. we show (1) labels provided by language learners are nearly as accurate, especially for easier tasks, (2) with additional experiments of aggregating their labels, learners' are almost on par with native speakers, and ( 3 ) language models trained time / session (min) expected hourly wage native speakers -8.08 ±0.89 krw 9,282 language learners dictionary 14.76 ±1.29 krw 20,325 translation 13.20 ±0.73 krw 22,727 by showing that nlp annotation does not require finding native speakers, we show the possibility of broadening nlp research for more languages, as it is very challenging to recruit native speakers for many languages. requiring native speakers for annotation can mean traveling to remote locations and working with an older, less-technology-savvy population. we show that it is possible to work with language learners to hurdle geographic and technological barriers when attempting to build annotated nlp datasets. we believe learners with high motivations and learning effects are more likely to be engaged in data annotation.