Verifying Annotation Agreement without Multiple Experts: A Case Study with G ujarati SNACS (2023)

Abstract: Good datasets are a foundation of NLP research, and form the basis for training and evaluating models of language use. While creating datasets, the standard practice is to verify the annotation consistency using a committee of human annotators. This norm assumes that multiple annotators are available, which is not the case for highly specialized tasks or low-resource languages. In this paper, we ask: Can we evaluate the quality of a dataset constructed by a single human annotator? To address this question, we propose four weak verifiers to help estimate dataset quality, and outline when each may be employed. We instantiate these strategies for the task of semantic analysis of adpositions in Gujarati, a low-resource language, and show that our weak verifiers concur with a double-annotation study. As an added contribution, we also release the first dataset with semantic annotations in Gujarati along with several model baselines.

## introduction most nlp research focuses only on a few languages: a small fraction of the about 7,000 languages of the world have datasets or linguistic tools. for example, the universal dependencies project @xcite , perhaps the most linguistically diverse community effort in recent years, only covers about 130 languages. even widely spoken languages like gujarati and hausa-both with about 50 million native speakers @xcite , almost equaling the population of englandare considered low-resource in the nlp context. in a "rich-get-richer" effect, these trends can increase disparities across the world's languages. progress in nlp for high-resource languages like english has been powered not only by advances in modeling techniques, but also by high quality linguistic datasets like the penn treebank @xcite , propbank @xcite , ontonotes @xcite , and timebank @xcite . making and tracking progress in low-resource languages requires building similarly large high quality datasets. but what defines a good dataset? the standard way to measure the quality of a manually annotated dataset involves computing an interannotator agreement metric such as the ubiquitous kappa score @xcite . 1 the very notion of inter-annotator agreement hinges on the availability of at least two annotators. but multiple annotators might not be available, e.g., for a low-resource language. in this paper, we ask: can we verify annotation quality when only a single expert annotator is available? to address this open question, we observe that evaluating the quality of annotated data involves measuring the compatibility between labels proposed by one annotator with a second source of information about the labels. for inter-annotator agreement, the second source is another human trained on the same task. we take the position that for low-resource languages where multiple annotators are unavailable, we can relax the requirement that the second source needs to be a human expert in the same language. to illustrate this position, we propose four weak annotation verifiers and outline the scenarios in which they can be helpful. we evaluate the efficacy of our verifiers via a semantic annotation effort in gujarati, a low-resource indic language. we consider the task of supersense disambiguation of adpositions, which are known to be extremely polysemous. we use an inventory of adpositional supersenses @xcite to construct an annotated corpus of a gujarati version of the book the little prince. experiments with this data indicate that our verifiers can be successfully employed in scenarios where multiple annotators are unavailable. in summary, the contributions of this work are: ## the problem of the single annotator dataset creation in nlp typically involves multiple annotators (experts or crowd workers) labeling a corpus using the task definition and annotation guidelines, followed by a manual or heuristic adjudication step to construct the aggregated ground truth. datasets form the backbone of computational linguistics and nlp research; ensuring their quality is of paramount importance. their quality is commonly measured using annotator agreement, and metrics such as cohen's kappa @xcite reflect consensus @xcite . a good inter-annotator agreement (iaa) score-over 0.6, per @xcite -implies a better agreed-upon dataset, whereas a poor one may indicate gaps in the task definition or annotation guidelines. interesting insights can be drawn by viewing iaa scores alongside model performances. for instance, a dataset with high human consensus and a poor model score suggests a task seemingly simple for humans like common sense reasoning, but difficult for our models @xcite . however, human agreement is undefined when we have only one annotator. this could happen when: (1) the task requires specialized expertise, like biomedical named entity tagging @xcite , or, (2) the language does not have readily accessible nlp expertise @xcite , such as the universal dependencies annotation for the k'iche' @xcite , and breton @xcite languages. in this paper, we study the question of evaluating annotation quality of such singly annotated datasets. ## gujarati snacs: a case study this section introduces a new semantic annotation dataset that will serve as a testbed for our verifiers. ## weak verifiers of annotation quality this section presents four weak verifiers that assess annotation quality. we introduce each one by first stating the prerequisite resources that are needed to use it. we refer to the low-resource language of interest as the target language. we note that the verifiers are all weak: they are not meant to replace a second annotator, but can help gauge dataset quality in the absence of multiple annotators. ## evaluating the verification strategies in this section, we instantiate the verification strategies from §4 for the gujarati snacs annotation task to empirically evaluate them. we show experiments on additional datasets wherever possible. ## commentary on results in this section, we compare our double annotation study with the results from the verifiers. ## conclusions an inter-annotator agreement study is an essential checklist item before the release of human curated datasets. but inter-annotator agreement cannot be computed when we only have one annotator. we address the open question of verifying singly annotated datasets with a new paradigm of exploiting ancillary resources that can serve as weak surrogates for other annotators. the intuition is that each such verifier provides hints about the dataset quality, and their cumulative success is more likely to point to a good dataset. we presented four verification strategies that operate in this paradigm, and a new agreement metric (contextual representation affin-ity). we also created the first semantically focused dataset of adpositions in gujarati, a lowresource language, in the single-annotator setting. we showed that our verification strategies, when instantiated to the new dataset, are promising and concur with a traditional double-annotation study.