Aggregating and Learning from Multiple Annotators (2021)

Abstract: The success of NLP research is founded on high-quality annotated datasets, which are usually obtained from multiple expert annotators or crowd workers. The standard practice to training machine learning models is to first adjudicate the disagreements and then perform the training. To this end, there has been a lot of work on aggregating annotations, particularly for classification tasks. However, many other tasks, particularly in NLP, have unique characteristics not considered by standard models of annotation, e.g., label interdependencies in sequence labelling tasks, unrestricted labels for anaphoric annotation, or preference labels for ranking texts. In recent years, researchers have picked up on this and are covering the gap. A first objective of this tutorial is to connect NLP researchers with state-of-the-art aggregation models for a diverse set of canonical language annotation tasks. There is also a growing body of recent work arguing that following the convention and training with adjudicated labels ignores any uncertainty the labellers had in their classifications, which results in models with poorer generalisation capabilities. Therefore, a second objective of this tutorial is to teach NLP workers how they can augment their (deep) neural models to learn from data with multiple interpretations.

## description the disagreement between annotators stems from ambiguous or subjective annotation tasks as well as annotator errors. crowdsourcing with non-expert annotators is especially prone to annotation errors, sometimes caused by workers who do not attempt to provide correct annotations (spammers). the traditional resolution to this problem is redundant labeling: collect multiple interpretations from distinct coders, allowing the resource creators to later aggregate these labels. to this end, probabilistic models of annotation have been successfully used to learn the coders' behavior and distill the labels from noise. the research on models of annotation contains a large body of work spanning multiple decades (from the work on latent structure analysis back in the early 70s), and has been substantially debated over the years at dedicated conferences such as hcomp and workshops, e.g., from the people's web meets nlp @xcite , to crowdml ( http://crowdml.cc/ ), and more recently annonlp @xcite . the plethora of models that had been published even prompted some researchers to ask, challengingly, whether the problem of aggregating crowd labels had been solved @xcite . as anticipated, there are still unaddressed issues -in particular, the bulk of work has focused on classification tasks, leaving room for innovation in other areas. the nlp field specifically contains a number of tasks with unique characteristics not considered by standard models of annotation. for example, in sequence labeling tasks such as part of speech tagging or named entity recognition, nearby labels have known inter dependencies. in other tasks such as anaphoric annotation for coreference resolution, the coders are asked to provide labels that are not from a fixed set of categories but consist of textual mentions. another example is pairwise preference labelling, where coders are asked to choose the instance from a pair that most strongly reflects a quality of interest, such as relevance to a topic or convincingness of an argument, with the goal of inferring an overall ranking of text instances. researchers have observed these gaps in the literature and are addressing them. a key objective of this tutorial is to connect nlp researchers with state-of-the-art aggregation methods suitable for canonical nlp tasks, covering classifications @xcite , sequence labels @xcite , anaphoric interpretations @xcite and pairwise preference labels @xcite . resource creators can use aggregation methods to adjudicate the disagreements inherent in annotated data, but at times, when the resource is to serve as training data to a machine learning model, the noise distillation procedure does not have to be separated and can be integrated into the learning process. in fact, by following the convention and training with adjudicated labels we ignore any of the uncertainty the labellers had in their classifications. including the coders' disagreements in the learning signal offers the models a richer source of information compared to adjudicated labels: they include not only the consensus, but may also indicate ambiguity, and how the humans make mistakes. this improves the generalisation capability of the models and offers them a more graceful degradation with less ridiculous mistakes @xcite . some of these approaches can also be used for their noise distillation capabilities, as their learning processes also produce aggregated labels that leverage not only coder annotation patterns but also the knowledge of the task accumulated by the model @xcite @xcite . often, this means that fewer redundant labels are required to attain the desired level of accuracy for the aggregated labels. thus, a second objective of the tutorial is to teach nlp researchers how they can augment their existing (deep) neural architectures to learn from data with disagreements.