Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with D is C o (2023)

Abstract: Annotator disagreement is common whenever human judgment is needed for supervised learning. It is conventional to assume that one label per item represents ground truth. However, this obscures minority opinions, if present. We regard “ground truth” as the distribution of all labels that a population of annotators could produce, if asked (and of which we only have a small sample). We next introduce DisCo (Distribution from Context), a simple neural model that learns to predict this distribution. The model takes annotator-item pairs, rather than items alone, as input, and performs inference by aggregating over all annotators. Despite its simplicity, our experiments show that, on six benchmark datasets, our model is competitive with, and frequently outperforms, other, more complex models that either do not model specific annotators or were not designed for label distribution learning.

## introduction human feedback remains a critical component as machine-learning-based systems play an ever larger role in society and our daily lives. furthermore, when these systems fail, they assume that there is a single, correct answer in every case. yet, due to differences in perception, context, experiences, attitudes that vary from person to person, and demographic differences, humans often disagree on what the "right response" should be. for instance, @xcite showed that female annotators frequently disagree with males on what constitutes offensive speech. this prevalence of disagreement in humanlabeled data has made annotator modeling a popular research problem. in its most elementary form, each item is assigned the majority label. more sophisticated approaches seek to understand annotator behavior @xcite @xcite via machine learning. yet the vast majority of these approaches require or assume that ground truth is a single (but unknown) label (or collection of labels) and that any deviation from the ground truth label is indicative of poor quality. consequently, most models learn to discriminate between "good" and "bad" annotators @xcite and resolve disagreement out of existence. however, on problems such as hate speech detection, language complexity, or machine translation, disagreement may actually signify the views of vulnerable communities that should be preserved @xcite @xcite , or even made predictable @xcite . a major barrier to achieving these goals is annotator sparseness. human annotators are an expensive and often limiting factor in a learning loop. it is usually not feasible to collect enough annotations from each item in order to have confidence in them as a representative sample of the underlying population's response. in this paper, we explore the idea that, in key settings, ground truth is more plainly seen as a distribution of labels representing the opinions and beliefs of a (partially observed) population of annotators, rather than a single label (or multi-label). in the extreme (as we do here), this approach ignores the near-certainty that some annotators are unreliable. however, the goal of modeling as precisely as possible annotator responses at the population level is a transparent, data-conservative approach to preserving annotator information, as opposed to the conventional approach of resolving disagreement before learning. later in this work, we will discuss extensions for modeling annotator reliability. we propose a new neural model, disco (distribution from context), designed to address the annotator sparsity problem. at training time, the model takes in as input a training example and an annotator id, and outputs three simultaneous predictions: the label the annotator gives the example, the distribution of all labels the example received (from all annotators who responded to it) and the distribution of all labels (over all examples) that the annotator provided. at inference time, it takes an unlabeled item as input and predicts the distribution of labels that it would receive from the population of annotators. it ties together two rather successful prior approaches: label distribution learning (ldl) @xcite and item-annotator modeling @xcite . our models are publicly available. 1 in this work, we address the following questions: rq1 how does the performance of disco compare to that of ldl approaches that do not model annotators? rq2 how does the performance of disco compare to that of non-ldl approaches that model annotators? to answer these questions, we evaluate disco against three competitive models that exemplify, respectively, the conventional ground truth approach, a label distribution approach without annotator modeling, and an approach that models annotators but, unlike disco, is not purpose-built for distributional ground truth. we test these models on six benchmark datasets that contain annotator-item assignments and annotator-level labels. we evaluate our models on two different gold standards: the most frequent label and the label distribution. our work is philosophically aligned with welldocumented analyses of inherent annotator disagreement @xcite @xcite or annotator bias @xcite . however, we are going beyond simply analyzing the difference. rather, we seek to leverage the distribution of annotator responses as a signal to be learned for its own sake. the study of annotator disagreement has a long history, coincident with the emergence of data-driven behavioral research @xcite . @xcite introduced item-annotator tableau models. they use the multiple labels associated with each data item and each annotator to jointly estimate the ground truth label of each item as well as the error rate of each annotator. their approach uses only the labels, not the data item features associated with them, and so, alone, this method cannot outperform supervised learning. rather, it is used as the first of a two-step learning process, where the second step can be any supervised learning algorithm. later researchers put this model on a fully bayesian foundation @xcite or considered more complex models of annotators, ground truth, or both @xcite . notably, (as spam is a common problem in crowdsourced label sets) several investigators distinguish between honest and dishonest annotators @xcite . more recently, investigators have studied clustering as an unsupervised approach to discover annotators with similar behavior @xcite . yet all of these approaches are still based on the assumption that each item is associated with a single ground truth label. label distribution learning (ldl), in contrast, assumes that the ground truth itself is a distribution. however, this distribution does not necessarily come from a population of annotators @xcite @xcite . notably, ldl has proven useful in a diverse range of settings @xcite @xcite @xcite @xcite . here, the goal is to predict the distribution of labels associated with an item rather than a single ground truth label. it is relatively natural, in this setting, to consider clustering together related data items in order to improve the ground truth estimates of label distributions, as several prior efforts have done, either in the feature space of the items @xcite @xcite or directly in the label space of the items themselves @xcite . note that models that cluster only in the label space can only be used as the first step in a two-step supervised learning process (for the same reason that the david and skene model can only be used in this way). our work is most closely aligned with others who seek not only to gain understanding of annotator disagreement, but to predict it for it's own sake. crowdtruth (aroyo and welty, 2013; dumitrache et al., 2018) views truth in crowdsourcing as a function of the data, response space, and workers who annotate the data. @xcite study modeling and predicting annotator behavior for specific demographic groups. their approach is based on a recommender system. @xcite also propsed a model to learn and predict labels using annotator demographics. in contrast to their work, disco is able to jointly model and learn from annotator behavior, their annotations, and the content of the data itself. we are interested in distributions over annotator responses, for any slice of y, horizontally (denoted y n,• ) or vertically (denoted y •,m ), as well as the response of an individual annotation to an individual item (denoted y m,n ). we also viewed the responses in each slice as a probability distribution over the space of possible responses. let # denote an operator that converts a (horizontal or vertical) slice y into a vector #@xmath0, 1] q , i #y @xmath1, 2, . . . , q} as a probability distribution. so, e.g., if there are three responses of "2" out of 10 responses total in y •,m , then #(y •,m ) 2 = 0.3. we conducted our experiments on the few publicly available human annotated datasets with annotator assignments. foot_1 see ## disco: a neural probabilistic model for estimating label distributions disco (figure 2 ) stands for "distribution from context," because it takes two inputs, an item x •,m and an annotator a n , and then learns to jointly predict the annotator's response to the item y n,m , the distribution of all responses to the item #y •,m , and the distribution of all responses the annotator provides (to all items) #y n,• . additionally, we intend the name to invoke the inclusive, diversity-celebrating spirit of the early disco movement, as preserving annotator diversity is the primary motivation behind the design. note that, because the annotators in of our datasets are completely anonymous (except for their set of responses), we represent a n as a onehot vector 0 n-1 10 n -n . in future work, we hope to have annotator features associated with key features believe to drive disagreement, such as age, race, gender, ethnicity, political affiliation etc.. also, because we only deal with vertical slices of x, for clarity we denote x •,m as x m . we denote the machine predictions of y n,m , #y •,m , and #y n,• as z y , z yi , and z ya , respectively. although, strictly speaking, only z y is needed for prediction, (x m , a n ) represents the intersection of a column and row of the label matrix y, and z yi or z ya represent the marginal distribution associated with this column or row, respectively. it also provides the same context during training that many of the established item-annotator models rely on @xcite . moreover, items and annotators tend to cluster in label distribution space @xcite notice that that x m and a n are first each embedded into their own respective sub-spaces (z i and z a ) before they are combined through a vector combination operator (such as concatenation). in order to facilitate tractable inference and parameter learning, we opted to craft a probabilistic encoder-decoder architecture. disco is defined by a set of synaptic weight parameter matrices housed in two constructs θ @xmath2, w a , w p , w e } and θ @xmath3, w ya , w y } (bias vectors omitted for clarity), where θ e contains the encoder parameters and θ d contains the decoder parameters. the model is designed, for each data item feature vector x m and annotator a n pair, to estimate the values of a set of target label distributions, i.e., the item label distribution y i the annotator label distribution y a , and the (ground-truth) label distribution y m,n . the output of the encoder is the latent representation of data items and annotators -note that the data item is projected to the space z i while the annotator identification integer is embedded into the space z a . as a result, the encoder, which takes in as input the data item feature vector x @xmath4, computes the following output: the decoder, which takes in as input the latent code produced by the encoder z e , computes its outputs (three different label distribution estimates) as follows: ## experimental setup before conducting the research described here, we consulted with our institutional review board(s). they determined it did not constitute human subjects research, primarily because the data was publicly available and secondary. beyond that, all authors have basic training on conducting human subjects research from citi. 4 moreover, we do not reveal any apparent personal identifiers in the data that we use. in cases when the original data splits are not provided, we use a 50/25/25-percent train/dev/test split. for natural language datasets d ge , d jq1-3 , d mr , and d si , we used sbert (reimers and gurevych, 2019) with the pretrained paraphrase-minilm-l6-v2 model to generate sentence embeddings as our feature vectors. the model generates embeddings over a 384 dimension space. for d lm , we use features that are distributed with the data set -these are pre-encoded using vgg-16 @xcite . there are relatively few publicly available datasets that provide label distributions, rather than single labels or label sets. there are even fewer that say which annotators labeled which items (i.e., that provide the annotator label matrix y). these annotator assignments (or annotator-level labels (prabhakaran et al., 2021)) are essential for modeling annotators. thus, we based our comparison models from those that had been previously tested on data with annotator assignments, even if some of the models in question do not use them. one model (cnn) is a baseline that does no pre-processing or modeling of the labels. another (mm+cnn) is ldl aware, but does not explicitly model annotators. the third (cl) models annotators, but is not explicitly designed for ldl. in the appendix, we present a short description of our baselines. ## results and discussion rq2 asks about the performance of disco versus approaches that model annotators, but are not ldlbased, i.e., cl and ds+cnn in our experiments. in contrast to rq1, disco outperforms all of these models across all datasets. learning label distributions can result in better single-label learning @xcite @xcite . when evaluated as a single label problem, disco beats all of the other models in all but one (mm+cnn beats it in terms of accuracy in d ge ). even then, a common dataset between sl and ld measure which bypasses disco is d ge . notably, this is one of the largest datasets with offensive language content, with a large label selection and high number of items per annotator. to get a sense of the impact of the w y i and w y a aggregating layers of disco, we included results using the model but with those layers removed (@xmath5. the model shows substantial improvement in nearly all tests. to gain a qualitative sense of our results, we inspected several of the test splits of the more interesting datasets for examples on which disco assigned nearly even weights to the two highestscoring labels. the sbic intent (d si ) dataset is one on which disco performed the best. it is also the one for which disco would be expected to yield the most interesting results, as annotator disagreement could be quite significant. many of these results were jokes, such as "what do you get when you mix human dna and goat dna? kicked out of the petting zoo." with (intended, not-intended) = (0.35, 0.65) or "why was the lord of the rings trilogy filmed in new zealand ? cause the us were missing the two towers." with (intended, notintended) = (0.30, 0.70), which are clearly offensive to some people, but apparently funny to others. there are also politically charged messages such as "we need to bring back monster trucks, guns, heavy metal 1776 maga i want trumps next speech to have monster trucks jumping over an ac/dc concert," with (intended, not-intended) = (0.35, 0.65), or the use of racially derogative terms that may not be universally recognized as such. wan et al. ( 2023 ) also proposed a model using d si dataset for modeling annotators based on their demographic details. we summarize f1-scores for all of the baselines and disco in figure 3 . the contextual learning ability of disco does show a significant improvement over the prior models which do not perform in a similar manner. on the other hand, items on which the prediction assigned all or nearly all of the probability mass to one label tended to be very obviously racist and/or hateful. in the specific research focus of hate or offensive speech and monitoring in real-world settings involving contentious issues @xcite , there is a growing consensus that human-in-the-loop systems aided by automated methods can be more robust in handling controversial edge cases. if our automated method assigns nearly even weights to two (or more) highest scoring labels, perhaps those instances merit greater scrutiny and vetting from multiple web moderators. since real-world human moderation is costly, our model can potentially serve as a guide in prioritizing human moderation resources. our involvement in label distribution learning came from a community-based participatory research group that we belonged to on the use of ai technology in vulnerable communities as a means of preserving, in ai pipelines, minority perspectives that would otherwise be erased when annotator disagreement is resolved (usually in favor of the plurality label, as is common practice today). we believe that these methods, coupled with demographic information on annotators and reliable left to right, (grey) cnn is a baseline with no label modeling, which receives for each data item the empirical label distributions provided by annotators. (red) ds+cnn uses the same cnn model, but the empirical label distributions are replaced with labels from the ds model. mm+cnn uses the same cnn model, but the empirical label distributions are replaced with mm centroids. (yellow) cl is the crowdlayer baseline. wkk is the baseline from wan et al. (2023). (pink) @xmath6. (black) disco is the new method introduced here. ## conclusion we proposed a novel neural architecture, disco, for modeling the distribution of labels an item receives and the distribution that annotators provide in the presence of item-annotator pairs. our design was motivated by the desire to break free of the standard assumption (in supervised learning) of single-label ground truth. experimental results indicate that disco performs at a level comparable to state-of-the-art models that were purpose-built for label distribution learning, but with faster training time, and outperforms state-of-the-art annotatormodeling models, even on single-label learning problems. qualitative inspection of the data shows that the model can predict striking examples of annotator disagreement. future work will explore ways to more flexibly increase the labels of related items/annotators in order to enhance the sparse label sets.