Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning (2021)

Abstract: Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human annotators, who often disagree. Prior work has shown that this disagreement can be helpful in training models. We propose a novel method to incorporate this disagreement as information: in addition to the standard error computation, we use soft-labels (i.e., probability distributions over the annotator labels) as an auxiliary task in a multi-task neural network. We measure the divergence between the predictions and the target soft-labels with several loss-functions and evaluate the models on various NLP tasks. We find that the soft-label prediction auxiliary task reduces the penalty for errors on ambiguous entities, and thereby mitigates overfitting. It significantly improves performance across tasks, beyond the standard approach and prior work.

## introduction usually, the labels used in nlp classification tasks are produced by sets of human annotators. as disagreement between annotators is common, many methods aggregate the different answers into a supposedly correct one @xcite @xcite @xcite . however, the aggregated labels obtained in this way mask the world's real complexity: instances can be intrinsically ambiguous @xcite @xcite , or so challenging to evaluate that considerable disagreement between different annotators is unavoidable. in those cases, it is reasonable to wonder whether the ambiguity is indeed harmful to the models or whether it carries valuable information about the relative difficulty of each instance @xcite . several authors followed that intuition, trying ways to incorporate the information about the level of annotator agree-ment in their models @xcite @xcite @xcite . usually, deep learning models compute the error as the divergence between the predicted label distribution and a one-hot encoded gold distribution (i.e., nothing but the gold label has any probability mass). however, for complex tasks, this binary black-and-white notion of truth is not plausible and can lead to overfitting. instead, we can use a more nuanced notion of truth by comparing against soft labels: we collect the probability distributions over the labels given by the annotators, rather than using one-hot encodings with a single correct label. to measure the divergence between probability distributions, we can use well-known measures like the kullback-leibler divergence @xcite , the jensen-shannon divergence @xcite , and the cross-entropy, which is also used to quantify the error with one-hot encoded labels. the main impediment to the direct use of soft labels as targets, though, is the lack of universally accepted performance metrics to evaluate the divergence between probability distributions. (most metrics lack an upper bound, making it difficult to assess prediction quality). usually, annotations are incorporated into the models without soft labels @xcite . where soft labels are used, they are variously filtered according to their distance from the correct labels and then used to weight the training instances rather than as prediction targets. these models still predict only true labels @xcite . in contrast to previous approaches, we use multi-task learning (mtl) to predict a probability distribution over the soft labels as additional output. we jointly model the main task of predicting standard gold labels and the novel auxiliary task of predicting the soft label distributions. due to the difficulty of interpreting its performance, we do not directly evaluate the distance between the target and the predicted probability distributions. however, the mtl framework allows us to indirectly evaluate its effect on the main task. exploiting the standard metrics for gold labels, we can also compare the effect of different loss functions for the soft label task. in particular, we propose a standard and an inverse version of the kl-divergence and cross-entropy. in previous work @xcite , filtering and weighting the training instances according to soft labels did not lead to consistent performance improvements. in contrast, we find that the information carried by mtl soft labels does significantly improve model performance on several nlp tasks. contributions 1) we show that mtl models, trained with soft labels, consistently outperform the corresponding single-task learning (stl) networks, and 2) we evaluate the use of different loss functions for soft labels. ## mtl with three loss functions for the experiments, we use different types of neural networks, depending on the type of task. however, we create two versions of each model architecture: an stl model and an mtl model. in stl, we predict the one-hot encoded labels. in mtl, we add the auxiliary task of predicting the soft label distributions to the previous main task. in both cases, we use adam optimization @xcite . the loss function for the main task is standard cross-entropy. for the auxiliary task, we have different options. the kldivergence is a natural choice to measure the difference between the prediction distribution q and the distribution of soft labels p . however, there are two ways we can do that, depending on what we want to capture.the standard kl-divergence is: this measures the divergence from q to p and encourages a wide q, because if the model overestimates the regions of small mass from p it will be heavily penalised. the inverse kl-divergence is: this measures the divergence from p to q and encourages a narrow q distribution because the model will try to allocate mass to q in all the places where p has mass; otherwise, it will get a strong penalty. considering that we use the auxiliary task to reduce overfitting on the main task, we expect equation 2 to be more effective because it encourages the model to learn a distribution that pays attention to the classes where the annotations possibly agree. a third option is to directly apply cross-entropy. this is actually derived from kl-divergence, the entropy of p added to the kl-divergence: therefore, regular kl-divergence and cross-entropy tend to lead to the same performance. for completeness, we report the results of cross-entropy as well. as overall loss of the main and of the auxiliary task, we compute the two's sum. we do not apply any normalization method to the two losses, as unnecessary. we use logsoftmax activation function for the main task, which is a standard choice for one-hot encoded labels, and standard softmax for the auxiliary task. against the distributions of gold (one-hot encoded) and soft labels, both summing up to one, the errors are on the same scale. we also derive the soft labels using the softmax function, which prevents the probability of the single labels from falling to zero. ## methods we evaluate our approach on two nlp tasks: pos tagging and morphological stemming. we use the respective data sets from @xcite and @xcite (where data sets are sufficiently large to train a neural model). in both cases, we use data sets where both one-hot (gold) and probabilistic (soft) labels (i.e., distributions over labels annotations) are available. the code for all models in this paper will be available on github.com/fornaciari. ## error analysis to gain further insights about their contributions, we inspect the soft labels' probability distributions, comparing the predictions of stl and mtl models. we perform the following analysis for the pos and the stemming tasks, and for each kind of loss function in the mtl models. in particular, we consider four-conditions of the predictions: 1) where both stl and mtl gave the correct answer, 2) where both gave the wrong answer, 3) where stl was correct and mtl incorrect, and 4) where mtl was correct and stl incorrect (see confusion matrix in not surprisingly, we find the highest average kurtosis where both stl and mtl models give the correct prediction. both kinds of models find it easier to predict the instances that are also unambiguous for the annotators. the opposite holds as well: the instances where both mtl and stl models are wrong show the lowest mean kurtosis. more interesting is the outcome where mtl models are correct and stl wrong, and vice-versa. in these cases, the average kurtosis lies between the two previous extremes. also, we find a consistent trend across data sets and mtl loss-functions: the instances where only the mtl models are correct show a slightly higher kurtosis than those instances where only the stl models give the right answer. to measure the significance of this trend, we apply the mann-whitney rank test @xcite . we use a non-parametric test because the kurtoses' distribution is not normal. we find two significant results: when we use cross-entropy as mtl loss-function in the pos data set, and with the kl inverse on the stemming data set. we report the pos results in in a broad sense, our study belongs to the research area of regularization methods for neural networks. among them, label smoothing @xcite penalizes the cases of over-confident network predictions. both label smoothing and soft labels reduce overfitting regulating the loss size. however, label smoothing relies on the gold labels' distribution, not accounting for the instances' inherent ambiguity, while soft labels selectively train the models to reduce the confidence when dealing with unclear cases, not affecting the prediction of clear cases. disagreement also relates to the issue of annotator biases @xcite @xcite , and our method can provide a possible way to address it. ## conclusion we propose a new method for leveraging instance ambiguity, as expressed by the probability distribution over label annotations. we set up mtl models to predict this label distribution as an auxiliary task in addition to the standard classification task. this setup allows us to incorporate uncertainty about the instances' class membership into the model. across two nlp tasks, three data sets, and three loss functions, we always find that our method significantly improves over the stl performance. while the performance difference between the loss functions is not significant, we find that the inverse version of kl gives the best results in all the experimental conditions but one. this finding supports our idea of emphasizing the coders' disagreement during training. we conjecture that predicting the soft labels acts as a regularizer, reducing overfitting. that effect is especially likely for ambiguous instances, where annotators' label distributions differ especially strongly from one-hot encoded gold labels.