Noise Correction on Subjective Datasets (2024)

Abstract: Incorporating every annotator’s perspective is crucial for unbiased data modeling. Annotator fatigue and changing opinions over time can distort dataset annotations. To combat this, we propose to learn a more accurate representation of diverse opinions by utilizing multitask learning in conjunction with loss-based label correction. We show that using our novel formulation, we can cleanly separate agreeing and disagreeing annotations. Furthermore, this method provides a controllable way to encourage or discourage disagreement. We demonstrate that this modification can improve prediction performance in a single or multi-annotator setting. Lastly, we show that this method remains robust to additional label noise that is applied to subjective data.

## introduction the practice of enhancing label accuracy in datasets through the use of multiple annotations per sample is found in works such as @xcite . this method capitalizes on the collective expertise of annotators to improve data quality. the reason is due to the assumption that individual annotators make mistakes and a collection of annotators can provide better-quality labels. traditional methods for reconciling differences are to use techniques such as majority voting, averaging, or expert opinions @xcite . these methods are designed to arrive at a single 'ground truth' for training supervised learning models. however, in tasks of a subjective nature where a single 'correct' answer may not exist, these practices can compromise the diverse perspectives of individual annotators. these issues are found across many domains such as in medical @xcite , social @xcite , and others @xcite . to improve the fair representation of annotators' opinions, datasets have been built to include indi-vidual annotations. these datasets contain labels that are not aggregated through majority voting. this enables the capturing of opinions of annotators who would otherwise have been removed due to the aggregation function. multitask learning is one way to model this data as demonstrated by mostafazadeh @xcite , where each task is to predict an annotator's individual label. however, these methods do not account for possible mistakes that individual annotators might make during the annotation process. datasets such as the gabhatecorpus by @xcite and goemotions by @xcite can contain thousands of annotations by a single annotator. mistakes such as misidentifying emotions due to annotation exhaustion would not be surprising. furthermore, annotation of a large number of samples may occur over multiple days and introduce temporal distribution shifts of opinion. we propose to address this problem by introducing loss-based label correction into a multitask learning setting. a fundamental property of lossbased label correction works by exploiting the network memorization effect by @xcite . they found that networks tended to learn simple patterns before learning more complex ones. this has been used extensively within the noisy learning community such as in @xcite . these methods first utilize the memorization effect to detect mislabeled instances. then they correct these possible sources of mislabeling using techniques such as incorporation of a network's own guess. motivated by this, we propose a novel formulation of multitask learning with label correction. however, a main challenge we try to overcome is that naive applications of these methods can erase the diverse perspectives of annotators. this is because the original technique makes use of a sample's loss to determine whether it is correctly or incorrectly labeled. on a dataset with subjective labels, we find that higher loss samples are asso-ciated with minority opinions which complicates the noisy sample detection process. we propose a novel method to address this by strengthening or weakening a model's guess of the true label. we find that our method is robust to added noise while also maintaining diverse opinions. we show that this is useful when modeling subjective datasets with differing agreement properties. we highlight our contributions as follows: ## methods we are given dataset denoted as d, comprising text samples x, and an annotation matrix @xmath0. here, x represents individual text entries, a signifies the number of annotators, and y is a matrix where each cell y i stores one hot value of a label class. due to the non-uniform distribution of annotator input across text instances, missing values in y are prevalent. our goal is to deduce a consensus label for each text instance. ## experimental setup we evaluate each of the four different optimization methods above in these experiments. ## results due to the nature of this problem, it is important to consider these results within the context of data collection processes. goemotions annotators were crowdworkers who spoke english from india. ghc annotation was a two-step process that involved expert opinions. the annotators of ghc were trained and the annotation process was much more controlled. furthermore, the annotator population was recruited from a smaller sample and consisted of undergraduate students. these processes would cause ghc to be less variable, due to likely similar backgrounds of annotators. on the other hand, using untrained crowdworkers from a foreign country would likely lead to more variability. with this in mind, we first discuss the comparison of performance metrics in evaluated methods. we then examine the effect of injecting additional noise. lastly, we examine the effect of adjusting the subjectivity parameter and its effect on results. ## discussion noisy-label tolerant techniques make the assumption that there is a single ground truth. many of these techniques, e.g. @xcite , utilize a label's loss to determine whether a sample is correctly labeled or not. that is higher loss is generally incorrectly labeled. however, when opinion is involved, we cannot naively do this. as we show in our results on figure 1 , the semantics of a samples' loss changes to majority or minority opinions. that is, higher loss is associated with minority opinions while lower loss is associated with majority ones. at a per-annotator level, there are still mistakes being made. thus we need to find a way to utilize these loss properties when learning representations. to do so, we propose a novel loss function (section 3.4) which models the predictions for each we believe the application of the manifold mixup to the training has some effects on this. this indicates a difference in the label patterns for subjectivity and noise. dataset metric ψ = 0.25 ψ = 0.5 ψ = 1 ghc anno. acc. 59.48 63.82 58.67 ghc pred. var. 14.38 13.59 13.01 goemo anno. acc. 49.92 50.11 49.85 goemo pred. var. 17.74 17.53 17.46 our experimental outcomes underscore the significance of adopting a balanced strategy for noise correction across varied datasets ( ## conclusion in this study, we developed a method for loss-based label correction for a multitask setting. specifically, this is applied to subjective datasets from multiple annotators. we show that our approach preserves disagreeing labels and that the performance increases under some conditions. we also demonstrated that it is robust to added noise. lastly, we demonstrate that adjusting the degree of loss correction is important and has a large impact on performance. this is beneficial for future work in collecting subjective labels at scale.