CML : A Contrastive Meta Learning Method to Estimate Human Label Confidence Scores and Reduce Data Collection Cost (2022)

Abstract: Deep neural network models are especially susceptible to noise in annotated labels. In the real world, annotated data typically contains noise caused by a variety of factors such as task difficulty, annotator experience, and annotator bias. Label quality is critical for label validation tasks; however, correcting for noise by collecting more data is often costly. In this paper, we propose a contrastive meta-learning framework (CML) to address the challenges introduced by noisy annotated data, specifically in the context of natural language processing. CML combines contrastive and meta learning to improve the quality of text feature representations. Meta-learning is also used to generate confidence scores to assess label quality. We demonstrate that a model built on CML-filtered data outperforms a model built on clean data. Furthermore, we perform experiments on deidentified commercial voice assistant datasets and demonstrate that our model outperforms several SOTA approaches.

## introduction deep neural networks' remarkable capacity for representation learning has resulted in performance gains across a wide range of applications. the majority of these gains are dependent on having high-fidelity data; however, in practice, large-scale datasets are frequently corrupted by noise caused by a variety of factors such as task difficulty, annotator experience, and annotator bias. this is a concern because training data with corrupted labels can adversely affect the performance of deep learning models. collecting ground truth data to alleviate this problem, on the other hand, has proven both time consuming and costly. we can broadly enhance model performance on corrupted labels by applying two approaches: improving model robustness and improving quality of feature representation. a number of existing methods such as robust loss function based meth-ods @xcite @xcite , loss adjustment based methods @xcite @xcite , and sample selection based methods @xcite @xcite , have been proposed to enhance model robustness. in spite of these advances, these methods are primarily concerned with computer vision and lack quality assessment for feature representation. there are also several approaches to improving the quality of existing feature representations. they do not, however, specifically address the adverse effects of noisy (corrupted) labels in the context of natural language processing (nlp). @xcite proposed a contrastive learning-based approach for improving feature representation quality for computer vision (not nlp) applications through augmentation steps in the feature learning process. @xcite proposed a contrastive learning-based approach to improving the quality of sentence embedding. however, this approach does not address the problems caused by noise-corrupted labels. in this paper, we propose a contrastive metalearning (cml) framework for simultaneously addressing the challenges introduced by corrupted labels, in the context of nlp. importantly, our framework learns a confidence score for evaluating the quality of annotations. in the real world, the work of data annotators is typically evaluated against ground truth data. the term "ground truth data" refers to validated data labels that have been subjected to multiple passes by data annotators and labelled using majority vote. @xcite . we will hereafter refer to a dataset consisting of ground truth labels as a "gold" dataset, contrasting it with a "standard" dataset annotated by a single data associate. collecting ground truth data ("gold" datasets) is time consuming and expensive, and sometimes in-volves heavy engineering efforts @xcite . the confidence score generated by our model offers the potential to perform large-scale evaluations of annotation tasks. another application of the confidence score generated by our model is to select high-quality data from a noisy dataset. to address the issue of corrupted labels, we employ meta learning, which combines meta data and training data. our meta data is comprised of a relatively small number of ground truth labels. the training data consists of unverified annotations (i.e. corrupted labels). our model also learns an explicit loss-weight function while performing classification, which predicts label confidence scores. we utilized a meta learning approach @xcite as our meta module. the meta module is concatenated with the classifier. the meta module learns an explicit loss-weight function in a metalearning manner. we use the output loss of the classifier as the input to the meta module. the meta module learns confidence scores using a multilayer perceptron on the output loss of the classifier, which reflects the quality of labels and can also improve classifier performance. each of our training processes contains two steps. the first step consists of using our training data to update the parameters of the classifier, and the second step consists of using our meta data to update the parameters of the meta weight net. to improve the quality of feature representations, we apply contrastive learning to a pretrained bert model @xcite . contrastive learning aims to learn effective representations by pulling semantically close neighbors together and pushing apart non-neighbors. in order to have semantically close neighbors, the same sentence pass the pretrained encoder twice to predict the sentence itself with noise introduced by standard dropout layer. in such way, we have two embeddings generated from the same sentence but with slight difference. these two embeddings are "positive pair". as we mentioned above, current methods @xcite designed to address corrupted labels are mainly geared towards computer vision applications. in addition, their motivation is largely centered around label correction or improving the performance of a classifier. in contrast, in the nlp domain, researchers mainly focus on improving the quality of embeddings or learning representations of text data. our proposed framework unifies the advantages of current meth-ods and is suitable for nlp. we demonstrate that cml not only addresses concerns stemming from having corrupted labels, but also learns feature representations from raw text data effectively. additionally, in real-world applications, we are concerned with the quality of annotations. as such, evaluating the work of annotators presents an important challenge. during the training process, cml learns a confidence score for labels, which can be used to evaluate annotators' work online. in other words, cml offers the potential for scaling the work of data annotators. furthermore, because cml predicts a confidence score for labels, it can be applied to a wide range of use cases. in many instances only incorrectly labelled data is available and collecting ground truth data is time-consuming and costly. we can therefore use cml to filter high-quality data for such problems, saving both time and money. to summarize, the main contributions of our work are listed as follows: the remainder of this paper is organized as follows: we introduce related work in section 2. we then formalize the problem and present our proposed approach in section 3. next, we present applications and corresponding results of our experiments in section 4 and 5. finally, we present our conclusion and propose future research directions. ## related work in this section, we discuss some of the related work in contrastive learning and learning from corrupted labels. ## application the confidence score obtained from cml's output has an important application for data labeling services: measuring label quality at scale. in the industrial setting, the quality of each annotator's work is measured by ground truth reference, which is usually of limited quantity. small volumes of gold reference data could cause high variance in assessing annotator's performance. as such, it often requires complex procedures to find root cause of quality issue. error detection model is broadly used in industry but remains a challenge due to limited ground truth labels. the confidence score from cml is a promising attempt to solve the aforementioned challenges. in particular we implement the following two applications: ## evaluation this section shows the experiment results. we also compare our method to state-of-the-art methods. ## conclusion in this paper, we propose a contrastive meta learning framework (cml) for estimating human label confidence scores and lowering data collection costs. we use contrastive learning and meta learning to jointly address the main challenges of label scarcity and poor feature representation. we design three sets of experiments with two application settings and three state-of-the-art baseline models to test the effectiveness of our proposed method. our experiments on a commercial voice assistant gsr dataset show that our method can predict a reliable confidence score for annotations while also effectively lowering the cost of ground truth data collection. moreover, our proposed method outperforms several sota approaches.