L EARNING FROM I MPERFECT A NNOTATIONS : A N E ND - TO -E ND A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Many machine learning systems today are trained on large amounts of humanannotated data.
Annotation tasks that require a high level of competency make data acquisition expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases.
To improve data quality, practitioners often need to collect multiple annotations per example and aggregate them before training models.
Such a multi-stage approach results in redundant annotations and may often produce imperfect “ground truth” labels that limit the potential of training supervised machine learning models.
We propose a new end-to-end framework that enables us to: (i) merge the aggregation step with model training, thus allowing deep learning systems to learn to predict ground truth estimates directly from the available data, and (ii) model difﬁculties of examples and learn representations of the annotators that allow us to estimate and take into account their competencies.
Our approach is general and has many applications, including training more accurate models on crowdsourced data, ensemble learning, as well as classiﬁer accuracy estimation from unlabeled data.
We conduct an extensive experimental evaluation of our method on 5 crowdsourcing datasets of varied difﬁculty and show accuracy gains of up to 25% over the current state-of-the-art approaches for aggregating annotations, as well as signiﬁcant reductions in the required annotation redundancy. We further conduct an ablation study to evaluate the effect of both end-to-end learning and instance features and show that both contribute to the performance gains achieved by the proposed method.
I NTRODUCTION
The rising popularity and recent success of deep learning has resulted in machine learning systems that rely on large amounts of annotated training data (LeCun et al., 2015;
Wu et al., 2016; Gulshan et al., 2016; Esteva et al., 2017).
The most common, scalable way to collect such large amounts of training data is through crowdsourcing (Howe, 2006).
Crowdsourcing works well in simple settings where annotation tasks do not require domain expertise—for example, in object detection and recognition tasks in natural images and videos (e.g., Deng et al., 2009; Kovashka et al., 2016).
However, annotation in specialized domains such as medical pathology requires a certain level of competency and expertise from the annotators which makes annotation expensive.
Moreover, often times there is high rate of disagreement even between experts, which results in increasingly subjective and inconsistent labels (Elmore et al., 2015; Hutson et al., 2019).
A typical approach to dealing with subjectivity is to treat each annotation as simply noisy, collect multiple redundant labels per example (e.g., from different annotators), and then aggregate them using majority voting or other more advanced techniques (e.g., Dawid & Skene, 1979; Carpenter, 2008; Liu et al., 2012; Bachrach et al., 2012;
Zhou et al., 2015; Zhou & He, 2016) to obtain a single “ground truth” label.
At the expense of redundancy, this results in better data quality and more accurate estimates of the ground truth.
More recently, the emerging systems for data programming and weak supervision also internally rely on label aggregation techniques similar to methods used for solving the crowdsourcing problem.
Snorkel (Ratner et al., 2017; Bach et al., 2019) is a popular such system and was designed for efﬁcient and low-cost creation of large-scale labeled datasets using programmatically generated, so-called weak labels .
However, as we show in our empirical evaluation none of these systems solve label aggregation effectively in the presence of high subjectivity.
We argue that to become more effective, these methods need to make use of meta-data and other types of information that may be available about the data instances and the annotators labeling them.
To this end, we propose a novel approach that allows us to train accurate predictive models of the ground truth directly on the non-aggregated imperfectly labeled data .
Our method merges the two steps of: (i) aggregating subjective, weak, or noisy annotations, and (ii) training machine learning models.
At training time, along with learning a model that predicts the ground truth, we also learn models of the difﬁculty of each example and the competence of each annotator in a generalizable manner (i.e., these models can make predictions for previously unseen examples and annotators).
Our approach can be effectively used for training on crowdsourced data as well as on weakly labeled data, and also be used within frameworks such as Snorkel (Ratner et al., 2017; Bach et al., 2019) and signiﬁcantly improve their performance.
We propose a method that can:
Learn truth estimators: Learn functions representing the underlying ground truth, while imposing almost no constraints (as opposed to prior work).
In fact, we are able to leverage the capacity of deep neural networks along with the interpretability provided by Bayesian models, in order to obtain highly expressive estimators of the underlying truth.
Learn quality estimators: Learn functions that estimate the quality of each annotator.
When annotators can be described by some features (e
g., age, gender, location, etc. of an Amazon Mechanical Turk annotator, instead of just an ID), our quality estimators are able to generalize to new, previously unseen, predictors.
Previous work only considered estimating accuracies of a ﬁxed set of predictors, without being able to leverage any information we might have about them.
Furthermore, in contrast to previous work, we are also able to predict the per-instance predictor comptencies (i e., our method can determine whether a human annotator is an expert for a subset of queries, instead of just estimating his/her overall accuracy), which is done by learning dependencies between the instances and the annotators.
Finally, our approach is able to distinguish between multiple different types of errors by estimating the full confusion matrix for each instance-predictor pair.
Be easily extended: The truth and quality estimators can take arbitrary functional forms and fully leverage the expressivity of deep neural networks.
Both human annotators and machine learning classiﬁers may sometimes be unable to make predictions about certain aspects of the ground truth (e.g., human annotators may be unsure about what the correct answer to a question).
The proposed method is formulated in a way that allows it to be extended such that it can also learn decision function estimators for the annotators (i.e., estimators that predict whether an annotator will be able to provide a prediction for a given data instance).
These estimators can have signiﬁcant implications for data annotation systems where the cost of querying annotators is high (e.g., when these annotators are highly qualiﬁed, such as doctors or other kinds of domain experts).
This is because it allows for better matching annotators to instances, thus reducing the required amount of annotation redundancy.
An overview of the proposed approach and model is shown in Figure 1, and a detailed description is provided in Section 3.
R ELATED W ORK
Research on label aggregation and crowdsourcing dates back to the early 1970s, when Dawid &
Skene (1979) proposed a probabilistic model to estimate ground truth labels using the expectation maximization (EM) algorithm.
Since then, a variety of generalizations and improvements upon the original method have been proposed (Whitehill et al., 2009; Welinder et al., 2010; Liu et al., 2012;
Zhou et al., 2015; Zhou & He, 2016).
One of the central parts of the label aggregation algorithms is estimation of the accuracy of the annotators (or predictors) without having access to the ground truth.
This problem has been of independent interest to the machine learning community, was termed as estimating accuracy from unlabeled data and studied by Collins & Singer (1999), Dasgupta et al.
(2001), Bengio & Chapados (2003), Madani et al.
Schuurmans et al.
(2006), Balcan et al.
(2013), and Parisi et al.
(2014), among others.
Almost none of the previous approaches explicitly considered modeling the ground truth, but rather assumed either some form of independence or knowledge of the true label distribution.
Collins & Huynh (2014) reviewed many methods that were proposed for estimating the accuracy of medical tests in the absence of a gold standard.
Platanios et al.
(2014) proposed formulating the
[Figure]
E- [Figure]
Compute the expected ground truth labels based on the current ground truth and predictor quality models.
M-Step
Learn the ground truth and predictor quality models based on the current expected ground truth labels.
Alternate between the following two steps un�l convergence:
ObservedComputedLatent INSTANCE
[Figure]
Legend
Predictors
DIFFICULTY
QUALITY
COMPETENCE
TRUELABEL
OBSERVATION
[Figure]
Is there a bone fracture in this image?
[Figure]
Es�mate the ground truth labels based on a majority vote among the annotators.
[Figure]
problem as an optimization problem that uses agreement rates between multiple noisy annotators. Platanios et al. (2017) improved upon agreement-based accuracy estimation using logical constraints between the noisy labels.
Tian & Zhu (2015) proposed a max-margin majority voting scheme applied to crowdsourcing.
More recently, Khetan et al.
(2017) proposed to use a parametric function to model the ground truth and showed that the approach can sometimes be functional even in the limit of a single noisy label per example.
Among recent approaches, Zhou et al.
(2015) formulated the problem as a form of regularized minimax conditional entropy and established one of the most competitive baselines on many public crowdsourcing datasets.
Our proposed method generalizes the approaches of Zhou et al.
, Platanios et al.
(2016), and Khetan et al.
Similar to Platanios et al.
, we deﬁne a generative process for our observations.
However, our model is able to handle categorical labels, as opposed to just binary.
Similar to Zhou et al.
, we deﬁne the confusion matrix for each instance-predictor pair as a function of instance difﬁculty and predictor competence.
However, we explicitly learn the difﬁculty and competence as functions, which allows us to generalize to previously unseen instances and annotators.
Interestingly, the inference algorithm for our generative probabilistic model has a similar form to that of Zhou et al.
(2015) (except for the explicit learning of the ground truth, difﬁculty, and competence functions).
Thus, we also show that the algorithm of Zhou et al.
(2015) can be re-derived as an EM inference algorithm for a generative model, simplifying the argument of the original paper.
Finally, similar to Khetan et al.
(2017), we use a parametric function to model the ground truth, and also go a step further and propose to use parametric functions to model the instance difﬁculties and predictor competences.
Thus, our approach enables estimation of which annotators are likely to perform better on which instances, potentially enabling more optimal allocation of annotators and thus annotation cost reductions.
In contrast to prior work, our method also allows for end-to-end learning .
Prior methods do not allow for this as they (implicitly) separate ground truth inference (i.e., label aggregation) from model training.
More speciﬁcally, previously one would have to train a machine learning model in two stages: (i) infer the ground truth labels from the provided annotations, and (ii) train a machine learning model on the inferred labels.
Our approach merges these two stages and allows us to train machine learning models directly on the imperfect annotations.
In Section 4.3, we conduct an ablation study that showcases the performance gains obtained by employing end-to-end learning in this fashion.
P ROPOSED M ETHOD
We denote the observed data by D = { x i , ˆ Y i } Ni =1 , where ˆ Y i =
{M i , { ˆ y ij } j ∈M i } , M i is the set of predictors that made predictions for instance x i , and ˆ y ij is the output of predictor ˆ f j for instance x i .
Our goal is to learn functions representing the underlying ground truth and predictor qualities, given our observations D .
Ground Truth.
We deﬁne the ground truth as a function h θ ( x i ) that is parameterized by θ and that approximates the true distribution of the label given x i .
In our setting, h θ ( x i ) ∈ R C ≥ 0 and (cid:80) j
[ h θ ( x i )]
j = 1 , where C is number of values the label can take (i.e., assuming categorical labels).
More speciﬁcally, [ h θ ( x i )]
k (cid:44) P ( y i = k
| x i ) , where we use square brackets and subscripts to denote indexing of vectors, matrices, and tensors.
For example, h θ could be a deep neural network that would normally be trained in isolation using the cross-entropy loss function.
In our method the network is trained using the Expectation-Maximization algorithm, as described in the next section.
Predictor Qualities.
We deﬁne the predictor qualities as the confusion matrices Q ij ∈ R C
×
C ≥ 0 , for each instance x i and predictor ˆ f j , where (cid:80) l [ Q ij ] kl = 1 , for all k ∈ { 1 , . . .
, C } .
[ Q ij ] kl represents the probability that predictor ˆ f j outputs label l given that the true label of instance x i is k .
We deﬁne these confusion matrix in a way that generalizes the successful approach of Zhou et al.
[Equation]
where • i represents an inner product along the i th dimension of the two tensors, and:
– D i =
d φ ( x i ) represents the difﬁculty tensor for instance x i , where d is a function parameterized by φ , D i ∈ R C × C × L , and L is a latent dimension (it is a hyperparameter of our model).
[ D i ] kl − is an L -dimensional embedding representing the likelihood of confusing x i as having label l instead of k , when k is its true label.
– C j = c ψ ( r j ) represents the competence tensor for predictor ˆ f j , where c is a function parameterized by ψ , r j is some representation of ˆ f j (e.g., could be a one-hot encoding of the predictor, in the simplest case), and C j ∈ R C × C × L .
[ C j ] kl − is an L -dimensional embedding representing the likelihood that predictor ˆ f j confuses label k for l , when k is the true label.
Using L > 1 allows the instance difﬁculties and predictor competences to encode more information.
An intuitive way to think about this is that we are embedding difﬁculties and competencies in a common latent space, which can be thought of as jointly clustering them.
This is in fact very similar to how matrix factorization methods are used for collaborative ﬁltering in recommender systems.
Our goal is to learn functions h θ , d φ , and c ψ , given observations D .
In order to do that, we propose the following generative process for our observations.
For i = 1 , . . .
, N , we ﬁrst sample the true label for x i , y i ∼ Categorical( h θ ( x i )) .
Then, for j ∈ M i , we sample the predictor output
ˆ y ij ∼ Categorical
([ Q ij ] y i − ) , where [ Q ij ] y i − represents the y i th row of Q ij .
In the next section, we propose an algorithm learning the parameters θ , φ , and ψ .
3. L EARNING
A widespread approach for performing learning with probabilistic generative models, is to maximize the likelihood of the observed data with respect to the model parameters.
Let y =
{ y i } Ni =1 .
The likelihood of a single observation, ˆ y ij , can be derived as follows:
[Equation]
where p (ˆ y ij | y i ) depends on Q ij .
There are two main approaches in which we can maximize the likelihood function of Equation 2: (i) marginalize out the y
i latent variables and then maximize with respect to θ , φ , and ψ , or (ii) use the expectation maximization (EM) algorithm originally proposed by Dempster et al.
It has previously been observed that the EM algorithm can perform much better than approach (i) for mixture models (Bishop, 2006).
This is because the latter tends to get stuck in bad local optima.
Since our model resembles a Bernoulli mixture model with the latent assignments being deﬁned by the y i s, we decided to use the EM algorithm.
3 We derive the EM algorithm steps for our model, as follows:
E-Step.
We need to compute the expectation of y i given D and y \ i (which denotes all of y except for y i ), for all i = 1 , . . .
, N , and we know that:
[Equation]
Therefore, by removing all terms that do not depend on y i and normalizing, we obtain the following expectation (which we compute during this step, while keeping θ , φ , and ψ ﬁxed):
[Equation]
For brevity, in what follows, we use the following notation ˜ y
ki =
E y |D {
[ y i = k ] } .
M-Step.
We maximize the following log-likelihood function with respect to θ , φ , and ψ , while using the values of ˜ y ki computed in the last E-step:
[Equation]
The training procedure for learning the parameters θ , φ , and ψ consists of iterating over the E-step and the M-step shown above, until convergence, where convergence can be measured by computing the change in the parameter values across learning iterations.
It is important to note that EM ﬁnds local optima of the likelihood function, and so the starting point can play an important role.
Also, as Platanios et al.
(2016) mention, there exists an inherent symmetry in our model that can be problematic.
The likelihood of any observed data is the same if we ﬂip the true underlying labels and the predictor qualities (i.e., set y ﬂipped i
[Equation]
We would like to somehow encode the prior assumption that most of the predictors are correct most of the time.
One way to do this is by choosing the starting point of the EM algorithm carefully.
Initialization.
In the E-step shown in Equation 4, we compute the expected values of the true underlying labels, y i .
We can encode the assumption mentioned in the previous paragraph by replacing the ﬁrst E-step with a majority vote among the predictors:
[Equation]
where |M i | denotes the size of set M i .
We initialize the EM algorithm by replacing the ﬁrst E-step with this majority vote approximation.
As we show in our experiments, this helps us avoid the aforementioned symmetry, and thus we refer to this initialization scheme as symmetry-breaking initialization .
Note also that in the case where the predictors provide us with P (ˆ y ij = k ) , instead of a single categorical value, we can still use this initialization scheme by replacing 1 [ˆ y ij = k ] with P (ˆ y ij = k ) , in Equation 8.
Marginal Likelihood Fine-Tuning.
In our experiments we found that maximizing the marginal likelihood function after EM converges tends to improve performance.
We refer to this step as marginal likelihood ﬁne-tuning .
More speciﬁcally, after the values of the parameters θ , φ , and ψ , converge to ﬁxed values across multiple EM steps, we solve the following maximization problem using these ﬁxed values as the initial point:
[Equation]
3. I NSTANCE AND P REDICTOR R EPRESENTATIONS
A major advantage of the proposed approach over prior work is that we learn models of the ground truth and the predictor qualities as functions of some representations (i.e., representations of the data instances, x i , and of the annotators, r j ).
It is thus important to deﬁne these representations.
For many problems, the representations of the data instances can be deﬁned in the same manner as was previously done when performing supervised learning (e.g., we can directly use raw pixel values representing images).
However, predictor representations are introduced here for the ﬁrst time.
A simple approach would be to use a one-hot encoding of the predictors.
However, this would not allow for any amount of information sharing across predictor (e.g., what if two predictors are very similar).
We know from prior work that modeling dependencies between the predictors can be very important (e.g., Platanios et al., 2016).
One way to allow for that is to learn vector embedding representations for the predictors, which would be implicitly equivalent to clustering them.
Ideally, one would want to use any available information about these predictors (e.g., Amazon Mechanical Turk annotators could be described by their age, location, etc.).
Unfortunately, we could not ﬁnd any public datasets that provide such information about the predictors/annotators, and hence we used embedding representations in our empirical study.
3. D ISCUSSION
The approach we have proposed in this section can be thought of as introducing a new loss function for training the model h θ using multiple imperfect labels per training instance, each coming from a different sources.
This new loss function introduces latent variables representing the ground truth labels, as well as a couple of auxiliary models that are learned, and which represent the instance difﬁculties and predictor competences.
We also proposed an EM-based algorithm to minimize this new loss function as well as an initialization scheme.
Perhaps most interestingly, a key difference between this approach and previous work is that we are able to explicitly learn functions that output the likelihood that a predictor will label a speciﬁc instance correctly.
This enables using this approach to perform crowdsourcing more actively by assigning annotators to instances they are more likely to label correctly, thus helping reduce redundancy and drive costs down.
3. E XTENDING TO M ULTI -L ABEL S ETTINGS
Our method can easily be extended to handle settings where we have multiple categorical labels that can be assigned to each instance.
In that case, the model per label is deﬁned in the same way as previously, except that now the functions h θ , d φ , and c ψ also take as input a representation for the label (e.g., a label embedding).
This allows us to share information across labels and can be thought of as a generalization of the approach by Platanios et al.
(2016), where information is shared by clustering the labels.
Furthermore, it allows us to use the proposed method in extreme classiﬁcation settings (e.g., Prabhu & Varma, 2014) or settings where the number of labels is not ﬁxed and known a priori and can keep increasing (e.g., face recognition; Weinberger & Saul, 2009; Liu et al., 2016).
This is made possible by learning label representations and then letting the difﬁculty and competence functions also take as input a pairs of labels and return a vector instead of a three-dimensional tensor.
In the next section, we show how learning label representations can signiﬁcantly enhance the robustness and performance of our approach. e g Grover et al., [Table]
E XPERIMENTS
We evaluate the proposed approach on multiple datasets from the crowdsourcing domain, all of which have ground truth labels, (multiple) subjective annotations for each example, as well as information on who provided each annotation (i [Table] e., the annotator ID):
Blue Birds (BB) (Welinder et al., 2010): Bird photos labeled as Indigo Bunting or Blue Grosbeak .
Word Similarity (WS) (Snow et al., 2008):
Pairs of words labeled as similar or dissimilar.
RTE (Snow et al., 2008): Pairs of sentences labeled as whether the ﬁrst entails the second.
Medical Causes (MC) (Dumitrache et al., 2018): Sentences that contain 2 medical terms labeled if one of the terms causes the other (e.g., pancreatic adenocarcinoma causes weight loss ).
Medical Treats (MT) (Dumitrache et al., 2018):
The same sentences with medical terms labeled if one of the terms treats the other (e.g., aspirin treats pain
The last two datasets are in fact part of a single dataset on medical relations, and thus we are able to perform experiments both with the single task formulation of our algorithm and the multi-task formulation.
As we discuss at the end of this section, this allows us to show how our approach can be used to share information across labels and improve the quality of the learned models.
Statistics for these datasets are provided in Table 1.
Note that these datasets were provided without the associated features of the annotator identiﬁers and thus we are unable to evaluate the usefulness of annotator features (we instead learn embeddings for the annotators). Unfortunately we were unable to obtain any crowdsourcing datasets with associated annotator meta-data. This is probably due to the fact that no prior method is able to make use of such information. However, we do make use of instance features for all of these datasets.
In cases where such features were not readily available, we manually computed them by using pre-trained machine learning models.
We are making all such features and annotator identiﬁcation information publicly available in a standardized format.
More details are provided in our code and data repository which is available at http://anonymous .
4. E XPERIMENTAL S ETUP
We perform experiments using the following two variants of our approach:
– LIA : A version of our method which uses instance and predictor features speciﬁc to each dataset.
When features are not available for the instances and/or the predictors we learn embeddings of size 16 which are initialized randomly and optimized along with the other model parameters during the M-step (see Section 3.1).
– LIA-ML : A multi-label variant of the aforementioned method.
This method is only used with the medical relations datasets.
In this case, we consider all 14 medical relations included in the dataset jointly and only evaluate on the two for which the ground truth is provided (i.e., “causes” and “treats”).
We use this method variant in order to show how our approach can effectively share information across labels.
In both instances of LIA , h θ and d φ are multi-layer perceptrons (MLPs) with 4 layers of 16 hidden units each, with the only exception being the medical relations dataset where we used 32 units for each layer. c ψ is always modeled as a linear function. Note that for both the embedding sizes and the MLP sizes, we did not perform an extensive search to choose these values; we rather performed a small grid search and selected the number that resulted in the highest validation data likelihood.
We compare against the following baselines for ground truth estimation:
– MAJ : Simple majority voting.
Note that we use soft majority voting whenever possible, i.e., we use soft labels (probabilities or conﬁdence scores) whenever the predictors provide them, instead of always thresholding them to obtain discrete labels.
– MMCE : Regularized minimax conditional entropy by Zhou et al. (2015), which has been shown to outperform alternatives.
We consider it the current state-of-the-art for crowdsourcing.
– Snorkel : A method originally designed for aggregating annotations of programmatic weak predictors proposed by Ratner et al.
(2017), which is part of a popular software package that allows for subsequent training of machine learning models on the aggregated data.
– MeTaL : Successor to Snorkel, proposed by Ratner et al.
For both this method and for Snorkel we use the original implementation provided by the authors.
Aside from these baselines, we also perform experiments using the following custom methods that we designed for the purposes of performing an ablation study (the study is presented in Section 4.3): – LIA-E : In order to evaluate the usefulness of instance features, we learn embeddings of sizefor the instances instead of using their features.
– MAJ (cid:63) -E : A two-step method that resembles how machine learning models are currently being trained when using crowdsourced data. First, we estimate ground truth using MAJ . Next, we train the h θ model used in LIA-E directly on the aggregated labels. – MAJ (cid:63) :
Same as MAJ (cid:63) -E , except that we use the model h θ of LIA (i e., making use of instance features instead of learning instance embeddings). – MMCE (cid:63) -E : Same as MAJ (cid:63) -E , but with MMCE used for label aggregation.
– MMCE (cid:63) : Same as MAJ (cid:63) , but with MMCE used for label aggregation.
During each M-step we use the AMSGrad optimizer (Reddi et al., 2018) to maximize the loglikelihood function with the learning rate set to 0.001, and we perform 1,000 optimization iterations using a batch size of 1,024.
Overall, we perform 10 EM iterations (all models did converge within that limit) with warm starting (i.e., the model parameters are always initialized to the values obtained during the previous M-step).
When using LIA with image instances we use as image features the activations of the last layer of a pre-trained ResNet-101 Convolutional Neural Network (CNN).
Similarly, for all text instances we use as text features the representations provided by a pre-trained
BERT model (Devlin et al., 2018).
More details on our setup and the model hyperparameters can be found in our code repository at http://anonymous .
We evaluate all methods by computing the accuracy of the predicted instance labels.
This is a common metric for evaluating crowdsourcing methods and it also implicitly measures the quality of the confusion matrices predicted by our model.
This is because these confusion matrices heavily inﬂuence the supervision provided to the ground truth model, h θ , while training.
Furthermore, instead of just computing accuracy for the full datasets, we also measure how performance varies as a function of redundancy —the maximum number of annotations provided per instance.
In order to limit the redundancy for existing datasets we randomly sample subsets of the provided annotations.
Performing well in low redundancy settings is very important because it can result in signiﬁcantly reduced crowdsourcing costs.
4. R ESULTS
Our results are presented in Table 2.
LIA methods consistently outperform alternative approaches.
In certain cases (e.g., in Blue Birds ) we are able to boost accuracy over the best alternative method by 14%, thus establishing a new state-of-the-art for this dataset.
In the multi-task setting, where we train the LIA-ML model to jointly infer ground truth for both Medical Causes (MC) and Medical Treats (MT) while sharing the representations of instance difﬁculties and annotator competencies.
We observe that multi-task training boosts performance by more 8% absolute (or over 20% relative) over the single task counterpart, outperforming the baselines by over 25% relative.
Finally, our approach can obtain the performance of the best alternative method using up to 4 times less redundancy, which can have signiﬁcant implications for the cost of crowdsourcing, especially when annotation requires domain expertise (e.g., in healthcare).
We note that Snorkel and MeTaL tend to perform well overall, but sometimes fail entirely (often performing on par with or worse than majority voting). 7 MeTaL [Table]
also suffers from calibration issues [Table] as it often achieves very low accuracy while having reasonable mean average precision.
Data programming systems could thus beneﬁt signiﬁcantly by integrating our method in their pipeline, tying together the label aggregation and model training phases [Table] 4. A BLATION S TUDY [Table]
Our main contributions are: (i) end-to-end learning by fusing the label aggregation and model training phases, and (ii) allowing for instance and annotator features to inform label aggregation.
In this section, we show how each one of these contributions is important on its own by performing experiments where we introduce each one on their own, while keeping everything else constant. End-to-End Learning. The best way to test the effectiveness of end-to-end learning is to compare end-to-end approaches with two-stage approaches where: (i) we ﬁrst aggregate labels, and (ii) we then train machine learning models using the aggregated labels. To this end, we introduced the baseline methods marked with a “ (cid:63) ” in Table 2.
The results indicate that the two-stage approach underperforms the base label aggregation method for both MAJ and MMCE .
This is most likely due to the fact that in both these cases the model being learned cannot inform the label aggregation stage.
In contrast, LIA is able to outperform all two-stage approaches because it allows for exactly that.
Note that MMCE models instance difﬁculty and annotator competence similar to LIA , with the exception that it does not use instance features and it does not allow for endto-end learning of the ground truth predictor.
Also note that LIA-E does not use instance features, but is still able to outperform MAJ (cid:63) -E and MMCE (cid:63) -E in many cases, indicating that end-to-end learning is, in fact, effective and accounts for at least part of the performance gains achieved by LIA .
Instance Features.
The methods which use instance features are LIA , MAJ (cid:63) -M , and MMCE (cid:63) -M .
To test for the usefulness of these features, we provide variants of these methods (labeled LIA-E ,
MAJ (cid:63) -E , and MMCE (cid:63) -E , respectively, in Table 2) that use indicator features instead and learn instance embeddings.
We observe that for MAJ and MMCE the results are inconclusive (the feature-based methods outperform the alternative in around half of the experiments).
However, in the context of end-to-end learning, we observe that LIA consistently outperforms LIA-E by a signiﬁcant margin.
This indicates that instance features are indeed useful, especially so in the context of end-to-end learning where they can inform the label aggregation phase.
It is important to also mention that results pertinent to this ablation study for the Word Similarity (WS) dataset were a bit unstable with many models effectively failing to learn anything meaningful (speciﬁcally MMCE (cid:63) -E , MMCE (cid:63) -M , and LIA-E ).
Our best explanation for this is that the Word
Similarity (WS) dataset is very small with only 30 instances and is thus highly prone to overﬁtting.
Finally, in order to evaluate the effect of the marginal likelihood ﬁne-tuning approach presented in
Section 3.1, we also run experiments using LIA without this ﬁne-tuning phase.
The results are shown in Table 3 and it is clear that marginal likelihood ﬁne-tuning results in better performance.
Also, in order to sanity check that LIA is able to predict the qualities of the predictors accurately, we also performed a synthetic experiment.
More speciﬁcally, we added an “always correct” and an “always wrong” oracle to all datasets used in our experiments.
It turns out the predicted qualities for the two oracles are the highest and lowest among all predictors, respectively.
This indicates that our model is indeed capable of uncovering such highly competent and incompetent predictors.
C ONCLUSION
In this paper, we have introduced a learning framework for: (i) training deep models directly on data with imperfect annotations, and (ii) modeling the processes that produced the labels.
Our approach improves upon the classical and widely used two-stage setup (ﬁrst aggregate and denoise the labels, then train the model) by merging the two stages.
As a result, we are able to train models end-to-end using multiple noisy labels, while estimating the difﬁculties of the examples and learning accurate representations for the annotators that produced the labels.
Experimental results on multiple small and large scale publicly available crowdsourcing datasets indicate that our method results in signiﬁcant gains in accuracy (up to 25% relative gain over the current state-of-the-art approaches for aggregating noisy labels).
Moreover, it turns out that training the model to predict multiple related labels simultaneously improves the learned representations and results in further gains in predictive performance of the model.
Finally, we performed an ablation study to evaluate the effect of both end-to-end learning and instance features and showed that both contribute to the performance gains achieved by the proposed method.