M AX -MIG: AN I NFORMATION T HEORETIC A PPROACH FOR J OINT L EARNING FROM C ROWDS
A BSTRACT
Eliciting labels from crowds is a potential way to obtain large labeled data.
Despite a variety of methods developed for learning from crowds, a key challenge remains unsolved: learning from crowds without knowing the information structure among the crowds a priori, when some people of the crowds make highly correlated mistakes and some of them label effortlessly (e.g. randomly) .
We propose an information theoretic approach, Max-MIG, for joint learning from crowds, with a common assumption: the crowdsourced labels and the data are independent conditioning on the ground truth.
Max-MIG simultaneously aggregates the crowdsourced labels and learns an accurate data classiﬁer.
Furthermore, we devise an accurate data-crowds forecaster that employs both the data and the crowdsourced labels to forecast the ground truth.
To the best of our knowledge, this is the ﬁrst algorithm that solves the aforementioned challenge of learning from crowds.
In addition to the theoretical validation, we also empirically show that our algorithm achieves the new state-of-the-art results in most settings, including the real-world data, and is the ﬁrst algorithm that is robust to various information structures.
I NTRODUCTION
Lack of large labeled data is a notorious bottleneck of the data-driven-based machine learning paradigm.
Crowdsourcing provides a potential solution to this challenge: eliciting labels from crowds.
However, the elicited labels are usually very noisy, especially for some difﬁcult tasks (e.g. age estimation, medical images annotation).
In the crowdsourcing-learning scenario, two problems are raised:
(i) how to aggregate and infer the ground truth from the imperfect crowdsourced labels?
(ii) how to learn an accurate data classiﬁer with the imperfect crowdsourced labels?
One conventional solution to the two problems is aggregating the crowdsourced labels using majority vote and then learning a data classiﬁer with the majority answer.
However, this naive method will cause biased results when the task is difﬁcult and the majority of the crowds label randomly or always label a particular class (say class 1) effortlessly.
Another typical solution is aggregating the crowdsourced labels in a more clever way, like spectral method (Dalvi et al., 2013; Zhang et al., 2014), and then learning with the aggregated results.
This method avoids the above ﬂaw that the majority vote method has, as long as their randomnesses are ∗ Equal Contribution. mutually independent.
However, the spectral method requires that the experts’ labeling noise are mutually independent, which often does not hold in practice since some experts may make highly correlated mistakes (see Figure 2 for example).
Moreover, the above solutions aim to train an accurate data classiﬁer and do not provide a method that can employ both the data and the crowdsourced labels to forecast the ground truth.
A common assumption in the learning from crowds literature is that conditioning on the ground truth, the crowdsourced labels and the data are independent, as shown in Figure 1 (a).
Under this assumption, the crowdsourced labels correlate with the data due to and only due to the ground truth.
Thus, this assumption tells us the ground truth is the “information intersection” between the crowdsourced labels and the data.
This “information intersection” assumption does not restrict the information structure among the crowds i.e. this assumption still holds even if some people of the crowds make highly correlated mistakes.
We present several possible information structures under the “information intersection” assumption in
Figure 1 (b).
The majority vote will lead to inaccurate results in all cases if the experts have different levels of expertise and will induce extremely biased results in case (2) when a large number of junior experts always label class 1.
The approaches that require the experts to make independent mistakes will lead to biased results in case (3), when the experts make highly correlated mistakes
In this paper, we propose an information theoretic approach, Max-MIG, for joint learning from crowds, with a common assumption: the crowdsourced labels and the data are independent conditioning on the ground truth.
To the best of our knowledge, this is the ﬁrst algorithm that is both theoretically and empirically robust to the situation where some experts make highly correlated mistakes and some experts label effortlessly, without knowing the information structure among the experts.
Our algorithm simultaneously aggregates the crowdsourced labels and learns an accurate data classiﬁer.
In addition, we propose a method to learn an accurate data-crowds forecaster that can employ both the data and the crowdsourced labels.
At a high level, our algorithm trains a data classiﬁer and a crowds aggregator simultaneously to maximize their “mutual information”.
This process will ﬁnd the “information intersection” between the data and crowdsourced labels i.e. the ground truth labels.
The data-crowds forecaster can be easily constructed from the trained data classiﬁer and the trained crowds aggregator.
This algorithm allows the conditional dependency among the experts as long as the intersection assumption holds.
We design the crowds aggregator as the “weighted average” of the experts.
This simple “weighted average” form allows our algorithm to be both highly efﬁcient in computing and theoretically robust to a large family of information structures (e.g. case (1), (2), (3) in Figure 1 (b)).
Particularly, our algorithm works when there exists a subset of senior experts, whose identities are unknown , such that these senior experts have mutually independent labeling biases and it is sufﬁcient to only use the seniors’ information to predict the ground truth label.
For other junior experts, they are allowed to have any dependency structure among themselves or between them and the senior experts.
R ELATED WORK
A series of works consider the learning from crowds problem and mix the learning process and the aggregation process together.
Raykar et al.
(2010) reduce the learning from crowds problem to a maximum likelihood estimation (MLE) problem and implement an EM algorithm to jointly learn the expertise of different experts and the parameters of a logistic regression classiﬁer.
Albarqouni et al.
(2016) extend this method to combine with the deep learning model.
Khetan et al.
(2017) also reduce the learning problem to MLE and assume that the optimal classiﬁer gives the ground truth labels and the experts make independent mistakes conditioning on the ground truth.
Unlike our method, these MLE based algorithms are not robust to correlated mistakes.
Recently, Guan et al.
(2017) and
Rodrigues & Pereira (2017) propose methods that model multiple experts individually and explicitly in a neural network.
However, their works lack theoretical guarantees and are outperformed by our method in the experiments, especially in the naive majority case.
Moreover, unlike our method, their methods cannot be used to employ both the data and the crowdsourced labels to forecast the ground truth.
Several works focus on modeling the experts.
Whitehill et al.
(2009) model both expert competence and image difﬁculty, but did not consider expert bias.
Welinder et al.
(2010) model each expert as a multidimensional classiﬁer in an abstract feature space and consider both the bias of the expert and the difﬁculty of the image.
Rodrigues et al.
(2014) model the crowds by a Gaussian process.
Khetan & Oh (2016); Shah et al.
(2016) consider the generalized Dawid-Skene model (Dawid & Skene, 1979) which involves the task difﬁculty.
However, these works are still not robust to correlated mistakes.
We model the crowds via the original Dawid-Skene model and do not consider the task difﬁculty, but we believe our Max-MIG framework can be incorporated with any model of the experts and allow correlated mistakes.
Our method differs from the works that focus on inferring ground truth answers from the crowds’ reports and then learn the classiﬁer with the inferred ground truth (e.g. (Dawid & Skene, 1979;
Zhou et al., 2012; Liu et al., 2012; Karger et al., 2014; Zhang et al., 2014;
Dalvi et al., 2013; Ratner et al., 2016)) since our method simultaneously infers the ground truth and learns the classiﬁer.
In addition, our method provides a data-crowds forecaster while those works do not.
Our method is also closely related to co-training.
Blum & Mitchell (1998)
ﬁrst propose the co-training framework: simultaneously training two classiﬁers to aggregate two views of data.
Our method interprets joint learning from crowds as a co-training style problem.
Most traditional co-training methods require weakly good classiﬁer candidates (e.g. better than random guessing).
We follow the general information theoretic framework proposed by Kong & Schoenebeck (2018) that does not have this requirement.
However, Kong & Schoenebeck (2018) only provide theoretic framework and assume an extremely high model complexity without considering the over-ﬁtting issue, which is a too strong assumption for practice.
Our work apply this framework to the learning from crowds problem and provide the proper design for the model complexity as well as the experimental validations.
M ETHOD
In this section, we formally deﬁne the problem, introduce our method, Max-MIG, and provide a theoretical validation for our method.
Notations For every set A , we use ∆ A to denote the set of all possible distributions over A .
For every integer M , we use [ M ] to denote { 1 , 2 ,...,M } .
For every matrix A
=
( A i,j ) i,j ∈ R + s × t , we deﬁne log A as a s × t matrix such that its the ( i,j ) th entry is log ( A i,j ) .
Similarly for every vector v = ( v i ) i ∈ R + s , we deﬁne log v as a vector such that its the i th entry is log ( v i ) .
Problem statement There are N datapoints.
Each datapoint x ∈ I (e.g. the CT scan of a lung nodule) is labeled by M experts y
[ M ] ∶= { y 1 ,y 2 ,...,
y M ∣ y m ∈ C} (e.g. C = { benign , malignant } ,experts’ labels: { benign, malignant, benign, benign, benign } ).
The datapoint x and the crowdsourced labels y
[ M ] are related to a ground truth y ∈ C
(e.g. the pathological truth of the lung nodule).
We are aiming to simultaneously train a data classiﬁer h and a crowds aggregator g such that h ∶ I ↦ ∆ C predicts the ground truth y based on the datapoint x ∈ I , and g ∶ C M
→ ∆ C aggregates M crowdsourced labels y
[ M ] into a prediction for ground truth y .
We also want to learn a data-crowds forecaster ζ ∶ I × C M ↦ ∆ C that forecasts the ground truth y based on both the datapoint x ∈ I and the crowdsourced labels y
[ M ] ∈ C .
3. M AX -MIG: AN INFORMATION THEORETIC APPROACH
Figure 3 illustrates the overview idea of our method.
Here we formally introduce the building blocks of our method.
Data classiﬁer h
The data classiﬁer h is a neural network with parameters Θ .
Its input is a datapoint x
and its output is a distribution over C .
We denote the set of all such data classifers by H NN .
Crowds aggregator g
The crowds aggregator g is a “weighted average” function to aggregate crowdsourced labels with parameters { W m ∈ R ∣C∣×∣C∣ } Mm = 1 and b .
Its input y [ M ] is the crowdsourced labels provided by M experts for a datapoint and its output is a distribution over C .
By representing each y m ∈ y
[ M ] as an one-hot vector e ( y m ) ∶= ( 0 ,..., 1 ,..., 0 ) ⊺ ∈ { 0 , 1 } ∣C∣ where only the y m th entry of e ( y m ) is 1,
[Equation]
where W m ⋅ e ( y m ) is equivalent to pick the y m th column of matrix W m , as shown in Figure 3.
We denote the set of all such crowds aggregators by G WA .
Data-crowds forecaster ζ
Given a data classiﬁer h , a crowds aggregator g and a distribution p =
( p c ) c ∈ ∆ C over the classes, the data-crowds forecaster ζ , that forecasts the ground truth based on both the datapoint x and the crowdsourced labels y
[ M ] , is constructed by
[Equation]
where Normalize ( v ) ∶= v ∑ c v c .
f -mutual information gain MIG f f -mutual information gain MIG f measures the “mutual information” between two hypotheses, which is proposed by Kong & Schoenebeck (2018).
Given
N datapoints x 1 ,x 2 ,...,x N ∈ I where each datapoint x
i is labeled by M crowdsourced labels
To calculate the f -mutual information gain, we reward them for the average “agreements” between their outputs for the same task, i.e. h ( x i ) and g ( y [ M ] i ) , as shown by the black lines, and punish them for the average “agreements” between their outputs for the different tasks, i.e. h ( x i ) and g ( y [ M ] j ) where i ≠ j , as shown by the grey lines.
Intuitively, the reward encourages the data classiﬁer to agree with the crowds aggregator, while the punishment avoids them naively agreeing with each other, that is, both of them map everything to ( 1 , 0 ,..., 0 ) .
The measurement of “agreement” depends on the selection of f .
See formal deﬁnition for MIG f in (1). y {+
+} i ,y 2 i ,...,y Mi ∈ C , the f -mutual information gain between h and g , associated with a hyperparameter p =
( p c ) c ∈ ∆ C , is deﬁned as the average “agreements” between h and g for the same task minus the average “agreements” between h and g for the different tasks, that is,
[Equation]
where f is a convex function satisfying f ( 1 ) =
and f ⋆ is the Fenchel duality of f .
We can use
Table 1 as reference for ∂f (⋅) and f ⋆ ( ∂f (⋅)) .
[Table]
[Equation] [Table]
[Equation] [Table] [Equation] K
[Table] [Equation] [Table]
[Equation]
Since the parameters of h is Θ and the parameters of g is { W m } Mm = 1 and b , we naturally rewrite MIG f ({ x i } , { y [ M ] i } ; h,g, p ) as [Equation]
We seek { Θ , { W m } Mm = 1 , b , p } that maximizes MIG f .
Later we will show that when the prior of the ground truth is p ∗ (e.g. p ∗ = ( 0 . 8 , 0 . 2 ) i.e. the ground truth is benign with probability 0.8 and malignant with probability 0.2 a priori), the best b and p are log p ∗ and p ∗ respectively.
Thus, we can set b as log p and only tune p .
When we have side information about the prior p ∗ , we can ﬁx parameter p as p ∗ , and ﬁx parameter b as log p ∗ .
T HEORETICAL JUSTIFICATION
This section provides a theoretical validation for Max-MIG, i.e., maximizing the f -mutual information gain over H NN and G WA ﬁnds the “information intersection” between the data and the crowdsourced labels.
In Appendix E, we compare our method with the MLE method (Raykar et al., 2010) theoretically and show that unlike our method, MLE is not robust to the correlated mistakes case.
Recall that we assume that conditioning on the ground truth, the data and the crowdsourced labels are mutually independent.
Thus, we can naturally deﬁne the “information intersection” as a pair of data classiﬁer and crowds aggregator h ∗ ,g ∗ such that they both fully use their input to forecast the ground truth.
Kong & Schoenebeck (2018) shows that when we have inﬁnite number of datapoints and maximize over all possible data classiﬁers and crowds aggregators, the “information intersection” will maximize MIG f ( h,g ) to the f -mutual information (Appendix C) between the data and the crowdsourced labels.
However, in practice, with a ﬁnite number of datapoints, the data classiﬁer and the crowds aggregator space should be not only sufﬁciently rich to contain the “information intersection” but also sufﬁciently simple to avoid over-ﬁtting.
Later, the experiment section will show that our picked H NN and G WA are sufﬁciently simple to avoid over-ﬁtting.
We assume the neural network space is sufﬁciently rich.
It remains to show that our weighted average aggregator space
G WA is sufﬁciently rich to contain g ∗ .
Model and assumptions Each datapoint x i with crowdsourced labels provided by M experts
y 1 i ,...,y Mi are drawn i.i.d.
from random variables X,Y 1 ,...,Y M .
Assumption 3.1 (Co-training assumption) .
X and Y [ M ] are independent conditioning on Y .
Note that we do not assume that the experts’ labels are conditionally mutually independent.
We deﬁne p ∗ ∈ ∆ C as the prior for Y , i.e. p ∗ c = P ( Y = c ) .
Deﬁnition 3.2 (Information intersection) .
We deﬁne h ∗ , g ∗ and ζ ∗ such that
[Equation]
We call them Bayesian posterior data classiﬁer / crowds aggregator / data-crowds forecaster respectively.
We call ( h ∗ ,g ∗ ) the information intersection between the data and the crowdsourced labels.
We also assume the neural network space is sufﬁciently rich to contain h ∗ .
Assumption 3.3 (Richness of the neural networks) .
h ∗ ∈ H NN .
Theorem 3.4.
With assumptions 3.1, 3.3, when there exists a subset of experts S ⊂ [ M ] such that the experts in S are mutually independent conditioning on Y and Y S is a sufﬁcient statistic for Y , i.e. P ( Y =
y
∣ Y
[ M ] = y [ M ] ) =
P ( Y = y
∣ Y S = y S )
for every y ∈ C ,y [ M ] ∈ C M , then ( h ∗ ,g ∗ , p ∗ ) is a maximizer of
[Equation]
and the maximum is the f -mutual information between X and Y [ M ] .
Moreover, ζ ∗ ( x,y [ M ] ) =
ζ ( x,y
[ M ] ; h ∗ ,g ∗ , p ∗ ) for every x,y
[ M ] .
Our main theorem shows that if there exists a subset of senior experts such that these senior experts are mutually conditional independent and it is sufﬁcient to only use the information from these senior experts, then Max-MIG ﬁnds the “information interstion”.
Note that we do not need to know the identities of the senior experts.
For other junior experts, we allow any dependency structure among them and between them and the senior experts.
Moreover, this theorem also shows that our method handles the independent mistakes case where all experts can be seen as senior experts (Proposition D.3).
To show our results, we need to show that G WA contains g ∗ , i.e. there exists proper weights such that g ∗ can be represented as a weighted average.
In the independent mistakes case, we can construct each expert’s weight using her confusion matrix.
Thus, in this case, each expert’s weight represents her expertise.
In the general case, we can construct each senior expert’s weight using her confusion matrix and make the junior experts’ weights zero.
Due to space limitation, we defer the formal proofs to Appendix D.
E XPERIMENT
In this section, we evaluate our method on image classiﬁcation tasks with both synthesized crowdsourced labels in various of settings and real world data.
Our method Max-MIG is compared with: Majority Vote , training the network with the major vote labels from all the experts; Crowd Layer , the method proposed by Rodrigues & Pereira (2017);
Doctor Net , the method proposed by Guan et al. (2017) and AggNet , the method proposed by
Albarqouni et al.
Image datasets Three datasets are used in our experiments.
The Dogs vs. Cats (Kaggle, 2013) dataset consists of 25 , 000 images from 2 classes, dogs and cats, which is split into a 12 , 500 -image training set and a 12 , 500 -image test set.
The CIFAR-10 (Krizhevsky et al., 2014) dataset consists of 60 , 000 32 × 32 color images from 10 classes, which is split into a 50 , 000 -image training set and a 10 , 000 -image test set.
The LUNA16 (Setio et al., 2016) dataset consists of 888 CT scans for lung nodule.
We preprocessed the CT scans by generating 8106 50 × 50 gray-scale images, which is split into a 6484 -image training set and a 1622 -image testing set.
LUNA16 is highly imbalanced dataset (85%, 15%).
Synthesized crowdsourced labels in various of settings For each information structure in Figure 1, we generate two groups of crowdsourced labels for each dataset: labels provided by (H) experts with relatively high expertise; (L) experts with relatively low expertise.
For each of the situation (H) (L), all three cases have the same senior experts.
Case 4.1.
(Independent mistakes)
M s senior experts are mutually conditionally independent.
Case 4.2.
(Naive majority)
M s senior experts are mutually conditional independent, while other M j junior experts label all datapoints as the ﬁrst class effortlessly.
Case 4.3.
(Correlated mistakes)
M s senior experts are mutually conditional independent, and each junior expert copies one of the senior experts.
Real-world dataset The LabelMe data (Rodrigues & Pereira, 2017; Russell et al., 2008) consists of a total of 2688 images, where 1000 of them were used to obtain labels from multiple annotators from Amazon Mechanical Turk and the remaining 1688 images were using for evaluating the different approaches.
Each image was labeled by an average of 2.547 workers, with a mean accuracy of 69.2%.
Networks We follow the four layers network in Rodrigues & Pereira (2017) on Dogs vs. Cats and
LUNA16 and use VGG-16 on CIFAR-10 for the backbone of the data classiﬁer h .
For Labelme data, we apply the same setting of Rodrigues & Pereira (2017): we use pre-trained VGG-16 deep neural network and apply only one FC layer (with 128 units and ReLU activations) and one output layer on top, using 50% dropout.
We defer other implementation details to appendix B.
[Table]
4. R ESULTS
We train the data classiﬁer h on the four datasets through our method 1 and other related methods.
The accuracy of the trained data classiﬁers on the test set are shown in Table 2 and Figure 4.
We also show the accuracy of our data-crowd forecaster and on the test set and compare it with AggNet (Table 3).
For the performances of the trained data classiﬁers, our method Max-MIG (red) almost outperform all other methods in every experiment.
For the real-world dataset, LabelMe, we achieve the new state-of-the-art results.
For the synthesized crowdsourced labels, the majority vote method (grey) fails in the naive majority situation.
The AggNet has reasonably good performances when the experts are conditionally independent, including the naive majority case since naive expert is independent with everything, while it is outperformed by us a lot in the correlated mistakes case.
This matches the theory in Appendix E: the AggNet is based on MLE and MLE fails in correlated mistakes case.
The Doctor Net (green) and the Crowd Layer (blue) methods are not robust to the naive majority case.
Our data-crowds forecaster (Table 3) performs better than our data classiﬁer, which shows that our data-crowds forecaster actually takes advantage of the additional information, the crowdsourced labels, to give a better result.
Like us, Aggnet also jointly trains the classiﬁer and the aggregator, and can be used to train a data-crowds forecaster.
We compared our data-crowds forecaster with Aggnet.
The results still match our theory.
When there is no correlated mistakes, we outperform Aggnet or have very similar performances with it.
When there are correlated mistakes, we outperform Aggnet a lot (e.g. +30%).
Recall that in the experiments, for each of the situation (H) (L), all three cases have the same senior experts.
Thus, all three cases’ crowdsourced labels have the same amount of information.
The results show that Max-MIG has similar performances for all three cases for each of the situation (H) (L), which validates our theoretical result: Max-MIG ﬁnds the “information intersection” between the data and the crowdsourced labels.
C ONCLUSION AND DISCUSSION
We propose an information theoretic approach, Max-MIG, for joint learning from crowds, with a common assumption: the crowdsourced labels and the data are independent conditioning on the ground truth.
We provide theoretical validation to our approach and compare our approach experimentally with previous methods (Doctor net (Guan et al., 2017), Crowd layer (Rodrigues &
Pereira, 2017), Aggnet (Albarqouni et al., 2016)) under several different information structures.
Each of the previous methods is not robust to at least one information structure and our method is robust to all and almost outperform all other methods in every experiment.
To the best of our knowledge, our approach is the ﬁrst algorithm that is both theoretically and empirically robust to the situation where some people make highly correlated mistakes and some people label effortlessly, without knowing the information structure among the crowds.
We also test our method on real-world data and achieve the new state-of-the-art result.
Our current implementation of Max-MIG has several limitations.
For example, we implement the aggregator using a simple linear model, which cannot handle the case when the senior experts are latent and cannot be linearly inferred from the junior experts.
However, note that if the aggregator space is sufﬁciently rich, the Max-MIG approach is still able to handle any situation as long as the “information intersection” assumption holds.
One potential future direction is designing more complicated but still trainable aggregator space.
A CKNOWLEDGMENTS
We would like to express our thanks for support from the following research grants NSFCand 61527804.
A D ATA -C ROWDS F ORECASTER C OMPARISON
Dataset
Method 4.1(H) 4.2(H) 4.3(H) 4.1(L)
4.2(L) 4.3(L)
Dogs vs.Cats
Max-MIG (d) 79 .80 .78 .
Max-MIG (dc) 88 .87 .87 .
AggNet (d) 79 .
AggNet (dc) 88 .88 .75 .
CIFARMax-MIG(d) 86 .86 .
Max-MIG(dc) 98 .98 .99 .75 .
AggNet(d)
AggNet(dc) 99 .99 .70 .
LUNA
Max-MIG(d) 90 .91 .
Max-MIG(dc) 94 .93 .
AggNet(d)
AggNet(dc) 91 .
Here (dc) is the shorthand for data-crowds forecaster and (d) is the shorthand for data-classiﬁer.
We take the average of ﬁve times experiments and the variance is pretty small.
Due to space limitation, we omit the variance here.
B E XPERIMENTS DETAILS B. E XPERTS ’ EXPERTISE
For each information structure in Figure 1, we generate two groups of crowdsourced labels for each dataset: labels provided by (H) experts with relatively high expertise; (L) experts with relatively low expertise.
For each of the situation (H) (L), all three cases have the same senior experts.
Case B.1.
(Independent mistakes)
M s senior experts are mutually conditionally independent.
(H)
M s = 5 .
(L) M s = 10 .
Dogs vs. Cats In situation (H), some senior experts are more familiar with cats, while others make better judgments on dogs.
For example, expert A is more familiar with cats, her expertise for dogs/cats is 0.6/0.8 in the sense that if the ground truth is dog/cat, she labels the image as “dog”/“cat” with probability 0.6/0.8 respectively.
Similarly, other experts expertise are B:0.6/0.6, C:0.9/0.6, D:0.7/0.7,
E:0.6/0.7.
In situation (L), all ten seniors’ expertise are 0.55/0.55.
CIFAR-
In situation (H), we generate experts who may make mistakes in distinguishing the hard pairs: cat/dog, deer/horse, airplane/bird, automobile/trunk, frog/ship, but can perfectly distinguish other easy pairs (e.g. cat/frog), which makes sense in practice.
When they cannot distinguish the pair, some of them may label the pair randomly and some of them label the pair the same class.
In detail, for each hard pair, expert A label the pair the same class (e.g. A always labels the image as “cat” when the image has cats or dogs), expert B labels the pair uniformly at random (e.g. B labels the image as “cat” with the probability 0.5 and “dog” with the probability 0.5 when the image has cats or dogs).
Expert C is familiar with mammals so she can distinguish cat/dog and deer/hose, while for other hard pairs, she label each of them uniformly at random.
Expert D is familiar with vehicles so she can distinguish airplane/bird, automobile/trunk and frog/ship, while for other hard pairs, she always label each of them the same class.
Expert E does not have special expertise.
For each hard pair, Expert E labels them correctly with the probability 0.6.
In situation (L), all ten senior experts label each image correctly with probability 0 .
2 and label each image as other false classes uniformly with probability 0 .
LUNA
In situation (H), some senior experts tend to label the image as “benign” while others tend to label the image as “malignant”.
Their expertise for benign/malignant are: A: 0.6/0.9, B:0.7/0.7,
C:0.9/0.6, D:0.6/0.7, E:0.7/0.6.
In situation (L), all ten seniors’ expertise are 0.6/0.6.
Case B.2.
(Naive majority)
M s senior experts are mutually conditional independent, while other
M j junior experts label all data as the ﬁrst class effortlessly.
(H)
M s = 5 , M j = 5 .
(L) M s = 10 ,
M j = 15 .
For Dogs vs. Cats, all junior experts label everything as “cat”.
For CIFAR-10, all junior experts label everything as “airplane”.
For LUNA16, all junior experts label everything as “benign”.
Case B.3.
(Correlated mistakes)
M s senior experts are mutually conditional independent, and each junior expert copies one of the senior experts.(H)
M s = 5 , M j = 5 .
(L) M s = 10 , M j = 2 .
For Dogs vs. Cats, CIFAR-10 and LUNA16, in situation (H), two junior experts copy expert
A ’s labels and three junior experts copy expert C ’s labels; in situation (L), one junior expert copies expert
A ’s labels and another junior expert copies expert C ’s labels.
B. I MPLEMENTATION DETAILS
Networks For Dogs vs. Cats and LUNA16, we follow the four layers network in Rodrigues &
Pereira (2017).
We use Adam optimizer with learning rate 1 .
× 10 − 4 for both the data classiﬁer and the crowds aggregator.
Batch size is set to 16 .
For CIFAR-10, we use VGG-16 as the backbone.
We use Adam optimizer with learning rate 1 .
× 10 − 3 for the data classiﬁer and 1 .
× 10 − 4 for the crowds aggregator.
Batch size is set to 64 .
For Labelme data, We apply the same setting of Rodrigues & Pereira (2017): we use pre-trained
VGG-16 deep neural network and apply only one FC layer (with 128 units and ReLU activations) and one output layer on top, using 50% dropout.
We use Adam optimizer with learning rate 1 .
× 10 −for both the data classiﬁer and the crowds aggregator.
For our method MAX-MIG’s crowds aggregator, for Dogs vs. Cats and LUNA16, we set the bias b as log p and only tune p .
For CIFAR-10 and Labelme data, we ﬁx the prior distribution p to be the uniform distribution p 0 and ﬁx the bias b as log p 0 .
Initialization For AggNet and our method Max-MIG, we initialize the parameters { W m } m using the method in Raykar et al.
[Equation]
where 1 ( y mi = c ′ )
= 1 when y mi = c ′ and 1 ( y mi = c ′ )
= 0 when y mi ≠ c ′ and N is the total number of datapoints.
We average all crowdsourced labels to obtain Q ( y i = c ) ∶= 1 M M ∑ m = 1 1 ( y mi = c ) .
For Crowd Layer method, we initialize the weight matrices using identity matrix on Dogs vs. Cats and LUNA as Rodrigues & Pereira (2017) suggest.
However, this initialization method leads to pretty bad results on CIFAR-10.
Thus, we use (2) for Crowd Layer on CIFAR-10, which is the best practice in our experiments.
C f - MUTUAL INFORMATION C. f - DIVERGENCE AND F ENCHEL ’ S DUALITY
f -divergence (Ali & Silvey, 1966; Csisz´ar et al., 2004)
f
-divergence D f
∶ ∆ Σ × ∆ Σ ↦ R is a non-symmetric measure of the difference between distribution p ∈ ∆ Σ and distribution q ∈ ∆ Σ and is deﬁned to be
[Equation]
where f ∶ R ↦ R is a convex function and f ( 1 ) = 0 .
C. f - MUTUAL INFORMATION
Given two random variables X,Y whose realization space are Σ X and Σ Y , let U X,Y and V X,Y be two probability measures where U X,Y is the joint distribution of ( X,Y ) and V X,Y is the product of the marginal distributions of X and Y .
Formally, for every pair of ( x,y ) ∈ Σ X × Σ Y ,
U X,
[Equation]
If U X,Y is very different from V X,Y , the mutual information between X and Y should be high since knowing X changes the belief for Y a lot.
If U X,Y equals to V X,Y , the mutual information between
X and Y should be zero since X is independent with Y .
Intuitively, the “distance” between U X,Y and V X,Y represents the mutual information between them.
Deﬁnition C.1 ( f -mutual information (Kong & Schoenebeck, 2016)) .
The f -mutual information between X and Y is deﬁned as
[Equation]
where D f is f -divergence.
f -mutual information is always non-negative.
Kong & Schoenebeck (2016) show that if we measure the amount of information by f -mutual information, any “data processing” on either of the random variables will decrease the amount of information crossing them.
With this property, Kong & Schoenebeck (2016) propose an information theoretic mechanism design framework using f -mutual information.
Kong & Schoenebeck (2018) reduce the co-training problem to a mechanism design problem and extend the information theoretic framework in Kong & Schoenebeck (2016) to address the co-training problem.
D P ROOF OF T HEOREM 3.
This section provides the formal proofs to our main theorem.
Deﬁnition D.1 (Confusion matrix) .
For each expert m , we deﬁne her confusion matrix as C m =
( C mc,c ′ )
c,c ′ ∈ R ∣C∣×∣C∣ where C mc,c ′ =
P ( Y m = c ′ ∣ Y = c ) .
We denote the set of all possible classiﬁers by H ∞ and the set of all possible aggregators by G ∞ .
Lemma D.2.
(Kong & Schoenebeck, 2018)
With assumption 3.1, 3.3, ( h ∗ ,g ∗ , p ∗ ) is a maximizer of
[Equation]
and the maximum is the f mutual information between X and Y [ M ] , MI f ( X,Y [ M ] ) .
Moreover, ζ ∗ ( x,y [ M ] ) =
ζ ( x,y
[ M ] ; h ∗ ,g ∗ , p ∗ ) for every x,y
[ M ] .
Proposition D.3.
[Independent mistakes] With assumptions 3.1, 3.3, if the experts are mutually independent conditioning on Y , then g ∗ ∈ G WA and
[Equation]
for every y [ M ] ∈ C M .
This implies that ( h ∗ ,g ∗ , p ∗ ) is a maximizer of
[Equation]
WA
[Equation]
p ) and the maximum is the f mutual information between X and Y [ M ] , MI f ( X,Y [ M ] ) .
Moreover, ζ ∗ ( x,y [ M ] ) =
ζ ( x,y
[ M ] ; h ∗ ,g ∗ , p ∗ ) for every x,y
[ M ] .
Proof.
We will show that when the experts are mutually conditionally independent, then
[Equation]
This also implies that g ∗ ∈ G WA .
Based on the result of Lemma D.2, by assuming that h ∗ ∈
H NN , we can see ( h ∗ ,g ∗ , p ∗ ) is a maximizer of max h ∈ H NN ,g ∈ G WA , p ∈ ∆ C MIG f ( h,g, p ) and the maximum is the f mutual information between X and Y
[ M ] .
Moreover, Lemma D.2 also implies that ζ ∗ ( x,y [ M ] )
=
ζ ( x,y
[ M ] ; h ∗ ,g ∗ , p ∗ ) for every x,y
[ M ] .
For every c ∈ C , every y [ M ] ∈ C M ,
[Equation]
Thus,
[Equation]
Then,
[Equation]
Thus,
[Equation]
We restate our main theorem, Theorem 3.4, here with more details and prove it.
Theorem 3.4 (General case) .
With assumption 3.1, 3.3, when there exists a subset of experts S ⊂ [ M ] such that the experts in S are mutually independent conditioning on Y and Y S is a sufﬁcient statistic for Y , i.e. P ( Y =
y
∣ Y
[ M ] = y [ M ] ) =
P ( Y = y
∣ Y S = y S )
for every y ∈ C ,y [ M ] ∈ C M , then g ∗ ∈ G WA and
[Equation]
for every y [ M ] ∈ C M where for every m ∈ S , W ∗ m = log C m , for every m ∉ S , W ∗ m = 0 2 .
This implies that ( h ∗ ,g ∗ , p ∗ ) is a maximizer of
[Equation]
and the maximum is the f mutual information between X and Y [ M ] , MI f ( X,Y [ M ] ) .
Moreover, ζ ∗ ( x,y [ M ] ) =
ζ ( x,y
[ M ] ; h ∗ ,g ∗ , p ∗ ) for every x,y
[ M ] .
Proof.
Like the proof for the above proposition, we need to show that
[Equation]
This also implies that g ∗ ∈ G WA as well as the other results of the theorem.
When Y S is a sufﬁcient statistic for Y , we have
[Equation]
Proposition D.3 shows that
[Equation]
Thus, we have
[Equation]
where for every m ∈ S , W ∗ m = log C m , for every m ∉ S , W ∗ m = 0 .
E T HEORETICAL COMPARISONS WITH MLE
Raykar et al.
(2010) propose a maximum likelihood estimation (MLE) based method in the learning from crowds scenario.
Raykar et al.
(2010) use logistic regression and Aggnet(Albarqouni et al., 2016) extends it to combine with the deep learning model.
In this section, we will theoretically show that these MLE based methods can handle the independent mistakes case but cannot handle even the simplest correlated mistakes case—only one expert reports meaningful information and all other experts always report the same meaningless information—which can be handled by our method.
Therefore, in addition to the experimental results, theoretically, our method is still better than these
MLE based methods.
We ﬁrst introduce these MLE based methods.
Let Θ be the parameter that control the distribution over X and Y .
Let Θ m be the parameter that controls the distribution over Y m and Y .
For each each x , y [ M ] ,
[Equation]
[Equation]
(conditioning on Y , X and Y [ M ] are independent)
[Equation]
(experts are mutually conditional independent.)
The MLE based method seeks Θ and Θ m that maximize
[Equation]
To theoretically compare it with our method, we use our language to reinterpret the above MLE based method.
We deﬁne T as the set of all ∣C∣ × ∣C∣ transition matrices with each row summing to 1.
For each expert m , we deﬁne W m ∈ T as a parameter that is associated with m .
Given a set of data classiﬁers h ∈ H where h ∶ I ↦ ∆ C , the MLE based method seeks h ∈ H and transition matrices W 1 , W 2 , ⋯ , W M ∈ T that maximize
[Equation]
The expectation of the above formula is
[Equation]
Note that Raykar et al.
(2010) set the data classiﬁers space H as all logistic regression classiﬁers and
Albarqouni et al.
(2016) extend this space to the neural network space.
Proposition E.1 (MLE works for independent mistakes) .
If the experts are mutually independent conditioning on Y, then h ∗ and C 1 , C 2 , ⋯ , C M are a maximizer of
[Equation]
Proof.
[Equation]
Since W 1 , W 2 , ⋯ , W m ∈ T , thus,
[Equation]
which means (∑ c ∈C h ( x ) c
Π Mm =
1 W mc,y m )
y
[ M ] can be seen as a distribution over all possible y [ M ] ∈ C M .
Moreover, for any two distribution vectors p and q , p ⋅ log q ≤ p ⋅ log p , thus
[Equation]
see equation (3))
Thus, the MLE based method handles the independent mistakes case.
However, we will construct a counter example to show that it cannot handle a simple correlated mistakes case which can be handled by our method.
Example E.2 (A simple correlated mistakes case) .
We assume there are only two classes C
= { 0 , 1 } and the prior over Y is uniform, that is, P ( Y = 0 ) =
P ( Y = 1 ) = 0 .
We also assume that X = Y .
There are 101 experts and one of the experts, say her the ﬁrst expert, fully knows Y and always reports Y
= Y .
The second expert knows nothing and every time ﬂips a random unbiased coin whose randomness is independent with X,Y .
She reports Y 2 = 1 when she gets head and reports Y
=otherwise.
The rest of experts copy the second expert’s answer all the time, i.e. Y m =
Y 2 , for m ≥ 2 .
Note that our method can handle this simple correlated mistakes case and will give all useless experts weight zero based on Theorem 3.4.
We deﬁne h 0 as a data classiﬁer such that h 0 ( x )
= h 0 ( x ) 1 = 0 .
We will show this meaningless data classiﬁer h 0 has much higher likelihood than h ∗ , which shows that in this simple correlated mistakes case, the MLE based method will obtain meaningless results.
We deﬁne a data classiﬁer h ’s maximal expected likelihood as
[Equation]
Theorem E.3 (MLE fails for correlated mistakes) .
In the scenario deﬁned by Example E.2, the meaningless classiﬁer h 0 ’s maximal expected likelihood is at least log 0 .
5 and the Bayesian posterior classiﬁer h ∗ ’s maximal expected likelihood is 100log 0 .
5 ≪ log 0 .
The above theorem implies that the MLE based method fails in Example E.2.
Proof.
For the Bayesian posterior classiﬁer h ∗ , since X =
Y =
Y 1 and Y 2 = ⋯ =
Y M , then h ∗ ( X = c ) is an one-hot vector where the c th entry is 1 and everything is determined by the realizations of Y and Y 2 .
[Equation]
( W m is a transition matrix and p ⋅ log q ≤ p ⋅ log p ) = 100log 0 .
( Y 2 equals 0 with probability 0.5 and 1 with probability 0.5 as well)
The maximal value is obtained by setting W 1 as an identity matrix and setting W 2 = ⋯ =
W M as ( 0 . 5 0 .
Thus, the Bayesian posterior data classiﬁer h ∗ ’s maximal expected likelihood is 100log 0 .
For the meaningless data classiﬁer h 0 ,
[Equation]
Note when we set every W m as an identity matrix, the above formula equals log 0 .
Thus, the meaningless data classiﬁer h 0 ’s maximal expected likelihood is at least log 0 .