F
IDELITY -W EIGHTED L
EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Training deep neural networks requires many training samples, but in practice training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing.
This creates a fundamental qualityversus-quantity trade-off in the learning process.
Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data?
We argue that if the learner could somehow know and take the label-quality into account when learning the data representation, we could get the best of both worlds.
To this end, we propose “ﬁdelity-weighted learning” (FWL), a semi-supervised studentteacher approach for training deep neural networks using weakly-labeled data.
FWL modulates the parameter updates to a student network (trained on the task we care about) on a per-sample basis according to the posterior conﬁdence of its label-quality estimated by a teacher (who has access to the high-quality labels).
Both student and teacher are learned from the data.
We evaluate FWL on two tasks in information retrieval and natural language processing where we outperform state-of-the-art alternative semi-supervised methods, indicating that our approach makes better use of strong and weak labels, and leads to better task-dependent data representations.
I NTRODUCTION
The success of deep neural networks to date depends strongly on the availability of labeled data which is costly and not always easy to obtain.
Usually it is much easier to obtain small quantities of high-quality labeled data and large quantities of unlabeled data.
The problem of how to best integrate these two different sources of information during training is an active pursuit in the ﬁeld of semi-supervised learning (Chapelle et al., 2006).
However, for a large class of tasks it is also easy to deﬁne one or more so-called “weak annotators”, additional (albeit noisy) sources of weak supervision based on heuristics or “weaker”, biased classiﬁers trained on e.g. non-expert crowd-sourced data or data from different domains that are related.
While easy and cheap to generate, it is not immediately clear if and how these additional weakly-labeled data can be used to train a stronger classiﬁer for the task we care about.
More generally, in almost all practical applications machine learning systems have to deal with data samples of variable quality.
For example, in a large dataset of images only a small fraction of samples may be labeled by experts and the rest may be crowd-sourced using e.g.
Amazon Mechanical Turk (Veit et al., 2017).
In addition, in some applications, labels are intentionally perturbed due to privacy issues (Wainwright et al., 2012; Papernot et al., 2016).
Assuming we can obtain a large set of weakly-labeled data in addition to a much smaller training set of “strong” labels, the simplest approach is to expand the training set by including the weakly-supervised samples (all samples are equal).
Alternatively, one may pretrain on the weak data and then ﬁne-tune on observations from the true function or distribution (which we call strong data).
Indeed, it has recently been shown that a small amount of expert-labeled data can be augmented in such a way by a large set of raw data, with labels coming from a heuristic function, to train a more accurate neural ranking model (Dehghani et al., 2017).
The downside is that such approaches are oblivious to the amount or source of noise in the labels.
In this paper, we argue that treating weakly-labeled samples uniformly (i.e. each weak sample contributes equally to the ﬁnal classiﬁer) ignores potentially valuable information of the label quality.
Instead, we propose Fidelity-Weighted Learning (FWL), a Bayesian semi-supervised approach that leverages a small amount of data with true labels to generate a larger training set with conﬁdence-weighted weakly-labeled samples , which can then be used to modulate the ﬁne-tuning process based on the ﬁdelity (quality) of each weak sample.
By directly modeling the inaccuracies
[Figure]
introduced by the weak annotator in this way, we can control the extent to which we make use of this additional source of weak supervision: more for conﬁdently-labeled weak samples close to the true observed data, and less for uncertain samples further away from the observed data.
We propose a setting consisting of two main modules.
One is called the student and is in charge of learning a suitable data representation and performing the main prediction task, the other is the teacher which modulates the learning process by modeling the inaccuracies in the labels.
We explain our approach in much more detail in Section 2, but at a high level it works as follows (see Figure 1): We pretrain the student network on weak data to learn an initial task-dependent data representation which we pass to the teacher along with the strong data.
The teacher then learns to predict the strong data, but crucially, based on the student’s learned representation .
This then allows the teacher to generate new labeled training data from unlabeled data, and in the process correct the student’s mistakes, leading to a better ﬁnal data representation and better ﬁnal predictor.
We introduce the proposed FWL approach in more detail in Section 2.
We then present our experimental setup in Section 3 where we evaluate FWL on a toy task and two real-world tasks, namely document ranking and sentence sentiment classiﬁcation.
In both latter cases, FWL outperforms competitive baselines and yields state-of-the-art results, indicating that FWL makes better use of the limited true labeled data and is thereby able to learn a better and more meaningful task-speciﬁc representation of the data.
One may also view FWL from the perspective of Vapnik’s learning with privileged information (LUPI) framework (Vapnik & Izmailov, 2015), for which we provide empirical support in Section 4.2.
We discuss this connection in more detail in Appendix A.
F IDELITY -W EIGHTED L EARNING (FWL)
We now describe FWL, our proposed approach for semi-supervised learning when we have access to weak supervision (e.g. heuristics or weak annotators).
We assume we are given a large set of unlabeled data samples, a heuristic labeling function called the weak annotator , and a small set of high-quality sampleslabeledbyexperts, calledthe strongdataset , consistingoftuplesoftrainingsamples x i andtheir true labels y
i , i.e. D s = { ( x i ,y i ) } .
We consider the latter to be observations from the true target function that we are trying to learn.
We use the weak annotator to generate labels for the unlabeled samples.
Generated labels are noisy due to the limited accuracy of the weak annotator.
This gives us the weak dataset consisting of tuples of training samples x i and their weak labels ˜
y i , i.e. D w = { ( x i , ˜ y i ) } .
Note that we can generate a large amount of weak training data D w at almost no cost using the weak annotator.
In contrast, we have only a limited amount of observations from the true function, i.e. |D s |(cid:28)|D w | .
Our proposed setup comprises a neural network called the student and a Bayesian function approximator called the teacher .
The training process consists of three phases which we summarize in Algorithm 1 and Figure 1.
Step 1 Pre-train the student on D w using weak labels generated with the weak annotator .
The goal of this stage is to learn a reasonably good representation of the data for the given task.
The student function is a neural network consisting of two parts.
The ﬁrst part ψ ( . ) learns the data representation and the second part φ ( . ) performs the prediction task (e.g. classiﬁcation).
Therefore the overall function is ˆ y = φ ( ψ ( x i )) .
The student is trained on all samples of the weak dataset
D w = { ( x i , ˜ y i ) } .
For brevity, in the following, we will refer to both data sample
x i and its representation ψ ( x i ) by x i when it is obvious from the context.
Step 2 Train the teacher on the strong data ( ψ ( x j ) ,
y j ) ∈
D s represented in terms of the student representation ψ ( . ) and then use the teacher to generate a soft dataset D sw consisting of (cid:104) sample , predicted label , conﬁdence (cid:105) pairs for all data samples.
We use a Gaussian process as teacher to capture the label uncertainty in terms of the student representation, estimated w.r.t the strong data.
We explain the ﬁner details of the GP in Appendix D, and just present the overall description here.
A prior mean and co-variance function is chosen for GP .
The learned embedding function ψ ( · ) in Step 1 is then used to map the data samples to dense vectors as input to the GP .
The GP is trained on this representation of the strong dataset to learn the posterior mean m post (used to generate soft labels) and posterior co-variance K post ( .,. )
(which represents label uncertainty).
We then create the soft dataset D sw =
{ ( x t , ¯ y t ) } using the posterior GP , input samples x t from
D w ∪D s , and predicted labels ¯ y t with their associated uncertainties as computed by T ( x t ) and Σ( x t ) :
[Equation]
The generated labels are called soft labels .
Therefore, we refer to D sw as a soft dataset.
g ( . ) transforms the output of GP to the suitable output space.
For example in classiﬁcation tasks, g ( . ) would be the softmax function to produce probabilities that sum up to one.
For multidimensional-output tasks where a vector of variances is provided by the GP , the vector K post ( x t ,x t ) is passed through an aggregating function h ( . ) to generate a scalar value for the uncertainty of each sample.
Note that we train GP only on the strong dataset D s but then use it to generate soft labels ¯ y t
= T ( x t ) and uncertainty Σ( x t ) for samples belonging to D sw =
D w ∪D s .
In practice, we furthermore divide the space of data into several regions and assign each region a separate GP trained on samples from that region.
By this division of space, we take advantage of the knowledge learned by several teachers, each an expert on its speciﬁc region of data space.
As a nice side-effect, this also solves the scalability issues of GP s in that we can increase the number of regions until the number of points in each region is tractable with a single GP , and train these models in parallel.
See Algorithm 2 in Appendix B for the detailed description.
Step 3 Fine-tune the weights of the student network on the soft dataset, while modulating the magnitude of each parameter update by the corresponding teacher-conﬁdence in its label.
The student network of Step 1 is ﬁne-tuned using samples from the soft dataset D sw
= { ( x t , ¯ y t ) } where ¯ y t
= T ( x t ) .
The corresponding uncertainty Σ( x t ) of each sample is mapped to a conﬁdence value according to Equation 1 below, and this is then used to determine the step size for each iteration of SGD.
So, intuitively, for data points where we have true labels, the uncertainty of the teacher is almost zero, which means we have high conﬁdence and a large step-size for updating the parameters.
However, for data points where the teacher is not conﬁdent, we down-weight the training steps of the student.
This means that at these points, we keep the student function as it was trained on the weak data in Step 1.
More speciﬁcally, we update the parameters of the student by training on D sw using stochastic gradient descent (SGD):
[Equation]
where l ( · ) is the per-example loss, η t is the total learning rate, N is the size of the soft dataset D sw , www is the parameters of the student network, and R ( . ) is the regularization term.
We deﬁne the total learning rate as η t = η 1 ( t )
η 2 ( x t ) , where η 1 ( t ) is the usual learning rate of our chosen optimization algorithm that anneals over training iterations, and
η 2 ( x t ) is a function of the label uncertainty Σ( x t ) that is computed by the teacher for each data point.
Multiplying these two terms gives us the total learning rate.
In other words, η 2 represents the ﬁdelity (quality) of the current sample, and is used to multiplicatively modulate η 1 .
Note that the ﬁrst term does not necessarily depend on each data point, whereas the second term does.
We propose
[Equation]
to exponentially decrease the learning rate for data point x t if its corresponding soft label ¯ y t is unreliable (far from a true sample).
In Equation 1, β is a positive scalar hyper-parameter.
Intuitively, small β results in a student which listens more carefully to the teacher and copies its knowledge, while a large β makes the student pay less attention to the teacher, staying with its initial weak knowledge.
More concretely speaking, as β → 0 student places more trust in the labels ¯ y t estimated by the teacher and the student copies the knowledge of the teacher.
On the other hand, as β →∞ , student puts less weight on the extrapolation ability of GP and the parameters of the student are not affected by the correcting information from the teacher.
E XPERIMENTS
The neural networks in the proposed method are implemented in TensorFlow (Abadi et al., 2015; Tang, 2016).
GPﬂow (Matthews et al., 2017) is employed for developing the GP modules.
For both tasks, we evaluate the performance of our method compared to the following baselines:
WA .
The weak annotator, i.e. the unsupervised method used for annotating the unlabeled data.
NN W .
The student trained only on weak data.
NN S .
The student trained only on strong data.
NN S
+ /W .
The student trained on samples that are alternately drawn from D w without replacement, and D s with replacement.
Since |D s |(cid:28)|D w | , it oversamples the strong data.
NN W → S .
The student trained on weak dataset D w and ﬁne-tuned on strong dataset D s . 6.
NN W ω → S .
The student trained on the weak data, but the step-size of each weak sample is weighted by a ﬁxed value 0 ≤ ω ≤ 1 , and ﬁne-tuned on strong data.
As an approximation for the optimal value for ω , we have used the mean of η 2 of our model (below).
FWL \ Σ .
The student trained on the weakly labeled data and ﬁne-tuned on examples labeled by teacher without taking the conﬁdence into account.
This baseline is similar to (Veit et al., 2017).
FWL .
Our FWL model, i.e. the student trained on the weakly labeled data and ﬁne-tuned on examples labeled by teacher using the conﬁdence scores.
In the following, we apply FWL ﬁrst to a toy problem and then to two different real tasks: document ranking and sentiment classiﬁcation .
We introduce each task and the results produced for it.
The student network and teacher GP for each task is given in more detail in the appendix.
3. T OY P ROBLEM
We ﬁrst apply FWL to a one-dimensional toy problem to illustrate the various steps.
Let f t ( x )=sin( x ) be the true function (red dotted line in Figure 2a) from which a small set of observations D
s = { x j ,y j } is provided (red points in Figure 2b).
These observation might be noisy, in the same way that labels obtained from a human labeler could be noisy.
A weak annotator function f w ( x )=
2 sinc ( x ) (magenta line in Figure 2a) is provided, as an approximation to f t ( . ) .
The task is to obtain a good estimate of f t ( . ) given the set D s of strong observations and the weak annotator function f w ( . ) .
We can easily obtain a large set of observations D w = { x i , ˜ y i } from f w ( . ) with almost no cost (magenta points in Figure 2a).
We consider two experiments:
A neural network trained on weak data and then ﬁne-tuned on strong data from the true function, which is the most common semi-supervised approach (Figure 2c).
A teacher-student framework working by the proposed FWL approach.
[Figure] As can be seen in Figure 2d, FWL by taking into account label conﬁdence, gives a better approximation ofthetruehiddenfunction.
Werepeatedtheaboveexperiment10times.
TheaverageRMSEwithrespect to the true function on a set of test points over those 10 experiments for the student, were as follows:
Student is trained on weak data (blue line in Figure 2a): 0 . 8406 , 2.
Student is trained on weak data then ﬁne tuned on true observations (blue line in Figure 2c): 0 . 5451 , 3.
Student is trained on weak data, then ﬁne tuned by soft labels and conﬁdence information provided by the teacher (blue line in Figure 2d): 0 . 4143 (best).
More details of the neural network and GP along with the speciﬁcation of the data used in the above experiment are presented in Appendix D and F.1.
3. D OCUMENT R ANKING
This task is the core information retrieval problem and is challenging as the ranking model needs to learn a representation for long documents and capture the notion of relevance between queries and documents.
Furthermore, the size of publicly available datasets with query-document relevance judgments is unfortunately quite small ( ∼ 250 queries).
We employ a state-of-the-art pairwise neural ranker architecture as the student (Dehghani et al., 2017).
In this model, ranking is cast as a regression task.
Given each training sample x as a triple of query q , and two documents d + and d − , the goal is to learn a function F : { <q,d + ,d − > }→ R , which maps each data sample x to a scalar output value y indicating the probability of d + being ranked higher than d − with respect to q .
[Figure]
The student follows the architecture proposed in (Dehghani et al., 2017).
The ﬁrst layer of the network, i.e. representation learning layer ψ : { < q,d + ,d − > } → R m maps each input sample to an m - dimensional real-valued vector.
In general, besides learning embeddings for words, function ψ learns to compose word embedding based on their global importance in order to generate query/document embeddings.
The representation layer is followed by a simple fullyconnected feed-forward network with a sigmoidal output unit to predict the probability of ranking d + higher than d − .
The general schema of the student is illustrated in Figure 3.
More details are provided in Appendix C.1.
The teacher is implemented by clustered GP algorithm.
See Appendix D for more details.
The weak annotator is BM25 (Robertson & Zaragoza, 2009), a well-known unsupervised method for scoring query-document pairs based on statistics of the matched terms.
More details are provided in Appendix E.1.
[Table]
Description of the data with weak labels and data with true labels as well as setups of the documentranking experiments is presented in Appendix F.2 in more details.
Results and Discussions We conducted k-fold cross validation on D s (the strong data) and report two standard evaluation metrics for ranking: mean average precision (MAP) of the top-rankeddocuments and normalized discounted cumulative gain calculated for the top 20 retrieved documents (nDCG@20).
Table 1 shows the performance on both datasets.
As can be seen, FWL provides a signiﬁcant boost on the performance over all datasets.
In the ranking task, the student is designed in particular to be trained on weak annotations (Dehghani et al., 2017), hence training the network only on weak supervision, i.e. NN W performs better than NN S .
This can be due to the fact that ranking is a complex task requiring many training samples, while relatively few data with true labels are available.
Alternating between strong and weak data during training, i.e. NN S + /W seems to bring little (but statistically signiﬁcant) improvement.
However, we can gain better results by the typical ﬁne-tuning strategy, NN W → S .
We can gain improvement by ﬁne-tuning the NN W using labels generated by teacher without considering their conﬁdence score, i.e. FWL \ Σ .
This means we just augmented the ﬁne-tuning process by generating a ﬁne-tuning set using teacher which is better than D s in terms of quantity and
D w in terms of quality.
This baseline is equivalent to setting β =0 in Equation 1.
However, we see a big jump in performance when we use FWL to include the estimated label quality from the teacher, leading to the best overall results.
3. S ENTIMENT C LASSIFICATION
In sentiment classiﬁcation, the goal is to predict the sentiment (e.g., positive, negative, or neutral) of a sentence.
Each training sample x consists of a sentence s and its sentiment label ˜ y .
The student for the sentiment classiﬁcation task is a convolutional model which has been shown to perform best on the dataset we used (Deriu et al., 2017; Severyn & Moschitti, 2015a;b; Deriu et al., 2016).
The ﬁrst layer of the network learns the function ψ ( . ) which maps input sentence s to a dense vector as its representation.
The inputs are ﬁrst passed through an embedding layer mapping the sentence to a matrix S ∈ R m ×| s | , followed by a series of 1d convolutional layers with max-pooling.
The representation layer is followed by feed-forward layers and a softmax output layer which returns the probability distribution over all three classes.
Figure 4 presents the general schema of the architecture of the student.
See Appendix C.2 for more details.
The teacher for this task is modeled by a GP .
See Appendix D for more details.
The weak annotator is a simple unsupervised lexicon-based method (Hamdan et al., 2013; Kiritchenko et al., 2014), which estimate a distribution over sentiments for each sentence, based on sentiment labels of its terms.
More details are provided in Appendix E.2.
Speciﬁcation of the data with weak labels and data with true labels along with the detailed experimental setups are given in Appendix F.3.
Results and Discussion
We report Macro-F1, the ofﬁcial SemEval metric, in Table 2.
We see that the proposed FWL is the best performing among all the baselines.
For this task, since the amount of data with true labels are larger compared to the ranking task, the performance of NN S is acceptable.
Alternately sampling from weak and strong data gives better
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
results. Pretraining on weak labels then ﬁne-tuning the network on true labels, further improves the performance. Weighting the gradient updates from weak labels during pretraining and ﬁne-tuning the network with true labels, i.e. NN W ω → S seems to work quite well in this task.
Similar to the ranking task, ﬁne-tuning NN S based on labels generated by GP instead of data with true labels, regardless of the conﬁdence score, works better than standard ﬁne-tuning.
Besides the baselines
[Figure]
we also report the best performing systems which are also convolution-based models (Rouvier & Favre 2016 on SemEval-14; Deriu et al.
2016 on SemEval-15).
Using FWL and taking the conﬁdence into consideration outperforms the best systems and leads to the highest reported results on both datasets.
A NALYSIS 4. H ANDLING THE B IAS -V ARIANCE T RADE - OFF
As mentioned in Section 2, β is a hyper-parameter that controls the contribution of weak and strong data to the training procedure.
In order to investigate its inﬂuence, we ﬁxed everything in the model and ran the ﬁne-tuning stage with different values of β ∈{ 0 . 0 , 0 . 1 , 1 . 0 , 2 . 0 , 5 . 0 } in all the experiments.
Figure 5 illustrates the performance on the ranking (on Robust04 dataset) and sentiment classiﬁcation tasks (on SemEval14 dataset).
For both sentiment classiﬁcation and ranking, β =1 gives the best results.
We also experimented on the toy problem with different values of β in three cases: 1) having 10 observations from the true function (samesetupasSection3.1), markedas“ToyData” in the plot, 2) having only 5 observations from the true function, marked as “Toy Data *” in the plot, and 3) having f ( x ) = x
+ 1 as the weak function, which is an extremely bad approximator of the true function, marked as “Toy Data **” in the plot.
For the “Toy Data” experiment, β =1 turned out to be optimal.
However, for “Toy Data *”, where we have an extremely small number of observations from the true function, setting β to a higher value acts as a regularizer by relying more on weak signals, and eventually leads to better generalization.
On the other hand, for “Toy Data **”, where the quality of the weak annotator is extremely low, lower values of β put more focus on the true observations.
Therefore, β lets us control the bias-variance trade-off in these extreme cases
. 4. A G OOD T EACHER IS B ETTER T HAN M ANY O BSERVATIONS
In this section, we study the rate of learning for the student as the amount of training data is varied.
We performed two types of experiments for all tasks:
In the ﬁrst experiment, we use all the available strong data but consider different percentages of the entire weak dataset.
In the second experiment, we ﬁx [Figure] the amount of weak data and provide the model with varying amounts of strong data.
We use standard ﬁne-tuning with similar setups as for the baseline models 1 .
Figure 6 presents the results of these experiments.
In general, for all tasks and both setups, the student learns faster when there is a teacher.
One caveat is in the case where we have a very small amount of weak data.
In this case the student cannot learn a suitable representation in the ﬁrst step, and hence the performance of FWL is pretty low, as expected.
It is highly unlikely that this situation occurs in reality as obtaining weakly labeled data is much easier than strong data.
The empirical observation of Figure 6 that our model learns more with less data can also be seen as evidenceinsupportofanotherperspectivetoFWL,called learningusingprivilegedinformation (Vapnik & Izmailov, 2015).
We elaborate more on this connection in Appendix A.
R ELATED W ORK
Learning from imperfect labels has been thoroughly studied in the literature (Fr ´ enay & Verleysen, 2014).
In the semi-supervised setup, some ideas were developed to utilize weakly or even unlabeled data.
For instance, the idea of self-training (Rosenberg et al., 2005), pseudo-labeling (Lee, 2013;
Hinton et al., 2015), and Co-training (Blum & Mitchell, 1998) are introduced for augmenting the training set by unlabeled data with predicted labels.
As a common approach in semi-supervised learning, the unlabeled set can be used for learning the distribution of the data.
In particular for neural networks, greedy layer-wise pre-training of weights using unlabeled data is followed by supervised ﬁnetuning (Hinton et al., 2006; Deriu et al., 2017; Severyn & Moschitti, 2015b;a; Go et al., 2009).
Other methods learn unsupervised encoding at multiple levels of the architecture jointly with a supervised signal (Ororbia II et al., 2015; Weston et al., 2012).
On the other hand, some noise cleansing methods were proposed to remove or correct mislabeled samples (Brodley & Friedl, 1999).
There are some studies showing that weak or noisy labels can be leveraged by employing a particular architecture or deﬁning a proper loss function to avoid over-ﬁtting to imperfections of the training data (Dehghani et al., 2017; Patrini et al., 2016; Beigman & Klebanov, 2009; Zeng et al., 2015; Bunescu & Mooney, 2007).
One direction of research focuses on modeling the pattern of the noise or weakness in the labels.
For instance, methods that use a generative model to correct weak labels such that a discriminative model can be trained more effectively (Ratner et al., 2016; Rekatsinas et al., 2017; Varma et al., 2017).
Furthermore, methods that aim to capture the pattern of the noise by inserting an extra layer or a separate module (Sukhbaatar et al., 2015; Veit et al., 2017) tries to infer better labels from noisy ones and use them to supervise the training of the network.
Our proposed method can be categorized in this class.
C ONCLUSION
Training neural networks using large amounts of weakly annotated data is an attractive approach in scenarios where an adequate amount of data with true labels is not available, a situation which often arises in practice.
In this paper, we introduced ﬁdelity-weighted learning (FWL), a new student-teacher framework for semi-supervised learning in the presence of weakly labeled data.
We applied FWL to document ranking and sentiment classiﬁcation, and empirically veriﬁed that FWL speeds up the training process and improves over state-of-the-art
semi-supervised alternatives.
A PPENDICES A C ONNECTION WITH V APNIK ’ S LEARNING USING PRIVILEGED INFORMATION
In this section we highlight the connections of our work with Vapnik’s learning using privileged information (LUPI) (Vapnik & Vashist, 2009; Vapnik & Izmailov, 2015).
FWL makes use of information from a small set of correctly labeled data to improve the performance of a semi-supervised learning algorithm.
The main idea behind LUPI comes from the fact that humans learn much faster than machines.
This can be due to the role that an Intelligent Teacher plays in human learning.
In this framework, the training data is a collection of triplets
[Equation]
where each ( x i ,y i ) is a pair of feature-label and x ∗ i is the additional information provided by an intelligent teacher to ease the learning process for the student.
Additional information for each ( x i ,y i ) is available only during training time and the learning machine must only rely on x i at test time.
The theory of LUPI studies how to leverage such a teaching signal x ∗ i to outperform learning algorithms utilizing only the normal features x i .
For example, MRI brain images can be augmented with high-level medical or even psychological descriptions of Alzheimer’s disease to build a classiﬁer that predicts the probability of Alzheimer’s disease from an MRI image at test time.
It is known from statistical learning theory (Vapnik, 1998) that the following bound for test error is satisﬁed with probability 1 − δ :
[Equation]
where R n ( f ) denotes the training error over n samples, |F| V C is the VC dimension of the space of functions from which f is chosen, and α ∈ [0 . 5 , 1] .
When the classes are not separable , α =0 .
i.e. the machine learns at a slow rate of O ( n − 1 / 2 ) .
For easier problems where classes are separable , α = resulting in a learning rate of O ( n − 1 ) .
The difference between these two cases is severe.
The same error bound achieved for a separable problem with 10 thousand data points is only obtainable for a non-separable problem when 100 million data points are provided.
This is prohibitive even when obtaining large datasets is not so costly.
The theory of LUPI shows that an intelligent teacher can reduce α resulting in a faster learning process for the student.
In this paper, we proposed a teacher - student framework for semi-supervised learning.
Similar to LUPI, in FWL a student is supposed to solve the main prediction task while an intelligent teacher provides additional information to improve its learning.
In addition, we ﬁrst train the student network so that it obtains initial knowledge of weakly labeled data and learns a good data representation.
Then the teacher is trained on truly labeled data enjoying the representation learnt by the student.
This extends LUPI in a way that the teacher provides privileged information that is most useful for the current state of student’s knowledge.
FWL also extends LUPI by introducing several teachers each of which is specialized to correct student’s knowledge related to a speciﬁc region of the data space.
Figure 6(a) provides evidence for the assumption that privileged information in our task can accelerate the learning process of the student.
It shows how the privileged information from an intelligent teacher affects the exponent α of the error bound in Equation 3.
Figure 6(b) shows the test error for various number of samples |D
s | with true label.
As expected,
In both extremes where |D s | is too small or too large, the performance of our model becomes close to the models without a teacher.
The reason is that student has enough strong samples to learn a good model of true function.
In more realistic cases where |D s |(cid:28)|D w | but |D s | is still large enough to be informative about |D w | , our model gives a lower test error than models without the intelligent teacher.
The theory of LUPI was ﬁrst developed and proved for support vector machines by Vapnik as a method for knowledge transfer.
Hinton introduced Dark knowledge as a spiritually close idea in the context of neural networks (Hinton et al., 2006).
He proposed to use a large network or an ensemble of networks for training and a smaller network at test time.
It turned out that compressing knowledge of a large system into a smaller system can improve the generalization ability.
It was shown in (Lopez-Paz et al., 2015) that dark knowledge and LUPI can be uniﬁed under a single umbrella, called generalized distillation .
The core idea of these models is machines-teaching-machines .
As the name suggests, a machine is learning the knowledge embedded in another machine.
In our case, student is correcting his knowledge by receiving privileged information about label uncertainty from teacher.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Our framework extends the core idea of LUPI in the following directions:
• Trainable teacher: It is often assumed that the teacher in LUPI framework has some additional true information.
We show that when this extra information is not available, one can still use the LUPI setup and deﬁne an implicit teacher whose knowledge is learned from the true data.
In this approach, the performance of the ﬁnal student-teacher system depends on a clever answer to the following question: which information should be considered as the privileged knowledge of teacher.
• Bayesian teacher: The proposed teacher is Bayesian.
It provides posterior uncertainty of the label of each sample.
• Mutual representation: We introduced module ψ ( . ) which learns a mutual embedding (representation) for both student and teacher.
This is in particular interesting because it deﬁnes a two-way channel between teacher and student.
• Multiple teachers: We proposed a scalable method to introduce several teachers such that each teacher is specialized in a particular region of the data space.
B D ETAILED DESCRIPTION OF CLUSTERED GP
We suggested using several GP =
{ GP c i } to cover the entire data space.
This results in a better specialization of each teacher.
In addition it solves the scalability issue when the size of the strong dataset D s on which we train GP is too large.
Here we propose a method called clustered Gaussian process inspired by (Shen et al., 2006) to alleviate the issue of large sample size.
Clustered GP : Let N be the size of the dataset on which we train the teacher.
Assume we allocate
K teachers to the entire data space.
Therefore, each GP sees a dataset of size n = N/K .
Then we use a simple clustering method (e.g. k-means) to ﬁnd centroids of K clusters C 1 ,C 2 ,...,C K where
C i consists of samples { x i, 1 ,x i, 2 ,...,x i,n } .
We take the centroid c i of cluster C i as the representative sample for all its content.
Note that c
i does not necessarily belong to { x i, 1 ,x i, 2 ,...,x i,n } .
We assign each cluster a GP trained by samples belonging to that cluster.
More precisely, cluster C i is assigned a GP whose data points are { x i, 1 ,x i, 2 ,...,x i,n } .
Because there is no dependency among different clusters, we train them in parallel to speed-up the procedure more.
The pseudo-code of the clustered
GP is presented in Algorithm 2.
When the main issue is computational resources, we can ﬁrst choose the number n which is the maximum size of the dataset on which our resources allow to train a GP , then ﬁnd the number of clusters K = N/n accordingly.
The rest of the algorithm remains unchanged.
C D ETAILED A RCHITECTURE OF THE S TUDENTS C. R ANKING T ASK
For the ranking task, the employed student is proposed in (Dehghani et al., 2017).
The ﬁrst layer of the network models function ψ that learns the representation of the input data samples, i.e. ( q,d + ,d − ) , and consists of three components: (1) an embedding function ε : V → R m (where V denotes the vocabulary set and m is the number of embedding dimensions), (2) a weighting function ω :
V → R , and (3) a compositionality function (cid:12) :(
R m , R ) n → R m .
More formally, the function ψ is deﬁned as:
[Equation]
where t qi and t di denote the i th term in query q respectively document d .
The embedding function ε maps each term to a dense m - dimensional real value vector, which is learned during the training phase.
The weighting function ω assigns a weight to each term in the vocabulary.
It has been shown that ω simulates the effect of inverse document frequency (IDF), which is an important feature in information retrieval (Dehghani et al., 2017).
The compositionality function (cid:12) projects a set of n embedding-weighting pairs to an m - dimensional representation, independent from the value of n :
[Equation]
which is in fact the normalized weighted element-wise summation of the terms’ embedding vectors.
Again, it has been shown that having global term weighting function along with embedding function improves the performance of ranking as it simulates the effect of inverse document frequency (IDF).
In our experiments, we initialize the embedding function ε with word2vec embeddings (Mikolov et al., 2013) pre-trained on Google News and the weighting function ω with IDF.
The representation layer is followed by a simple fully connected feed-forward network with l hidden layers followed by a softmax which receives the vector representation of the inputs processed by the representation learning layer and outputs a prediction ˜ y .
Each hidden layer z k in this network computes z k
= α ( W k z k − 1 + b k ) , where W k and b k denote the weight matrix and the bias term corresponding to the k th hidden layer and α ( . ) is the non-linearity.
These layers follow a sigmoid output.
We employ the cross entropy loss:
[Equation]
where B is a batch of data samples.
C. S ENTIMENT C LASSIFICATION T ASK
The student for the sentiment classiﬁcation task is a convolutional model which has been shown to perform best in the dataset we used (Deriu et al., 2017; Severyn & Moschitti, 2015a;b; Deriu et al., 2016).
The ﬁrst layer of the network learns the function ψ which maps input sentence s to a vector as its representation consists of an embedding function ε : V → R m , where V denotes the vocabulary set and m is the number of embedding dimensions.
This function maps the sentence to a matrix S ∈ R m ×| s | , where each column represents the embedding of a word at the corresponding position in the sentence.
Matrix S is passed through a convolution layer.
In this layer, a set of f ﬁlters is applied to a sliding window of length h over S to generate a feature map matrix C .
Each feature map c
i for a given ﬁlter F is generated by c i = (cid:80) k,j S
[ i : i + h ] k,j F k,j , where
S [ i : i + h ] denotes the concatenation of word vectors from position i to i
+ h .
The concatenation of all c i produces a feature vector c ∈ R | s |−
h
+1 .
The vectors c are then aggregated over all f ﬁlters into a feature map matrix C ∈ R f × ( | s |− h +1) .
We also add a bias vector b ∈ R f to the result of a convolution.
Each convolutional layer is followed by a non-linear activation function (we use ReLU(Nair & Hinton, 2010)) which is applied element-wise.
Afterward, the output is passed to the max pooling layer which operates on columns of the feature map matrix C returning the largest value: pool ( c i ) : R 1 × ( | s |− h +1)
→ R (see Figure 4).
This architecture is similar to the state-of-the-art model for Twitter sentiment classiﬁcation from Semeval 2015 and 2016 (Severyn & Moschitti, 2015b; Deriu et al., 2016).
We initialize the embedding matrix with word2vec embeddings (Mikolov et al., 2013) pretrained on a collection of 50M tweets 2 .
The representation layer then is followed by a feed-forward layer similar to the ranking task (with different width and depth) but with softmax instead of sigmoid as the output layer which returns ˆ y i , the probability distribution over all three classes.
We employ the cross entropy loss:
[Equation]
where B is a batch of data samples, and K is a set of classes.
D D ETAILED A RCHITECTURE OF THE T EACHERS
We use Gaussian Process as the teacher in all the experiments.
For each task, either regression or (multi-class) classiﬁcation, in order to generate soft labels, we pass the mean of GP through the same function g ( . )
that is applied on the output of the student network for that task, e.g. softmax, or sigmoid.
For binary classiﬁcation or one dimensional regression, Σ( x t ) is scalar and h ( . ) is identity.
For multi-class classiﬁcation or multi-dimensional regression tasks, h ( . ) is an aggregation function that takes variance over several dimensions and outputs a single measure of variance.
As a reasonable choice, the aggregating function h ( . ) in our sentiment classiﬁcation task (three classes) is mean of variances over dimensions.
In the teacher, linear combinations of different kernels are used for different tasks in our experiments.
Toy Problem: We use standard Gaussian process regression 3 with this kernel:
[Equation]
Document Ranking: We use sparse variational GP regression 4 (Titsias, 2009) with this kernel:
[Equation]
Sentiment Classiﬁcation: We use sparse variational GP for multiclass classiﬁcation 5 (Hensman et al., 2015) with the following kernel:
[Equation]
where,
[Equation]
k
[Equation]
constant value, ∀ x
1 = x 2 and
otherwise
We empirically found l =1 satisfying value for the length scale of RBF and Matern3/2 kernels.
We also set σ 0 =0 to obtain a homogeneous linear kernel.
The constant value of K White ( .,. ) determines the level of noise in the labels.
This is different from the noise in weak labels.
This term explains the fact that even in true labels there might be a trace of noise due to the inaccuracy of human labelers.
We set the number of clusters in the clustered GP algorithm for the ranking task to 50 and for the sentiment classiﬁcation task to 30 .
E W EAK A NNOTATORS E. D OCUMENT R ANKING
The weak annotator in the document ranking task is BM25 (Robertson & Zaragoza, 2009), a wellknown unsupervised retrieval method.
This method heuristically scores a given pair of query-document based on the statistics of their matched terms.
In the pairwise document ranking setup, ˜ y
i for a given sample x j = ( q, d + , d − ) is the probability of document d + being ranked higher than d − : ˜
y i
=
P q,d + ,d − =
s q,d + / s q,d + + s q,d − , where s q,d is the score obtained from the weak annotator.
E. S ENTIMENT C LASSIFICATION
The weak annotator for the sentiment classiﬁcation task is a simple lexicon-based method (Hamdan et al., 2013; Kiritchenko et al., 2014).
We use SentiWordNet03 (Baccianella et al., 2010) to assign probabilities (positive, negative and neutral) for each token in set D w .
We use a bag-of-words model for the sentence-level probabilities (i.e. just averaging the distributions of the terms), yielding a noisy label ˜
y i ∈ R | K | , where | K | =3 is the number of classes.
We found empirically that using soft labels from the weak annotator works better than assigning a single hard label.
F D ATA C OLLECTION , P ARAMETERS AND S ETUP F. T OY P ROBLEM
Weak/True Data
In all the experiments with the toy problem, we have randomly sampled 100 data points from the weak function and 10 data points from the true function.
We introduce a small amount of noise to the observation of the true function to model the noise in the human labeled data.
Setup The neural network employed in the toy problem experiments is a simple feed-forward network with the depth of 3 layers and width of 128 neurons per layer.
We have used tanh as the nonlinearity for the intermediate layers and a linear output layer.
As the optimizer, we used Adam (Kingma & Ba, 2014) and the initial learning rate has been set to 0 .
For the teacher in the toy problem, we ﬁt only one GP on all the data points (i.e. no clustering).
Also during ﬁne-tuning, we set β =1 .
Setup of experiments in Section 4.2 We ﬁxed everything in the model and tried running the ﬁnetuning stage with different values for β ∈ { 0 . 0 , 0 . 1 , 1 . 0 , 2 . 0 , 5 . 0 } in all the experiments.
For the experiments on toy problem in Section 4.2, the reported numbers are averaged over 10 trials.
In the ﬁrst experiment (i.e. Figure 6a), the size of sampled data data is: |D s | =50 and |D w | =100 (Fixed) and for the second one (i.e. Figure 6a): |D w | =100 and |D s | =10 (ﬁxed).
F. R ANKING T ASK
Collections We use two standard TREC collections for the task of ad-hoc retrieval: The ﬁrst collection ( Robust04 ) consists of 500k news articles from different news agencies as a homogeneous collection.
The second collection ( ClueWeb ) is ClueWeb09 Category B, a large-scale web collection with overmillion English documents, which is considered as a heterogeneous collection.
Spam documents were ﬁltered out using the Waterloo spam scorer 6 (Cormack et al., 2011) with the default threshold 70% .
Data with true labels We take query sets that contain human-labeled judgments: a set of 250 queries (TREC topics 301–450 and 601–700) for the Robust04 collection and a set of 200 queries (topics 1-200) for the experiments on the ClueWeb collection.
For each query, we take all documents judged as relevant plus the same number of documents judged as non-relevant and form pairwise combinations among them.
Data with weak labels We create a query set Q using the unique queries appearing in the AOL query logs (Pass et al., 2006).
This query set contains web queries initiated by real users in the AOL search engine that were sampled from a three-month period from March 2006 to May 2006.
We ﬁltered out a large volume of navigational queries containing URL substrings (“http”, “www.”, “.com”, “.net”, “.org”, “.edu”).
We also removed all non-alphanumeric characters from the queries.
For each dataset, we took queries that have at least ten hits in the target corpus using our weak annotator method.
Applying all these steps, We collect 6.15 million queries to train on in Robust04 and 6.87 million queries for
ClueWeb.
To prepare the weakly labeled training set D w , we take the top 1 , 000 retrieved documents using BM25 for each query from training query set Q , which in total leads to ∼ | Q |× 10 6 training samples.
Setup For the evaluation of the whole model, we conducted a 3-fold cross-validation.
However, for each dataset, we ﬁrst tuned all the hyper-parameters of the student in the ﬁrst step on the set with true labels using batched GP bandits with an expected improvement acquisition function (Desautels et al., 2014) and kept the optimal parameters of the student ﬁxed for all the other experiments.
The size and number of hidden layers for the student is selected from { 64 , 128 , 256 , 512 } .
The initial learning rate and the dropout parameter were selected from { 10 − 3 , 10 − 5 } and { 0 . 0 , 0 . 2 , 0 . 5 } , respectively.
We considered embedding sizes of { 300 , 500 } .
The batch size in our experiments was set to 128 .
We use ReLU (Nair & Hinton, 2010) as a non-linear activation function α in student.
We use the Adam optimizer (Kingma & Ba, 2014) for training, and dropout (Srivastava et al., 2014) as a regularization technique.
At inference time, for each query, we take the top 2 , 000 retrieved documents using BM25 as candidate documents and re-rank them using the trained models.
We use the Indri 7 implementation of BM with default parameters (i.e., k 1 =1 . 2 , b =0 .
75 , and k 3 =1 , 000 ).
F. S ENTIMENT C LASSIFICATION T ASK
Collections We test our model on the twitter message-level sentiment classiﬁcation of SemEvalTask 10B (Rosenthal et al., 2015).
Datasets of SemEval-15 subsume the test sets from previous editions of SemEval, i.e. SemEval-13 and SemEval-14.
Each tweet was preprocessed so that URLs and usernames are masked.
Data with true labels We use train (9,728 tweets) and development (1,654 tweets) data from SemEval13 for training and SemEval-13-test (3,813 tweets) for validation.
To make your results comparable to the ofﬁcial runs on SemEval we us SemEval-14 (1,853 tweets) and SemEval-15 (2,390 tweets) as test sets (Rosenthal et al., 2015; Nakov et al., 2016).
Data with weak labels We use a large corpus containing 50M tweets collected during two months for both, training the word embeddings and creating the weakly annotated set D w using the lexicon-based method explained in Section 3.3.
Setup Similar to the document ranking task, we tuned hyper-parameters for the student in the ﬁrst step with respect to the true labels of the validation set using batched GP bandits with an expected improvement acquisition function (Desautels et al., 2014) and kept the optimal parameters ﬁxed for all the other experiments.
The size and number of hidden layers for the classiﬁer and is selected from { 32 , 64 , 128 } .
We tested the model with both, 1 and 2 convolutional layers.
The number of convolutional feature maps and the ﬁlter width is selected from { 200 , 300 } and { 3 , 4 , 5 } , respectively.
The initial learning rate and the dropout parameter were selected from { 1 E − 3 , 1 E − 5 } and { 0 . 0 , 0 . 2 , 0 . 5 } , respectively.
We considered embedding sizes of { 100 , 200 } and the batch size in these experiments was set to 64 .
ReLU (Nair & Hinton, 2010) is used as a non-linear activation function in student.
Adam optimizer (Kingma & Ba, 2014) is used for training, and dropout (Srivastava et al., 2014) as a regularizer.