L EARNING FROM S AMPLES OF V ARIABLE Q UALITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing.
This creates a fundamental quality-versus-quantity trade-off in the learning process.
Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data?
We argue that if the learner could somehow know and take the label-quality into account, we could get the best of both worlds.
To this end, we introduce “ﬁdelity-weighted learning” (FWL), a semi-supervised student-teacher approach for training deep neural networks using weakly-labeled data.
FWL modulates the parameter updates to a student network, trained on the task we care about on a per-sample basis according to the posterior conﬁdence of its label-quality estimated by a teacher , who has access to limited samples with high-quality labels.
I NTRODUCTION
“ All samples are equal, but some samples are more equal than others.”
—InspiredbyGeorgeOrwellquote,AnimalFarm,
The success of deep neural networks to date depends strongly on the availability of labeled data and usually it is much easier and cheaper to obtain small quantities of high-quality labeled data and large quantities of unlabeled data.
For a large class of tasks, it is also easy to deﬁne one or more so-called “weak annotators” (Ratner et al., 2016), additional (albeit noisy) sources of weak supervision based on heuristics or weaker, biased classiﬁers trained on e.g. non-expert crowd-sourced data or data from different domains that are related.
While easy and cheap to generate, it is not immediately clear if and how these additional weakly-labeled data can be used to train a stronger classiﬁer for the task we care about.
Assuming we can obtain a large set of weakly-labeled data in addition to a much smaller training set of “strong” labels, the simplest approach is to expand the training set simply by including the weakly-supervised samples (all samples are equal).
Alternatively, one may pretrain on the weak data and then ﬁne-tune on strong data, which is one of the common practices in semi-supervised learning.
We argue that treating weakly-labeled samples uniformly (i.e. each weak sample contributes equally to the ﬁnal classiﬁer) ignores potentially valuable information of the label quality.
Instead, we introduce Fidelity-Weighted Learning (FWL), a Bayesian semi-supervised approach that leverages a small amount of data with true labels to generate a larger training set with conﬁdence-weighted weakly-labeled samples , which can then be used to modulate the ﬁne-tuning process based on the ﬁdelity (or quality) of each weak sample.
By directly modeling the inaccuracies introduced by the weak annotator in this way, we can control the extent to which we make use of this additional source of weak supervision: more for conﬁdently-labeled weak samples close to the true observed data, and less for uncertain samples further away from the observed data.
F IDELITY -W EIGHTED L EARNING (FWL)
In this section, we describe FWL.
We assume we are given a large set of unlabeled data samples, a heuristic labeling function called the weak annotator , and a small set of high-quality samples labeled by experts, called the strong dataset , consisting of tuples of training samples x i and their true labels y i ,
i.e. D s = { ( x i ,y i ) } .
We consider the latter to be observations from the true target function that we are trying to learn.
We use the weak annotator to generate labels for the unlabeled samples.
Generated labels are noisy due to the limited accuracy of the weak annotator.
This gives us the weak dataset consisting of tuples of training samples x i and their weak labels ˜
y i , i.e. D w = { ( x i , ˜ y i ) } .
Note that we can generate a large amount of weak training data D w at almost no cost using the weak annotator.
In contrast, we have only a limited amount of observations from the true function, i.e. |D s |(cid:28)|D w | .
[Figure]
Our proposed setup comprises a neural network called the student and a Bayesian function approximator called the teacher .
The training process consists of three phases which we summarize in Figure 1.
Step 1 Pre-train the student on D w using weak labels generated by the weak annotator .
The main goal of this step is to learn a task dependent representation of the data as well as pretraining the student.
The student function is a neural network consisting of two parts.
The ﬁrst part ψ ( . ) learns the data representation and the second part φ ( . ) performs the prediction task (e.g. classiﬁcation).
Therefore the overall function is ˆ y = φ ( ψ ( x i )) .
The student is trained on all samples of the weak dataset D w = { ( x i , ˜ y i ) } .
For brevity, in the following, we will refer to both data sample
x i and its representation ψ ( x i ) by x i when it is obvious from the context.
Step 2 Train the teacher on the strong data ( ψ ( x j ) ,
y j ) ∈
D s represented in terms of the student representation ψ ( . ) and then use the teacher to generate a soft dataset D sw consisting of (cid:104) sample , predicted label , conﬁdence (cid:105) for all data samples.
We use a Gaussian process as the teacher to capture the label uncertainty in terms of the student representation, estimated w.r.t the strong data.
A prior mean and covariance function is chosen for GP .
The learned embedding function ψ ( · ) in Step 1 is then used to map the data samples to dense vectors as input to the GP .
We use the learned representation by the student in the previous step to compensate lack of data in D s and the teacher can enjoy the learned knowledge from the large quantity of the weakly annotated data.
This way, we also let the teacher see the data through the lens of the student.
The GP is trained on the samples from D s to learn the posterior mean m post (used to generate soft labels) and posterior co-variance K post ( .,. )
(which represents label uncertainty) 1 .
We then create the soft dataset D sw
=
{ ( x t , ¯ y t ) } using the posterior GP , input samples x t from D w ∪ D s , and predicted labels ¯ y t with their associated uncertainties as computed T ( x t )
=
g ( m post ( x t )) and Σ( x t ) =
h ( K post ( x t ,x t )) .
The generated labels are called soft labels .
Therefore, we refer to D sw as a soft dataset.
g ( . ) transforms the output of GP to the suitable output space.
For example in classiﬁcation tasks, g ( . ) would be the softmax function to produce probabilities that sum up to one.
For multidimensional-output tasks where a vector of variances is provided by the GP , the vector K post ( x t ,x t ) is passed through an aggregating function h ( . ) to generate a scalar value for the uncertainty of each sample.
Note that we train GP only on the strong dataset D s but then use it to generate soft labels ¯ y t
= T ( x t ) and uncertainty Σ( x t ) for samples belonging to D sw =
D w ∪D s .
Step 3 Fine-tune the weights of the student network on the soft dataset, while modulating the magnitude of each parameter update by the corresponding teacher-conﬁdence in its label.
The student network of Step 1 is ﬁne-tuned using samples from the soft dataset D sw
= { ( x t , ¯ y t ) } where ¯ y t
= T ( x t ) .
The corresponding uncertainty Σ( x t ) of each sample is mapped to a conﬁdence value, and this is then used to determine the step size for each iteration of the stochastic gradient descent (SGD).
So, intuitively, for data points where we have true labels, the uncertainty of the teacher is almost zero, which means we have high conﬁdence and a large step-size for updating the parameters.
However, for data points where the teacher is not conﬁdent, we down-weight the training steps of the student.
This means that at these points, we keep the student function as it was trained on the weak data in Step 1.
[Table]
More speciﬁcally
[Table]
we update the parameters of the student by training on D sw using SGD:
[Equation]
where l ( · ) is the per-example loss, η t is the total learning rate, N is the size of the soft dataset D sw , www is the parameters of the student network, and R ( . ) is the regularization term.
We deﬁne the total learning rate as η t = η 1 ( t )
η 2 ( x t ) , where η 1 ( t ) is the usual learning rate of our chosen optimization algorithm that anneals over training iterations, and
η 2 ( x t ) is a function of the label uncertainty Σ( x t ) that is computed by the teacher for each data point.
Multiplying these two terms gives us the total learning rate.
In other words, η 2 represents the ﬁdelity (quality) of the current sample, and is used to multiplicatively modulate η 1 .
Note that the ﬁrst term does not necessarily depend on each data point, whereas the second term does.
We propose η 2 ( x t )=
exp[ − β Σ( x t )] to exponentially decrease the learning rate for data point x t if its corresponding soft label ¯ y t is unreliable (far from a true sample).
In practice, when using mini-batches, we implement this by multiplying the loss of each example in the batch by its ﬁdelity score and average over these ﬁdelity-weighted losses in the batch when calculating the batch gradient based on that loss.
β is a positive scalar hyper-parameter that controls the contribution of weak and strong data to the training procedure.
A small β results in a student which listens more carefully to the teacher and copies its knowledge, while a large β makes the student pay less attention to the teacher, staying with its initial weak knowledge.
Hence, β gives a handle to control the bias-variance trade-off.
In Appendix A, we apply FWL to a one-dimensional toy problem to illustrate its various steps.
E XPERIMENTS
[Figure]
In this section, we apply FWL to document ranking task and evaluate its performance compared to the baselines presented in Table 1.
Document Ranking is the core information retrieval problem and is challenging as the ranking model needs to learn a representation for long documents and capture the complex notion of relevance between queries and documents.
Furthermore, the size of publicly available datasets with query-document relevance judgments is unfortunately quite small ( ∼ 250 queries).
We employ a state-of-the-art pairwise neural ranker architecture as the student (Dehghani et al., 2017b) in which the ranking is cast as a regression task.
Given each training sample x as a triple of query q , and two documents d + and d − , the goal is to learn a function F : { <q,d + ,d − > }→ R , which maps each data sample x to a scalar output value y indicating the probability of d + being ranked higher than d − with respect to q .
The student follows the architecture proposed in (Dehghani et al., 2017b).
The ﬁrst layer of the network, i.e. representation learning layer ψ : { <q,d + ,d − > }→ R m maps each input sample to an mdimensional real-valued vector.
In general, besides learning embeddings for words, function ψ learns to compose word embedding based on their global importance in order to generate query/document embeddings.
The representation layer is followed by a simple fully-connected feed-forward network with a sigmoidal output unit to predict the probability of ranking d + higher than d − .
The general schema of the student is illustrated in Figure 2.
More details are provided in Appendix C.1.
The teacher is implemented by clustered GP algorithm.
See Appendix C.2 for more details.
The weak annotator is BM25 (Robertson & Zaragoza, 2009), a well-known unsupervised method for
[Table] [Table]
scoring query-document pairs based on statistics of the matched terms.
More details are provided in
Appendix C.3.
Description of the data with weak labels and data with true labels as well as the setup of the document-ranking experiments is presented in Appendix C.4 in more details.
Results and Discussions.
We conducted k-fold cross-validation on D s (the strong data) and report two standard evaluation metrics for ranking: mean average precision (MAP) of the top-ranked 1 ,documents and normalized discounted cumulative gain calculated for the top 20 retrieved documents (nDCG@20).
Table 2 shows the performance on both datasets.
As can be seen, FWL provides a signiﬁcant boost on the performance over all datasets.
In the ranking task, the student is designed in particular to be trained on weak annotations (Dehghani et al., 2017b), hence training the network only on weak supervision, i.e. NN W performs better than NN S .
This can be due to the fact that ranking is a complex task requiring many training samples, while relatively few data with true labels are available.
Alternating between strong and weak data during training, i.e. NN S + /W seems to bring little (but statistically signiﬁcant) improvement.
However, we can gain better results by the typical ﬁne-tuning strategy, NN W → S .
We can gain improvement by ﬁne-tuning the NN W using labels generated by the teacher without considering their conﬁdence score, i.e. FWL \ Σ .
This means we just augmented the ﬁne-tuning process by generating a ﬁne-tuning set using teacher which is better than D s in terms of quantity and D w in terms of quality.
This baseline is equivalent to setting β =0 .
However, we see a big jump in performance when we use FWL to include the estimated label quality from the teacher, leading to the best overall results.
Sensitivity of the FWL to the Quality of the Weak Annotator.
Our proposed setup in FWL requires deﬁning a socalled “weak annotator” to provide a source of weak supervision for unlabelled data.
In this section, we study how the quality of the weak annotator may affect the performance of the FWL on the Robust04 dataset.
To do so, besides
BM25 (Robertson & Zaragoza, 2009), we use three other weak annotators: vector space model (Salton & Yang, 1973) with binary term occurrence (BTO) weighting schema and vector space model with TF-IDF weighting schema, which are both weaker than BM25, and BM25+RM3 (Abduljaleel et al., 2004) that uses RM3 as the pseudo-relevance feedback method on top of BM25, leading to better labels.
Figure 3 illustrates the performance of these four weak annotators in terms of their mean average precision (MAP) on the test data, versus the performance of FWL given the corresponding weak annotator.
As it is expected, the performance of
FWL depends on the quality of the employed weak annotator.
The percentage of improvement of FWL over its corresponding weak annotator on the test data is also presented in Figure 3.
As can be seen, the better the performance of the weak annotator is, the less the improvement of the FWL would be.
C ONCLUSION
Training neural networks using large amounts of weakly annotated data is an attractive approach in scenarios where an adequate amount of data with true labels is not available, a situation which often arises in practice.
In this paper, we introduced ﬁdelity-weighted learning (FWL), a new student-teacher framework for semi-supervised learning in the presence of weakly labeled data.
We applied FWL to document ranking and empirically veriﬁed that FWL speeds up the training process and improves over state-of-the-art
semi-supervised alternatives.
A PPENDICES A T OY E XAMPLE
To better understand FWL, we apply FWL to a one-dimensional toy problem to illustrate the various steps.
Let f t ( x ) = sin( x ) be the true function (red dotted line in Figure 4a) from which a small set of observations D
s = { x j ,y j } is provided (red points in Figure 4b).
These observation might be noisy, in the same way that labels obtained from a human labeler could be noisy.
A weak annotator function f w ( x )=
2 sinc ( x ) (magenta line in
Figure 4a) is provided, as an approximation to f t ( . ) .
The task is to obtain a good estimate of f t ( . ) given the set D s of strong observations and the weak annotator function f w ( . ) .
We can easily obtain a large set of observations D w = { x i , ˜ y i } from f w ( . ) with almost no cost (magenta points in Figure 4a).
As the teacher, we use standard Gaussian process regression 2 with this kernel:
[Equation]
) where,
[Equation]
= constant value, ∀ x
1 = x 2 and
otherwise
We ﬁt only one GP on all the data points (i.e. no clustering).
Also during ﬁne-tuning, we set β =1 .
The student is a simple feed-forward network with the depth of 3 layers and width of 128 neurons per layer.
We have used tanh as the nonlinearity for the intermediate layers and a linear output layer.
As the optimizer, we used Adam (Kingma & Ba, 2015) and the initial learning rate has been set to 0 .
We randomly sample 100 data points from the weak annotator and 10 data points from the true function.
We introduce a small amount of noise to the observation of the true function to model the noise in the human labeled data.
[Figure]
We consider two experiments:
A neural network trained on weak data and then ﬁne-tuned on strong data from the true function, which is the most common semi-supervised approach (Figure 4c).
A teacher-student framework working by the proposed FWL approach.
As can be seen in Figure 4d, FWL by taking into account label conﬁdence, gives a better approximation of the true hidden function.
We repeated the above experiment 10 times.
The average RMSE with respect to the true function on a set of test points over those 10 experiments for the student, were as follows:
Let N
[Equation]
where c i represents the center of cluster C i containing samples
D c i s =
{ x i, 1 ,x i, 2 ,...x i,n } .
3: Assign each of K clusters a Gaussian process and train them in parallel to approximate the label of each sample.
[Equation]
where GP c i is trained on D c
i s containing samples belonging to the cluster c i .
Other elements are deﬁned in Section4: Use trained teacher T c i ( . ) to evaluate the soft label and uncertainty for samples from D sw to compute η 2 ( x t ) required for step 3 of FWL.
We use T ( . ) as a wrapper for all teachers { T c i } .
Student is trained on weak data (blue line in Figure 4a): 0 . 8406 , 2.
Student is trained on weak data then ﬁne tuned on true observations (blue line in Figure 4c): 0 . 5451 , 3.
Student is trained on weak data, then ﬁne tuned by soft labels and conﬁdence information provided by the teacher (blue line in Figure 4d): 0 .
4143 (best).
B D ETAILED DESCRIPTION OF CLUSTERED GP
We suggest using several GP =
{ GP c i } to explore the entire data space more effectively.
Even though inducing points and stochastic methods make GP s more scalable we still observed poor performance when the entire dataset was modeled by a single GP .
Therefore, the reason for using multiple GP s is mainly empirically inspired by (Shen et al., 2006) which is explained in the following:
We used the Sparse Gaussian Process implemented in GPﬂow.
The algorithm is scalable in the sense that it is not O ( N 3 ) as original GP is.
It introduces inducing points in the data space and deﬁnes a variational lower bound for the marginal likelihood.
The variational bound can now be optimized by stochastic methods which make the algorithm applicable in large datasets.
However, the tightness of the bound depends on the location of inducing points which are found through the optimization process.
The pseudo-code of the clustered GP is presented in Algorithm 1.
When the main issue is computational resources (when the number of inducing points for each GP is large), we can ﬁrst choose the number n which is the maximum size of the dataset on which our resources allow to train a GP , then ﬁnd the number of clusters K = N/n accordingly.
The rest of the algorithm remains unchanged.
C D ETAILED S ETUP OF THE M ODEL AND E XPERIMENTS C. D ETAILED A RCHITECTURE OF THE S TUDENTS
The employed student is proposed in (Dehghani et al., 2017b).
The ﬁrst layer of the network models function ψ that learns the representation of the input data samples, i.e. ( q,d + ,d − ) , and consists of three components: (1) an embedding function ε : V → R m (where V denotes the vocabulary set and m is the number of embedding dimensions), (2) a weighting function ω :
V → R , and (3) a compositionality function (cid:12) :(
R m , R ) n → R m .
More formally, the function ψ is deﬁned as:
[Equation]
) where t qi and t di denote the i th term in query q respectively document d .
The embedding function ε maps each term to a dense m - dimensional real value vector, which is learned during the training phase.
The weighting function ω assigns a weight to each term in the vocabulary.
It has been shown that ω simulates the effect of inverse document frequency (IDF), which is an important feature in information retrieval (Dehghani et al., 2017b).
The compositionality function (cid:12) projects a set of n embedding-weighting pairs to an m - dimensional representa- tion, independent from the value of n :
[Equation]
which is in fact the normalized weighted element-wise summation of the terms’ embedding vectors.
Again, it has been shown that having global term weighting function along with embedding function improves the performance of ranking as it simulates the effect of inverse document frequency (IDF).
In our experiments, we initialize the embedding function ε with word2vec embeddings (Mikolov et al., 2013) pre-trained on Google News and the weighting function ω with IDF.
The representation layer is followed by a simple fully connected feed-forward network with l hidden layers followed by a softmax which receives the vector representation of the inputs processed by the representation learning layer and outputs a prediction ˜ y .
Each hidden layer z k in this network computes z k
= α ( W k z k − 1 + b k ) , where W k and b k denote the weight matrix and the bias term corresponding to the k th hidden layer and α ( . ) is the non-linearity.
These layers follow a sigmoid output.
We employ the cross entropy loss:
[Equation]
where B is a batch of data samples.
C. D ETAILED A RCHITECTURE OF THE T EACHERS
We use Gaussian Process as the teacher and pass the mean of GP through the same function g ( . ) that is applied on the output of the student network.
h ( . ) is an aggregation function that takes variance over several dimensions and outputs a single measure of variance.
As a reasonable choice, the aggregating function h ( . ) in our sentiment classiﬁcation task (three classes) is mean of variances over dimensions.
In the teacher, linear combinations of different kernels are used in our experiments.
We use sparse variational GP regression 3 (Titsias, 2009) with this kernel:
[Equation]
where,
[Equation]
x
[Equation]
x 2 and 0 otherwise
We empirically found l =1 satisfying value for the length scale of Matern3/2 kernels.
We also set σ 0 =0 to obtain a homogeneous linear kernel.
The constant value of K White ( .,. ) determines the level of noise in the labels.
This is different from the noise in weak labels.
This term explains the fact that even in true labels there might be a trace of noise due to the inaccuracy of human labelers.
We set the number of clusters in the clustered GP algorithm for the ranking task to 50 .
C. W EAK A NNOTATORS
The weak annotator is BM25 (Robertson & Zaragoza, 2009), a well-known unsupervised retrieval method.
This method heuristically scores a given pair of query-document based on the statistics of their matched terms.
In the pairwise document ranking setup, ˜ y
i for a given sample x j =( q,d + ,d − ) is the probability of document d + being ranked higher than d − : ˜
y i
=
P q,d + ,d − =
s q,d + / s q,d + + s q,d − , where s q,d is the score obtained from the weak annotator.
C.
[Equation]
OLLECTION
[Equation]
AND S ETUP
Collections We use two standard TREC collections for the task of ad-hoc retrieval: The ﬁrst collection ( Robust04 ) consists of 500k news articles from different news agencies as a homogeneous collection.
The second collection
( ClueWeb ) is ClueWeb09 Category B, a large-scale web collection with over 50 million English documents, which is considered as a heterogeneous collection.
Spam documents were ﬁltered out using the Waterloo spam scorer 4 (Cormack et al., 2011) with the default threshold 70% .
Data with true labels We take query sets that contain human-labeled judgments: a set of 250 queries (TREC topics301–450and601–700)fortheRobust04collectionandasetof200queries(topics1-200)fortheexperiments ontheClueWebcollection.
Foreachquery,wetakealldocumentsjudgedasrelevantplusthesamenumberofdocumentsjudgedasnon-relevantandformpairwisecombinationsamongthem.
Data with weak labels We create a query set Q using the unique queries appearing in the AOL query logs (Pass et al., 2006).
This query set contains web queries initiated by real users in the AOL search engine that were sampled from a three-month period from March 2006 to May 2006.
We applied standard pre-processing Dehghani et al.
(2017b;a) on the queries: We ﬁltered out a large volume of navigational queries containing URL substrings (“http”, “www.”, “.com”, “.net”, “.org”, “.edu”).
We also removed all non-alphanumeric characters from the queries.
For each dataset, we took queries that have at least ten hits in the target corpus using our weak annotator method.
Applying all these steps, We collect 6.15 million queries to train on in Robust04 and 6.87 million queries for ClueWeb.
To prepare the weakly labeled training set D w , we take the top 1 , 000 retrieved documents using BM for each query from training query set Q , which in total leads to ∼| Q |× 10 6 training samples.
Setup For the evaluation of the whole model, we conducted 3-fold cross-validation.
However, for each dataset, we ﬁrst tuned all the hyper-parameters of the student in the ﬁrst step on the set with true labels using batched GP bandits with an expected improvement acquisition function (Desautels et al., 2014) and kept the optimal parameters of the student ﬁxed for all the other experiments.
The size and number of hidden layers for the student is selected from { 64 , 128 , 256 , 512 } .
The initial learning rate and the dropout parameter were selected from { 10 − 3 , 10 − 5 } and { 0 . 0 , 0 . 2 , 0 . 5 } , respectively.
We considered embedding sizes of { 300 , 500 } .
The batch size in our experiments was set to 128 .
We use ReLU (Nair & Hinton, 2010) as a non-linear activation function α in student.
We use the Adam optimizer (Kingma & Ba, 2015) for training, and dropout (Srivastava et al., 2014) as a regularization technique.
At inference time, for each query, we take the top 2 , 000 retrieved documents using BM25 as candidate documents and re-rank them using the trained models.
We use the Indri 5 implementation of BM25 with default parameters (i.e., k 1 =1 . 2 , b =0 .
75 , and k 3 =1 , 000 ).