T RANSFER AND M ARGINALIZE : E XPLAINING A WAY L ABEL N OISE WITH P RIVILEGED I NFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time
e.g. the
ID of the annotator that provided the label.
We argue that privileged information is useful for explaining away label noise, thereby reducing the harmful impact of noisy labels.
We develop a simple and efﬁcient method for supervised neural networks: it transfers via weight sharing the knowledge learned with privileged information and approximately marginalizes over privileged information at test time.
Our method,
TRAM (TRansfer and Marginalize), has minimal training time overhead and has the same test time cost as not using privileged information.
TRAM performs strongly on CIFAR-10H, ImageNet and Civil Comments benchmarks.
I NTRODUCTION
Supervised learning problems are typically formalized as learning a conditional distribution p ( y | x ) , y 2 Y and x 2 X from ( x i , y i ) , i = 1 , ..., N pairs.
Yet we often have access to additional features a 2 A at training time that will not be available at test time.
These features are known as privileged information (Vapnik & Vashist, 2009), or PI for short.
An example of PI are features describing the human annotator that provided a given label, such as the annotator ID, the length of time to provide the label, the experience of the annotator, etc.
Annotators do not always agree on the correct label for a given x , some annotators may be more reliable than others and the reliability of annotators may depend on the location of x in the input domain X (Snow et al., 2008; Sheng et al., 2008).
The expanded training dataset consists of ( x i , a i , y i ) triplets.
Given that our test time predictive distribution cannot be conditioned on a , what use is this PI?
As a thought experiment, suppose there exists a malicious (or lazy) annotator that provides random labels.
It is known that random labels harm the performance of supervised learning models (Fr´enay & Verleysen, 2013; Song et al., 2020;
Cordeiro & Carneiro, 2020).
If these random labels can be explained away via access to PI, such as the annotator ID, then this harm can be prevented.
In particular we can use the PI to explain away noise in the labels which otherwise would be irreducible aleatoric uncertainty.
More formally, suppose A , the PI random variable, is predictive of Y given X , in the sense that the conditional mutual information I ( Y ; A | X ) is non-zero.
Then, the entropy of Y is reduced if we condition on both X and A rather than X alone, as summarised in Lemma 1.1.
Lemma 1.1.
I ( Y ; A | X ) > 0 )
H ( Y | X , A ) <
H ( Y | X ) .
In § 2.1 we examine this lemma for a particular model, proving that under certain conditions, PI can be leveraged to lower the expected risk for linear regression problems.
Additionally, prior work has proven that PI can lead to generalization bounds with better sample complexity (Vapnik & Vashist, 2009; Lambert et al., 2018).
Inspired by prior work and our theoretical analysis of a simple linear model, we focus on exploiting
PI in supervised deep neural networks.
The production deployment of such models often has tight latency and memory constraints.
Hence a number of methods have been developed to utilize PI with the same test time memory and computation cost as networks trained without PI (Yang et al., 2017;
Lambert et al., 2018; Lopez-Paz et al., 2015).
Yang et al.
(2017) uses PI as a form of input-dependent regularizer.
Lambert et al.
(2018) train with heteroscedastic Gaussian dropout, with the training-time dropout variance a function of the PI.
Lopez-Paz et al.
(2015) distill a network trained with PI into a network without access to a .
Below we develop a method, TRAM, which transfers knowledge via weight sharing from the part of the network trained using PI to the test time network which does not have access to PI.
At test time, TRAM makes a simple, efﬁcient approximation to the integral p ( y | x ) =
R p ( y | x , a ) p ( a | x ) d a .
Making predictions without PI is no more costly than that with a standard network trained without access to PI.
Unlike prior work which requires speciﬁc techniques such as Gaussian Dropout, we need not constrain the form of the predictors to make the downstream marginalization possible.
Implementation and training are simple.
In summary the paper contributions are the following:
• To better illustrate when PI is useful, we show analytically that, under certain conditions, PI reduces the expected risk for linear regression models.
• We provide empirical evidence suggesting that the representations learned with access to PI are more robust against label noise.
• We propose a novel efﬁcient method, TRAM, which exploits PI in supervised deep neural networks and has zero computational overhead at prediction time.
• Empirically, we show that our method performs better than a series of baselines on CIFAR10H, ImageNet and CivilComments benchmarks.
E XPLOITING P RIVILEGED I NFORMATION
To build up intuition and to better illustrate situations where PI can be useful, we start with a simple linear model where a formal analysis can be carried out.
Next, we look into non-linear models and provide a motivating experiment suggesting that useful PI can be leveraged in deep networks to improve representation learning.
2. W HEN CAN PI BE HELPFUL ? A N ANALYSIS IN A S IMPLE L INEAR M ODEL
We consider the following regression generative model with target y
[Equation]
where x 2 R d and a 2 R m correspond to standard and PI features respectively, while " ⇠ N (0 ,  2 ) stands for some additive noise.
The two unknown parameters ( w ? , v ? )
establish the relationships between the target y and the features ( x , a ) .
To model the fact that the PI features can themselves depend on the x —e.g., raters having diverging assessments on ambiguous input samples—we assume that a ⇠ p ( a | x ) = N ( µ ( x ) | ⌃ ( x )) for some mean and covariance dependent on x .
Let us assume we have n observations from this generative model represented by y 2 R n , X 2 R n ⇥ d ,
A 2 R n ⇥ m and µ ( X ) 2 R n ⇥ m .
We are interested in comparing different predictors ⌧ ( X ) that can predict y only based on X , as required in the case of PI.
To compare the predictors, we use the concept of risk E " ⇠ p ( " ) , a ⇠ p ( a | x ) [ R ( ⌧ ( X ))] , formally deﬁned in Appendix E, to capture the expected error of ⌧ in predicting y ; see Section 3.5 in Bach (2021) for more background about risk analysis.
We defer to Appendix E a rigorous exposition of the results and convey instead here some intuitive messages.
We ﬁrst focus on the comparison between
• (NO-PI) the least-square estimate ˆ w 0 =
( X > X )  1 X > y that ignores A , and • (PI) the joint least-square estimate [ ˆ w 1 ; ˆ v 1 ]
=
( Q > Q )  1 Q > y with Q =
[ X , A ] in R n ⇥ ( d + m ) .
At prediction time, if we had access to ( x test , a test ) for (PI) , we would predict with ˆ w > 1 x test + ˆ v > 1 a test .
However, since a test is not available in our context, we use instead its (assumed known) mean µ ( x test ) , i
e., mean imputation (Little & Rubin, 2019).
Denoting by ⇧ x the orthogonal projector associated with X , deﬁned in Appendix E, our analysis shows that as long as
• Variance of PI :
The variance { ( v ? ) >
⌃ ( x i ) v ? } ni =1 due to PI is large enough and/or • Alignment of PI :
The PI features A have a signiﬁcant average component outside of the subspace spanned by the features X , i.e., the term below is large enough
[Equation]
[Figure]
then the estimator (PI) has a lower risk compared to (NO-PI) .
In other words, it is provably better to exploit the privileged information A at training time instead of ignoring it.
Our analysis further covers the case of (marg.
NO-PI) where we marginalize ˆ w 0 with respect to
PI and we predict with X E a ⇠ p ( a | x ) [ ˆ w 0 ] , which we compare with (marg.
PI)
the marginalized predictions E a ⇠ p ( a | x ) [ X ˆ w 1 + A ˆ v 1 ] .
In that case, we can show the same conclusion as previously, with the exception that the variance term { ( v ? ) >
⌃ ( x i ) v ? } ni =1 does not have inﬂuence anymore, only (1) drives the comparison.
Indeed, the proof in Appendix E shows that marginalizing removes from the risk expressions the terms related to the variance of PI.
2. PI HELPS TO LEARN BETTER REPRESENTATIONS : A MOTIVATING EXPERIMENT
Our analysis of the effect of PI in a linear model has established key insights into the conditions under which PI is provably useful.
We now look at a toy non-linear learning experiment using neural networks, which provides empirical evidence that PI can be helpful in deep networks.
In particular, we show that representations learned with access to useful PI can explain away label noise and transfer better than representations learned without access to PI.
This motivating experiment forms the basis of our TRAM method.
We simulate a noisy annotator by assuming in this toy example that the PI is a binary indicator, a ⇠ Bernoulli(0 . 3) , with a = 1 representing the case where the noisy annotator provides a label independent of x :
[Equation]
where x 2 [0 , 1] , v ⇠ U (  1 , 1) and ✏ ⇠ N (0 , 0 . 1) .
We then ﬁt two networks to N = 2 , 500 training examples generated according to this process.
The ﬁrst network does not have access PI and is a two layer MLP, both layers of dimension 64, with tanh hidden activations and linear output activation; see top left of Fig.
1a for an illustration.
The second network has access to PI, and is deﬁned as per the two-step TRAM approach.
The part of the network which learns the x representation,  , is deﬁned exactly as the no-PI MLP.
q ( y | x , a ) , the output head with access to PI, see Fig. 1a, is a single layer MLP with 64 units, tanh activation and linear output layer, with the concatenation of a and  ( x ) as inputs; see bottom left of Fig. 1a.
Both networks are ﬁt for 10 epochs by the Adam optimizer (Kingma & Ba, 2014) with mean squared error loss function.
We then extract the non-linear representations of x learned by both networks and ﬁt a linear model on the dataset {  ( x i ) , y i } , i = 1 , . . .
, N .
The linear model can be solved exactly using the L2 regularized ordinary least squares solution.
We plot the results in Fig.
We see that the representations learned by the model with access to PI in step #1 enable a near perfect ﬁt to the true expected marginal distribution, E ( a ,y )
⇠ p ( a ,y | x ) [ y ] accross X space.
However without access to PI the noise term a · v cannot be explained away .
As a result, the linear model ﬁt on top of the representations learned without access to PI is substantially worse than the model ﬁt using the two step procedure.
We emphasize that both models have exactly the same capacity.
In Appendix F we extend this representation learning procedure to a large-scale image classiﬁcation case.
We learn a representation with and without access to PI on a relabeled version of ImageNet (details in § 5.2) using a ResNet50 (He et al., 2016).
We then freeze the representation and evaluate using a linear model.
Access to PI improves the representations learned.
M ETHOD : TRAM
We consider learning under privileged information (Vapnik & Vashist, 2009), LUPI .
Our proposed method, TRAM, consists of a single neural network with two output heads, providing predictions for both p ( y | x , a ) and p ( y | x ) ; see Fig. 2.
There are two key ingredients to TRAM; (i) the p ( y | x ) head is a simple, yet a provably valid, approximation to the marginal
R p ( y | x , a ) p ( a | x ) d a and (ii) a partition of the parameter space such that the neural network weights are shared between the two output heads, and that these shared weights are updated solely based on the lower variance gradients from the p ( y | x , a ) head which has access to PI.
Below we develop TRAM in the classiﬁcation setting, in § 3.4 we extend the method to the regression setting.
3. I NGREDIENT #1: M ARGINALIZE OVER PI AT TEST TIME
A natural probabilistic approach to LUPI is (i) to learn the conditional distribution p ( y | x , a ) during training and (ii) then, at test time, marginalize over the A domain, computing p ( y | x ) =
R p ( y | x , a ) p ( a | x ) d a (Lambert et al., 2018).
Predicting with the marginal p ( y | x ) at test time is motivated by the following key observation.
Consider the set of distributions Q over the C class labels, Q = { q ( y |· ) | 8 x 2 X , q ( y | x ) 2  C } where
C is the C -dimensional simplex.
Among all the distributions q 2 Q , the marginal distribution x 7! p ( y | x ) minimizes the following optimization problem:
[Equation]
See proof in Appendix B.
In words, p ( y | x ) is optimal in the sense that it minimizes the expected KL divergence to p ( y | x , a ) .
Access to p ( y | x ) enables minimizing the Bayes risk (Murphy, 2012) at test time.
Note further that the mean imputation scheme which provably reduces the expected risk for a linear model, § 2.1, corresponds precisely to marginalizing over PI at test time.
Directly computing p ( y | x ) has two problems; (i) it is typically intractable and (ii) p ( a | x ) is unknown and therefore must be learned, which is a challenging generative modelling problem in itself, especially as a may have mixed type features.
A Monte Carlo estimate of the integral using the samples from A in the training set is only feasible with the independence assumption p ( a | x )
= p ( a ) , so that p ( y | x ) reduces to
[Equation]
Unfortunately this independence assumption is often violated in practice.
In addition the memory and computational cost of MC estimation scales linearly in S , the number of MC samples.
This O ( S ) scaling is undesirable for production deployment with strict latency requirements.
Due to the challenge of computing the integral directly, we propose a simple approximation q ( y | x ; w ) to p ( y | x ) .
It exploits the property (2) of p ( y | x ) as the distribution minimizing the expected KL divergence to its conditional p ( y | x , a ) .
We choose q to be cheap to evaluate at test time.
For example, for a multi-class vanilla TRAM classiﬁer q ( y | x ; w ) =
softmax ( W  ( x )) .
3. I NGREDIENT #2: T RANSFER VIA WEIGHT SHARING
We partition the parameter space into four disjoint subsets;
Let  ( x ) be a feature extractor for x 2 X . 2.
Similarly, let (  ( x ) , a ) be a feature extractor jointly applied to (  ( x ) , a ) for ( x , a ) in X ⇥ A . 3.
The weights w parameterize the marginal distribution: q ( y | x ; w ) =
q ( y |  ( x ); w ) .
The weights u parameterize the conditional distribution q ( y | x , a ; u ) , namely q ( y | x , a ; u ) =
q ( y | (  ( x ) , a ); u ) .
Two-step approach.
Motivated by Eq.
(2), the connection between LUPI and multi-task learning (Jonschkowski et al., 2016) and our toy representation learning experiments, § 2.2, we consider the following two-step approach:
[Equation]
L ⇤ are arbitrary loss functions.
We assume  and are parameterized as neural networks, so min  , refers to optimizing the network weights.
Crucially  ? is the feature extractor learned in (3) with access to PI.
This weight sharing enables knowledge transfer to the network trained without PI.
Given Eq.
(2), we know that Eq.
(4) approximates the true marginal distribution p ( y | x ) (observe that the KL divergence in Eq.
(2) reduces to the cross-entropy loss function for L 2 when taking the one-hot training labels for p ( y | x , a ) ).
Merging the two steps.
To further simplify the above approach, we propose to merge Eq.
(3) and
Eq.
(4) into a single training procedure.
To that end, and reusing the terminology commonly used in deep-learning frameworks, let us deﬁne
[Equation]
which coincides with q ( y | x ) except that its gradient only depends on w .
For some  > 0 , we then consider:
[Equation]
⇤ as the joint training objective.
In practice, since the parameters of the two losses are partitioned, we can set  = 1 and fold instead the search over  into the search of the learning rate, hence not introducing an extra hyperparameter.
3. TRAM VARIANTS
Privileged information may only explain away some of label noise uncertainty.
Below we propose two TRAM variants which combine TRAM with existing noisy labels methods.
Het-TRAM.
Heteroscedastic classiﬁers are capable of modeling label noise that is input-dependent and have been successfully applied in this setting (Collier et al., 2021).
Further note that even in some cases where the conditional distribution q ( y | x , a ) is homoscedastic, the marginal q ( y | x ) is heteroscedastic, see Appendix C for details.
Hence we propose Het-TRAM , a TRAM variant in which q ( y | x ) is heteroscedastic.
This increases the expressiveness of q , improving the approximation in the second step of our optimization procedure,
Eq.
We implement the method of Collier et al.
(2021) to make q ( y | x ) heteroscedastic.
Distilled-TRAM.
Distillation (Hinton et al., 2015) is a technique for transferring knowledge between two neural networks.
Distillation has been previously applied to LUPI (Lopez-Paz et al., 2015).
In Distilled-TRAM we use the two step TRAM procedure, setting the loss function, L 1 in
Eq.
(3), to the distillation loss.
The teacher network is ﬁrst trained with access to PI and in the second step the distilled-TRAM model is trained.
3. R EGRESSION
We developed TRAM and Het-TRAM focussing on the classiﬁcation setting but our approach is trivial to generalize to regression problems.
In the regression case, we can choose the predictive distribution to be Gaussian, q ( y | x ) = N ( µ ( x ) ,  2 ( x )) .
For vanilla TRAM we can choose  2 ( x ) = 1 , while for Het-TRAM we can choose  2 ( x ) =
softplus ( w >   ( x )) .
µ and  2 are parameterized as neural networks with our shared feature extractor  ( x ) , similar to Kendall & Gal (2017).
L 1 in Eq.
(3) and L 2 in Eq.
(4) are replaced by the Gaussian negative log-likelihood.
R ELATED W ORK
Vapnik & Vashist (2009) develop a theoretic framework for the LUPI paradigm and introduce the
SVM+ method for training Support Vector Machines in this regime.
The slack variables for the
SVM+ constraints are a function of the PI.
SVM+ has been extended in the SVM literature (Lapin et al., 2014; Vapnik & Izmailov, 2015).
Jonschkowski et al.
(2016) provide a unifying framework that connects together multi-task learning, multi-view learning and LUPI.
Yang et al.
(2017) extend the SVM+ approach to neural network models with their MIML-FCN+ method.
The authors formulate a two-tower network similar to ours, but without weight sharing between the towers.
Both towers make independent predictions given x or a as inputs.
The tower with access to PI predicts the loss of the other tower and this prediction is regularized to be close to the true loss.
In this way the PI tower outputs a neural network analogue to the SVM+ slack variables.
Lambert et al.
(2018) utilize PI by making the training-time Gaussian-dropout variance (Kingma et al., 2015) a function of the PI.
At test time the PI is approximately marginalized over by removing the dropout.
Similarly Hern´andez-Lobato et al.
(2014) allow the additive Gaussian noise component of a heteroscedastic Gaussian Process Classiﬁer (Rasmussen & Williams, 2006) to be a function of the PI.
The classiﬁer is homoscedastic at test time.
Lopez-Paz et al. (2015) propose a distillation (Hinton et al., 2015) style approach to learning with
PI.
The teacher network is trained with access to PI.
In the distillation step the student network is given x as input and a convex combination of soft labels from the teacher network and true labels y as targets.
Xu et al.
(2020) extend and apply this distillation method to a recommender system.
TRAM implements knowledge transfer via weight sharing, performs efﬁcient approximate marginalization at test time and can be applied to many widely used architectures.
Lambert et al.
(2018) also share weights and approximate the marginal p ( y | x ) however they require the use of Gaussian dropout, which is not widely used.
The distillation and MIML-FCN+ methods do not transfer via weight sharing and do not approximate p ( y | x ) .
Distillation also requires a two-step training procedure.
See
Table 5, in Appendix F for a comparison of the key features of selected LUPI methods.
E XPERIMENTS
Our experiments tackle the general LUPI problem under label noise.
There are a few large-scale public datasets with PI
We thus use both real-world datasets with PI as well as synthesizing PI for a re-labelled version of ImageNet (Deng et al., 2009).
We evaluate a number of baselines in addition to our method.
• The “No PI” baseline is standard neural network training which directly learns p ( y | x ) and never uses PI.
• Zero and mean imputation learn p ( y | x , a ) at training time and substitute a = 0 and a = 1 N P i a i respectively at test time.
For mean imputation, averaging takes place after feature pre-processing, e.g., one-hot encoding of the annotator ID.
[Table] • The “Full marginalization” baseline is an expensive MC estimate of p ( y | x ) =
R p
( y | x , a ) p ( a | x ) d a at test time, see § 3.1 for details.
It is a gold standard (up to independence assumption error), impractical to compute in many applications.
• We also compare against distillation based approaches.
“Distillation No PI” is an ablation to see the effect of distillation alone, independent of PI, in which a network trained without access to PI is distilled into another network also without access to PI.
Prior work did not evaluate against these simple imputation baselines or full marginalization (LopezPaz et al., 2015; Yang et al., 2017; Lambert et al., 2018).
5. CIFAR-10H
One dataset with annotator features is CIFAR-10H (Peterson et al., 2019), which is a re-labelled version of the CIFAR-10 (Krizhevsky & Hinton, 2009) test set.
The new labels are provided by crowd-sourced human annotators.
We make use of three annotator features; the annotator ID, the reaction time of the annotator to provide the label and how much experience the annotator had with the task, as measured by the number of labels the annotator had previously provided.
As we only have annotator features for the CIFAR-10 test set, we use this as our training set and evaluate on the ofﬁcial training set.
As a result we have only 10,000 images for training.
To achieve reasonable performance we start from a MobileNet (Howard et al., 2017) pretrained on ImageNet.
Images have on average > 50 annotations each.
This is unrealistic for typical applications where 1-3 labels per example is more common.
Therefore, we subsample 16,400 labels (1.64 labels per example), see Appendix D for details of the subsampling procedure.
The subsampled labels agree with the true CIFAR-10 test set labels 79.4% of the time.
In Table 1 we see the results.
First, and as expected, using annotator features via TRAM, marginalization or the imputation methods provides a performance improvement over standard neural network training without PI.
Second, we see that TRAM performs on par with full marginalization (which uses 16,400 MC samples of a from the training set), despite having constant time compute and memory requirements w.r.t.
the number of MC samples for the full marginalization baseline.
Mean imputation is a strong baseline on CIFAR-10H. Het-TRAM improves over TRAM demonstrating the efﬁcacy of making q ( y | x , a ) heteroscedastic.
It is noteworthy that distillation using PI, (Lopez-Paz et al., 2015) does not improve over standard distillation without PI.
However Distilled-TRAM with makes use of PI for distillation but then performs approximate marginalization and transfer learning via weight sharing improves over the distillation baselines on both accuracy and log-likelihood metrics. 5.1. Q UALITATIVE ANALYSIS OF CIFAR-10H RESULTS
We qualitatively analyse how PI is helping improve the performance of TRAM on CIFAR-10H. The PI for CIFAR-10H does not contain a feature for annotator reliability. However the PI does contain features such as the annotator ID and the annotator reaction time from which it may be possible to
[Figure] [Table] learn to trust some annotators more than others. TRAM can learn output a less conﬁdent distribution for unreliable annotators, thus reducing the harmful impact of incorrect labels.
In Fig.
3 we show an analysis of the conﬁdence of the TRAM and No PI (i
[Table]
e., standard) models for each annotator in CIFAR-10H.
We see that the trend for TRAM is a strong linear relationship between the reliability of an annotator and the conﬁdence of the model, Fig. 3a.
The TRAM model is consistently more conﬁdent than the No PI model for reliable annotators while the No PI model is overconﬁdent for unreliable annotators, Fig.
3b 5. I MAGE N ET ILSVRC  In order to create a large-scale dataset with annotator features, we re-label the ImageNet ILSVRC training set by the following procedure.
We download 16 different models pre-trained on ImageNet, see Appendix D for further details.
We also add a 17th malicious annotator model which picks a label uniformly at random from the 1,000 ImageNet ILSVRC12 classes.
For each image in the training set we select the malicious annotator with 10% probability and otherwise sample one of the 16 models with equal probability.
We then sample a label from the predictive distribution of that model for that image.
This is the label used for training.
On average the sampled label agrees with the true
ImageNet label 68.3% of the time.
The annotator features are the model ID (a proxy for a human annotator ID) and the probability of the label assigned by the model (a proxy for the conﬁdence of a human annotator).
The ImageNet image is used as the non-privileged information x .
is randomly initialized ResNet-50 (He et al., 2016).
See Table 2 for the results.
The full marginalization baseline uses 1,000 MC samples of a from the training set.
The imputation baselines perform worse than not using PI, perhaps due to the imputed values having low density p ( a | x ) .
Again TRAM performs on par with full marginalization and HetTRAM has higher accuracy than both.
The ImageNet labels are known to exhibit heteroscedasticity [Table] (Collier et al., 2021), therefore we make both q ( y | x ) and q ( y | x , a ) heads heteroscedastic for HetTRAM.
Distilled-TRAM has signiﬁcantly better NLL and accuracy than the two distillation baselines
[Table]
5. C IVIL C OMMENTS
We further evaluate our method on a large-scale text classiﬁcation dataset.
Civil Comments 1 is a collection of comments from independent news websites annotated with 7 toxicity labels (identity attack, insult, obscene, severe toxicity, sexually explicit, threat, toxicity).
The Civil Comments
Identities subset of the Civil Comments data contains privileged information in the form ofattributes identiﬁed in the comment (male, female, christian and so on).
The Identities subset consists of 405,130 training examples, 21,293 validation examples and 21,577 test set examples.
The shared network  is a pre-trained Universal Sentence Encoder (Cer et al., 2018).
Table 3 contains the test set results.
We report negative log-likelihood and accuracy averaged over the 7 labels.
The
TRAM, Het-TRAM and imputation methods perform similarly well in terms of average accuracy, outperforming the No PI baseline as well as the Gaussian Dropout and full marginalization methods.
The poor accuracy of the full marginalization method is interesting to note.
The PI is directly derived from the non-PI (in the form of 24 identity human labelled attributes for the non-PI).
This is clearly a violation of the independence assumption required for a MC estimate of full marginalization to be computable.
The dependence of a on x is most clearly identiﬁable for the Civil Comments Identities dataset; as a result the relative performance of the full marginalization method is poorest on this dataset.
Further note that the TRAM and Het-TRAM methods have lower negative log-likelihood than all other baseline methods.
Standard distillation with no PI and Lopez-Paz et al.
(2015) style distillation where the teacher network is trained with PI does not provide a performance improvement over the no PI baseline.
Distilled-TRAM performs on par with vanilla TRAM.
C ONCLUSION
We introduced TRAM, a new method for LUPI in supervised neural networks.
TRAM (i) learns an efﬁcient, simple distribution to approximately marginalize over PI at test time and (ii) partitions the parameter space enabling transfer via weight sharing of the knowledge learned with access to
PI.
TRAM can be successfully combined with established methods for dealing with noisy labels; distillation (Distilled-TRAM) and heteroscedastic output layers (Het-TRAM).
We have analysed a linear model with PI where deriving analytic results are feasible.
In this setting we have shown the utility of using PI and ingredient #1 of our TRAM method, the marginalization over PI.
Using a toy low-dimensional problem we have further shown the effectiveness of ingredient #2 of our proposed
TRAM method, transfer learning via weight sharing of representations learned with access to PI.
We then have empirically validated the single-step TRAM procedure on larger-scale datasets in the image and text domain; CIFAR-10H, a noisy version of ImageNet and Civil Comments Identities
E THICS AND R EPRODUCIBILITY S TATEMENTS
Ethics.
We present a generic algorithm for classiﬁcation/regression with privileged information.
The method is not application speciﬁc and has minimal computational overhead.
We do not see ethical issues speciﬁc to our method beyond any already associated with LUPI methods and/or deep neural network classiﬁers.
Reproducibility.
We provide detailed information on the data generation process for the CIFAR10H and Imagenet datasets in Appendix D.1.
Hyperparameters and further details required to reproduce our results are provided in Appendix D.2.
We plan to release publicly Colab notebooks to reproduce our toy transfer learning and Civil Comments experiments.
We also plan to make public our Imagenet dataset in Tensorﬂow Dataset format to be used as a large-scale dataset with annotator features by researchers working on the LUPI problem.
Detailed derivations with conditions for our theoretical analysis are presented in Appendix E.