Original: Toward Understanding Privileged Features Distillation in Learning-to-Rank
Revised: Toward Understanding Privileged Features Distillation in Learning-to-Rank

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In learning-to-rank problems, a privileged feature is one that is available duringmodel training, but not available at test time.
Revised: In learning-to-rank problems, a privileged feature is one that is available during model training, but not available at test time.

Original: Such features naturally arise inmerchandised recommendation systems; for instance, “user clicked this item” as afeature is predictive of “user purchased this item” in the ofﬂine data, but is clearlynot available during online serving.
Revised: Such features naturally arise in merchandised recommendation systems; for instance, “user clicked this item” as a feature is predictive of “user purchased this item” in the offline data, but is clearly not available during online serving.

Original: Another source of privileged features is thosethat are too expensive to compute online but feasible to be added ofﬂine.
Revised: Another source of privileged features is those that are too expensive to compute online but feasible to be added offline.

Original: Privilegedfeatures distillation (PFD) refers to a natural idea: train a “teacher” model using allfeatures (including privileged ones) and then use it to train a “student” model thatdoes not use the privileged features.
Revised: Privileged features distillation (PFD) refers to a natural idea: train a “teacher” model using all features (including privileged ones) and then use it to train a “student” model that does not use the privileged features.

Original: In this paper, we ﬁrst study PFD empirically on three public ranking datasetsand an industrial-scale ranking problem derived from Amazon’s logs.
Revised: In this paper, we first study PFD empirically on three public ranking datasets and an industrial-scale ranking problem derived from Amazon’s logs.

Original: We showthat PFD outperforms several baselines (no-distillation, pretraining-ﬁnetuning,self-distillation, and generalized distillation) on all these datasets.
Revised: We show that PFD outperforms several baselines (no-distillation, pretraining-finetuning, self-distillation, and generalized distillation) on all these datasets.

Original: Next, we analyze why and when PFD performs well via both empirical ablation studies andtheoretical analysis for linear models.
Revised: Next, we analyze why and when PFD performs well via both empirical ablation studies and theoretical analysis for linear models.

Original: Both investigations uncover an interestingnon-monotone behavior: as the predictive power of a privileged feature increases,the performance of the resulting student model initially increases but then decreases.
Revised: Both investigations uncover an interesting non-monotone behavior: as the predictive power of a privileged feature increases, the performance of the resulting student model initially increases but then decreases.

Original: We show the reason for the later decreasing performance is that a very predictiveprivileged teacher produces predictions with high variance, which lead to highvariance student estimates and inferior testing performance.
Revised: We show the reason for the later decreasing performance is that a very predictive privileged teacher produces predictions with high variance, which lead to high variance student estimates and inferior testing performance.

Original:  Introduction
Revised:  Introduction

Original: For recommendation systems, the features at test time are typically a subset of features availableduring training.
Revised: For recommendation systems, the features at test time are typically a subset of features available during training.

Original: Those missing features at test time are either too expensive to compute in real-time,or they are post-event features.
Revised: Those missing features at test time are either too expensive to compute in real-time, or they are post-event features.

Original: For instance, for an e-commerce website, “click” is a strong featurefor predicting “purchase”, but “click” exists as a feature only in the ofﬂine training data but notduring online serving (i e., one cannot observe “click” before recommendations are generated).
Revised: For instance, for an e-commerce website, “click” is a strong feature for predicting “purchase”, but “click” exists as a feature only in the offline training data, but not during online serving (i.e., one cannot observe “click” before recommendations are generated).

Original: Thosefeatures that exist only during training are called privileged features .
Revised: Those features that exist only during training are called privileged features .

Original: Those that exist during bothtraining and testing are called regular features [XLG + 20].
Revised: Those that exist during both training and testing are called regular features [XLG + 20].

Original: The naive approach is to ignore the privileged features and train a model that only takes regularfeatures.
Revised: The naive approach is to ignore the privileged features and train a model that only takes regular features.

Original: Such methods inevitably miss the information in the privileged features and lead to inferiorperformance.
Revised: Such methods inevitably miss the information in the privileged features and lead to inferior performance.

Original: A natural instinct to resolve this is to (a) use the privileged features (either by themselves[LPSBV16] or in conjunction with regular features [XLG + 20]) to train a “teacher” model, and then(b) use it to transfer information via distillation 1 into a “student” model that only uses the regular features.
Revised: A natural instinct to resolve this is to (a) use the privileged features (either by themselves [LPSBV16] or in conjunction with regular features [XLG + 20]) to train a “teacher” model, and then (b) use it to transfer information via distillation 2 into a “student” model that only uses the regular features.

Original: The approach of a teacher only using privileged features is named generalized distillation(GenD) [LPSBV16], and the approach of a teacher using both privileged and regular features hasbeen referred to as privileged feature distillation (PFD) [XLG + 20].
Revised: The approach of a teacher only using privileged features is named generalized distillation (GenD) [LPSBV16], and the approach of a teacher using both privileged and regular features has been referred to as privileged feature distillation (PFD) [XLG + 20].

Original: In this paper we provide a detailed investigation – ﬁrst via empirical ablation studies on moderatescale public and industrial-scale proprietary datasets with deep-learning-to-rank models, and secondvia rigorous theoretical analysis on simple linear models – into why and when privileged featuredistillation works and when it does not. 
Revised: In this paper we provide a detailed investigation – first via empirical ablation studies on moderatescale public and industrial-scale proprietary datasets with deep-learning-to-rank models, and second via rigorous theoretical analysis on simple linear models – into why and when privileged feature distillation works and when it does not. While this paper focuses on learning-to-rank, our results apply to regression/classification problems in general.

Original: As a summary, our main contributions are:
Revised: As a summary, our main contributions are:

Original: • We evaluate PFD on three moderate-scale public ranking datasets: Yahoo, Istella, and MSLRWeb30k
Revised: • We evaluate PFD on three moderate-scale public ranking datasets: Yahoo, Istella

Original: and  an industrial-scale proprietary dataset derived from Amazon search logs.•
Revised: and MSLRWeb30k, and an industrial-scale proprietary dataset derived from Amazon search logs.

Original: In all evaluated settings, PFD is better than or as good as the baselines: no-distillation, GenD(teacher model only uses privileged features), self-distillation (teacher model only uses regularfeatures), and pretraining on privileged features then ﬁnetuning (when applicable) 
Revised: In all evaluated settings, PFD is better than or as good as the baselines: no-distillation, GenD (teacher model only uses privileged features), self-distillation (teacher model only uses regular features), and pretraining on privileged features then finetuning (when applicable) (Table 2).

Original: (Table 2).• We conduct comprehensive ablation studies for PFD.
Revised: • We conduct comprehensive ablation studies for PFD.

Original: We ﬁnd that– PFD is effective as long as the teacher loss dominates the distillation loss and the performanceis not sensitive to ↵ .
Revised: We find that – PFD is effective as long as the teacher loss dominates the distillation loss and the performance is not sensitive to α .

Original: Speciﬁcally, distillation loss is a linear combination of the loss w.r
Revised: Specifically, distillation loss is a linear combination of the loss w.r

Original: t. dataand the loss w.r.t.
Revised: t. data and the loss w.r

Original:  teacher predictions and ↵ is the mixing ratio (Figure 3).
Revised: t. teacher predictions and α is the mixing ratio (Figure 3).

Original: – While it is known that the gains from self-distillation (over a no-distillation one-shot trainingbaseline) are larger when the positive labels are sparser, we see that these gains are furtherampliﬁed by PFD; i e. the relative gain of PFD over self-distillation also increases as thelabels become sparser (Figure 4).
Revised: – While it is known that the gains from self-distillation (over a no-distillation one-shot training baseline) are larger when the positive labels are sparser, we see that these gains are further amplified by PFD; i.e. the relative gain of PFD over self-distillation also increases as the labels become sparser (Figure 4).

Original: – Non-monotonicity in the effectiveness of PDF: as the predictive power of a privileged featureincreases, the resulting student performance initially increases but then decreases (Figure 5).• To provide a deeper insight into the landscape of privileged features and distillation, we next rigorously
Revised: – Non-monotonicity in the effectiveness of PFD: as the predictive power of a privileged feature increases, the resulting student performance initially increases but then decreases (Figure 5). • To provide a deeper insight into the landscape of privileged features and distillation

Original: analyze it in

Original:  a stylized setting involving linear models.
Revised: we next rigorously analyze it in a stylized setting involving linear models.

Original: We show that – PFD works because the teacher can explain away the variance arising from the privileged features, thus allowing the student to focus on the part it can predict.
Revised: We show that – PFD works because the teacher can explain away the variance arising from the privileged features, thus allowing the student to focus on the part it can predict.

Original: (Theorem 1).
Revised: (Theorem 1).

Original: – The reason that GenD is inferior to PFD (as seen in our empirical evaluation) is because itresults in a weaker teacher, and also because in the case where the privileged and regularfeatures are independent, the teacher predictions appear as pure noise to the student (whocannot learn from them)
Revised: – The reason that GenD is inferior to PFD (as seen in our empirical evaluation) is because it results in a weaker teacher, and also because in the case where the privileged and regular features are independent, the teacher predictions appear as pure noise to the student (who cannot learn from them)

Original: (Remark 2).
Revised: (Remark 2).

Original: – A very predictive privileged feature induces high variance teacher predictions, which lead toinaccurate student estimates and inferior testing performance.
Revised: – A very predictive privileged feature induces high variance teacher predictions, which lead to inaccurate student estimates and inferior testing performance.

Original: This explains the observationthat the most predictive privileged features do not give the best performance (i
Revised: This explains the observation that the most predictive privileged features do not give the best performance (i

Original: e., the non- monotonicity) in our empirical ablation studies (Theorem 2).
Revised: e., the nonmonotonicity) in our empirical ablation studies (Theorem 2).

Original: The rest of the paper is organized as follows: Section 2 covers related works.
Revised: The rest of the paper is organized as follows: Section 2 covers related works.

Original: Section 3 introduces theproblem setup, the PFD algorithm and other algorithms for comparison.
Revised: Section 3 introduces the problem setup, the PFD algorithm and other algorithms for comparison.

Original: Section 4 presents empiricalevaluation and ablation studies of PFD; and Section 5 presents theoretical insights.
Revised: Section 4 presents empirical evaluation and ablation studies of PFD; and Section 5 presents theoretical insights.

Original:  Related Work
Revised:  Related Work

Original: Privileged features widely exist in different machine learning problems, including speechrecognization [MM16], medical imaging [GCA + 19], image super-resolution [LLKH20], etc[FA12, FTRS13, FKSH14, ALL17].
Revised: Privileged features widely exist in different machine learning problems, including speech recognization [MM16], medical imaging [GCA + 19], image super-resolution [LLKH20], etc [FA12, FTRS13, FKSH14, ALL17].

Original: Privileged features are not accessible during testing eitherbecause they are too expensive to compute in real time, or because they are post-event features (thuscannot be used as input)
Revised: Privileged features are not accessible during testing either because they are too expensive to compute in real time, or because they are post-event features (thus cannot be used as input)

Original: [CM18].
Revised: [CM18].

Original: Learning with privileged features is pioneered in [VV09], where they propose a framework named“learning using privileged information” (LUPI).
Revised: Learning with privileged features is pioneered in [VV09], where they propose a framework named “learning using privileged information” (LUPI).

Original: At the core, LUPI uses privileged information todistinguish between easy and hard examples.
Revised: At the core, LUPI uses privileged information to predictions, and using these as distinguish between easy and hard examples.

Original: The methods are thus closely related to SVM, as thehardness of an example can be expressed by the slack variable.
Revised: The methods are thus closely related to SVM, as the hardness of an example can be expressed by the slack variable.

Original: For instance, [VV09, PIVV10] proposethe “SVM+” algorithm which generates slack variables from privileged features and learns an SVMbased on regular features with those slack variables; [SQL13] proposes a pair-wise SVM algorithmfor ranking, which uses privileged features to distinguish easy and hard pairs.
Revised: For instance, [VV09, PIVV10] propose the “SVM+” algorithm which generates slack variables from privileged features and learns an SVM based on regular features with those slack variables; [SQL13] proposes a pair-wise SVM algorithm for ranking, which uses privileged features to distinguish easy and hard pairs.

Original: [LHS14] presents avariation where the privileged features are used to generate importance weighting for different training samples.
Revised: [LHS14] presents a variation where the privileged features are used to generate importance weighting for different training samples.

Original: Empirically, [SER14] demonstrates that whether LUPI is effective critically depends onexperimental settings (e.g., preprocessing, training/validation split, etc).
Revised: Empirically, [SER14] demonstrates that whether LUPI is effective critically depends on experimental settings (e.g., preprocessing, training/validation split, etc).

Original: [VI15] considers transferringthe kernel function from a teacher SVM that only uses privileged features to a student SVM that onlyuses regular features; [LDX + 20] extends the SVM+ algorithm to imperfect privileged features.
Revised: [VI15] considers transferring the kernel function from a teacher SVM that only uses privileged features to a student SVM that only uses regular features; [LDX + 20] extends the SVM+ algorithm to imperfect privileged features.

Original: Model distillation [HVD + 15] is a common method for knowledge transfer, typically from a largemodel to a smaller one [PPA18, GYMT21].
Revised: Model distillation [HVD + 15] is a common method for knowledge transfer, typically from a large model to a smaller one [PPA18, GYMT21].

Original: Recent works have shown great empirical success inranking problems [TW18, HAS + 20, RPM + 21] and even the cases where the teacher model andstudent model have the identical structure [FLT + 18, QYT + 21].
Revised: Recent works have shown great empirical success in ranking problems [TW18, HAS + 20, RPM + 21] and even the cases where the teacher model and student model have the identical structure [FLT + 18, QYT + 21].

Original: Using distillation to learn from privileged features are ﬁrst proposed in [LPSBV16] as “generalizeddistillation” (GenD).
Revised: Using distillation to learn from privileged features are first proposed in [LPSBV16] as “generalized distillation” (GenD).

Original: It provides a uniﬁed view of LUPI and distillation.
Revised: It provides a unified view of LUPI and distillation.

Original: GenD, along with thevariants [MM16, GMM19, LLKH20], train a teacher model with only privileged features and thentrain a student model to mimic the teacher’s predictions.
Revised: GenD, along with the variants [MM16, GMM19, LLKH20], train a teacher model with only privileged features and then train a student model to mimic the teacher’s predictions.

Original: PFD is recently proposed in [XLG + 20],where the teacher model takes both regular and privileged features as input.
Revised: PFD is recently proposed in [XLG + 20], where the teacher model takes both regular and privileged features as input.

Original: PFD and GenD differfrom the standard model distillation as they focus on exploiting privileged features but not onreducing the model size.
Revised: PFD and GenD differ from the standard model distillation as they focus on exploiting privileged features but not on reducing the model size.

Original: [XLG + 20] empirically demonstrates the superior performance of PFD forrecommendation systems on a non-public data set.
Revised: [XLG + 20] empirically demonstrates the superior performance of PFD for recommendation systems on a non-public data set.

Original: Understanding of privileged features distillation is lacking, despite the aforementioned empiricalsuccess.
Revised: Understanding of privileged features distillation is lacking, despite the aforementioned empirical success.

Original: Previously, [PV10] shows that LUPI brings faster convergence under a strong assumptionthat the best classiﬁer is realizable with only privileged features.
Revised: Previously, [PV10] shows that LUPI brings faster convergence under a strong assumption that the best classifier is realizable with only privileged features.

Original: [LPSBV16] shows that GenDenjoys a fast convergence rate.
Revised: [LPSBV16] shows that GenD enjoys a fast convergence rate.

Original: It assumes that the teacher model has a much smaller function classcomplexity than the student model, which does not match with PFD.
Revised: It assumes that the teacher model has a much smaller function class complexity than the student model, which does not match with PFD.

Original: [GCFY18] studies GenD undersemi-supervised learning and shows that the beneﬁts come from student function class complexityreduction.
Revised: [GCFY18] studies GenD under semi-supervised learning and shows that the benefits come from student function class complexity reduction.

Original: However, it does not quantify such reduction and the theory does not explain what is thebeneﬁt of using privileged features.
Revised: However, it does not quantify such reduction and the theory does not explain what is the benefit of using privileged features.

Original: To the best of our knowledge, there is no empirical or theoreticalstudy explaining why PFD is effective.
Revised: To the best of our knowledge, there is no empirical or theoretical study explaining why PFD is effective.

Original: Other ways of utilizing privileged features are also previously proposed.
Revised: Other ways of utilizing privileged features are also previously proposed.

Original: [CJFY17] uses privilegedinformation to learn a more diverse representation to improve image classiﬁcation performance.[LLKH20, WZW + 21] propose distillation schemes for better feature extraction from regular features.
Revised: [CJFY17] uses privileged information to learn a more diverse representation to improve image classification performance. [LLKH20, WZW + 21] propose distillation schemes for better feature extraction from regular features.

Original: A more recent work [CJKB22] considers training a model with both regular and privileged featuresto obtain a better internal representation of the regular features.
Revised: A more recent work [CJKB22] considers training a model with both regular and privileged features to obtain a better internal representation of the regular features.

Original:  Problem Setup and 
Revised:  Problem Setup and Algorithms

Original: Algorithms

Original: Consider a learning-to-rank problem where each query-document pair has features x 2 X and z 2 Z116 and a label y 2 Y (e.g., click or human-annotated relevance) drawn from an unknown distribution
Revised: Consider a learning-to-rank problem where each query-document pair has features x ∈ X and z ∈ Z and a label y ∈ Y (e.g., click or human-annotated relevance) drawn from an unknown distribution

Original: D ( y | x , z ) .
Revised: D ( y | x , z ) .

Original: Suppose x is the regular feature that is available during both training and testing and z isonly available during training.
Revised: Suppose x is the regular feature that is available during both training and testing and z is only available during training.

Original: Concretely, privileged feature is deﬁned in the literature as below:
Revised: Concretely, privileged feature is defined in the literature as below:

Original: Deﬁnition 1 (Privileged Feature [CJKB22]) .
Revised: Definition 1 (Privileged Feature [CJKB22]) .

Original: For feature z that exists during training but not testing,we say z is a privileged feature if and only if I ( y ; z
Revised: For feature z that exists during training but not testing, we say z is a privileged feature if and only if I ( y ; z

Original: | x ) :
Revised: | x ) :

Original: = H ( y | x )  H ( y | x , z ) > 0 .
Revised: = H ( y | x ) − H ( y | x , z ) > 0 .

Original: Conditional mutual information
Revised: Conditional mutual information

Original: I ( y ; z | x ) and conditional entropy H ( ·|· ) follow from the standard122 notation of information theory.
Revised: I ( y ; z | x ) and conditional entropy H ( ·|· ) follow from the standard notation of information theory.

Original: According to Deﬁnition 1, the privileged feature z provides extrapredictive power of y .
Revised: According to Definition 1, the privileged feature z provides extra predictive power of y .

Original: For the rest of this paper, we focus on the setting that z is a privileged feature.
Revised: For the rest of this paper, we focus on the setting that z is a privileged feature.

Original: Remark 1.
Revised: Remark 1.

Original: An implication of Deﬁnition 1 is that the privileged feature z can be independent of theregular feature x .
Revised: An implication of Definition 1 is that the privileged feature z can be independent of the regular feature x .

Original: In such cases, any transformation of z is not learnable from x .
Revised: In such cases, any transformation of z is not learnable from x , and therefore using z as auxiliary learning target does not help.

Original: Interestingly, PFDcan still improve the student performance, even when z and x are independent (see Section 5).
Revised: Interestingly, PFD can still improve the student performance, even when z and x are independent (see Section 5).

Original: We consider the following general learning problem: we are given a labeled training set of size n ,
Revised: We consider the following general learning problem: we are given a labeled training set of size n ,

Original: S label := { ( x i , z i , y i ) } i 2 [ n ] , and a unlabeled training set of size m , S unlabel :
Revised: S label := { ( x i , z i , y i ) } i ∈ [ n ] , and a unlabeled training set of size m , S unlabel :

Original: = { ( x i , z i ) } i 2 [ m ] .
Revised: = { ( x i , z i ) } i ∈ [ m ] .

Original: Ourgoal is to generate good ranking based only on regular features x .
Revised: Our [Figure] goal is to generate good ranking based only on regular features x .

Original: For clarity of exposition, weonly consider pointwise scoring functions F :=
Revised: For clarity of exposition, we only consider pointwise scoring functions F :=

Original: { f | f : X 7! Y} , which generates a score for each131 document, and the ranking is induced by sorting the scores.
Revised: { f | f : X (cid:55)→ Y} , which generates a score for each document, and the ranking is induced by sorting the scores.

Original: The results in this paper can be easilyextended to models beyond pointwise scoring functions (e.g., DASALC [QYZ + 21]).
Revised: The results in this paper can be easily extended to models beyond pointwise scoring functions (e.g., DASALC [QYZ + 21]).

Original: The distinction between labeled and unlabeled datasets is for generality.
Revised: The distinction between labeled and unlabeled datasets is for generality.

Original: The unlabeled datasetnaturally appears in recommendation systems, where the majority of search logs do not contain anyuser interactions.
Revised: The unlabeled dataset naturally appears in recommendation systems, where the majority of search logs do not contain any user interactions.

Original: Instead of taking all such logs as negative samples, it is more proper to view them[Table] as unlabeled data due to the lack of user engagement.
Revised: Instead of taking all such logs as negative samples, it is more proper to view them as unlabeled data due to the lack of user engagement.

Original: For the logs that contain click, the documentstherein with no click can be treated as negative samples.
Revised: For the logs that contain click, the documents therein with no click can be treated as negative samples.

Original: 3.1 Privileged features distillation
Revised: 3.1 Privileged features distillation

Original: PFD ﬁrst trains a teacher model that takes both x and z as input to predict y , i.e., teacher functionclass is G PFD :
Revised: PFD first trains a teacher model that takes both x and z as input to predict y , i.e., teacher function class is G PFD :

Original: = { g | g : X ⇥ Z 7! Y} .
Revised: = { g | g : X × Z (cid:55)→ Y} .

Original: For simplicity, we consider pointwise loss l : Y ⇥ Y 7! R141 in this section, while the method can be easily extended to other loss functions (see an extension topairwise loss in Section 4).
Revised: For simplicity, we consider pointwise loss l : Y × Y (cid:55)→ R in this section, while the method can be easily extended to other loss functions (see an extension to pairwise loss in Section 4).

Original: The privileged features distillation takes the following two steps:
Revised: The privileged features distillation takes the following two steps:

Original: Step I: Training a teacher model g PFD 2 G PFD by minimizing the loss on the labeled dataset: P [Equation] [Equation]
Revised: Step I: Training a teacher model g PFD ∈ G PFD by minimizing the loss on the labeled dataset: (cid:80) ( x i , z i ,y i ) ∈S label l ( g ( x i , z i ) , y i ) .

Original: [Equation]

Original: l ( g ( x i , z i ) , y i ) .

Original: In practice, gradient-based optimizer is used for loss minimization.
Revised: In practice, gradient-based optimizer is used for loss minimization.

Original: Step II: Training a student model by distillation.
Revised: Step II: Training a student model by distillation.

Original: The teacher model g PFD trained from Step I isused to generate pseudo labels on S label and S unlabel .
Revised: The teacher model g PFD trained from Step I is used to generate pseudo labels on S label and S unlabel .

Original: Let S all denote the union of S label and S unlabel .
Revised: Let S all denote the union of S label and S unlabel .

Original: The student model is trained by minimizing the following distillation loss :
Revised: The student model is trained by minimizing the following distillation loss :

Original: [Equation]
Revised: [Equation]

Original: where ↵ 2 (0 , 1) controls the mixing ratio between the data loss and teacher loss.
Revised: where α ∈ (0 , 1) controls the mixing ratio between the data loss and teacher loss.

Original: The student modelis trained by minimizing the distillation loss in Equation (1).
Revised: The student model is trained by minimizing the distillation loss in Equation (1).

Original: 3.2 Other algorithms for comparisons
Revised: 3.2 Other algorithms for comparisons

Original: Here we introduce two other algorithms for comparison.
Revised: Here we introduce two other algorithms for comparison.

Original: See illustration in Figure 1.
Revised: See illustration in Figure 1.

Original: GenD [LPSBV16] is a distillation method where the teacher model takes only privileged features asinput, i.e., the teacher function class is G GenD =
Revised: GenD [LPSBV16] is a distillation method where the teacher model takes only privileged features as input, i.e., the teacher function class is G GenD =

Original: { g | g : Z 7! Y} .
Revised: { g | g : Z (cid:55)→ Y} .

Original: The teacher model g GenD 2 G GenD154 is obtained by minimizing P ( z i ,y i ) 2 S label l ( g ( z i ) , y i ) .
Revised: The teacher model g GenD ∈ G GenD is obtained by minimizing (cid:80) ( z i ,y i ) ∈S label l ( g ( z i ) , y i ) .

Original: Similar to PFD, the distillation loss is a linearcombination of the data loss and teacher loss.
Revised: Similar to PFD, the distillation loss is a linear combination of the data loss and teacher loss.

Original: Self-distillation [FLT + 18, QYT + 21] is a distillation method where the teacher model has the samestructure as the student model.
Revised: Self-distillation [FLT + 18, QYT + 21] is a distillation method where the teacher model has the same structure as the student model.

Original: Speciﬁcally, the teacher model g self-dist.
Revised: Specifically, the teacher model g self-dist.

Original: 2 F is obtained by minimizing P ( x i ,y i ) 2 S label l ( g ( x i ) , y i ) .
Revised: ∈ F is obtained by minimizing (cid:80) ( x i ,y i ) ∈S label l ( g ( x i ) , y i ) .

Original: Notice that F is also the student function class.
Revised: Notice that F is also the student function class.

Original: Similar to PFD, thedistillation loss is a linear combination of the data loss and teacher loss.
Revised: Similar to PFD, the distillation loss is a linear combination of the data loss and teacher loss.

Original: Comparing PFD againstself-distillation allows us to separate the beneﬁts of adopting privileged features and the beneﬁts ofdistillation without privileged features.
Revised: Comparing PFD against self-distillation separates the benefits of adopting privileged features and distillation.

Original:  Experiments 
Revised:  Experiments 4.1 Main results on public datasets

Original: 4.1 Main results on public datasets

Original: We ﬁrst evaluate the performance of PFD on three widely used public ranking datasets.
Revised: We first evaluate the performance of PFD on three widely used public ranking datasets.

Original: Speciﬁcally,we use the Set1 from “Yahoo!
Revised: Specifically, we use the Set1 from “Yahoo!

Original: Learn to rank challenge” [CC11]; “Istella Learning to Rank” dataset[DLN + 16]; and Microsoft Learning to Rank “MSLR-Web30k” dataset [QL13].
Revised: Learn to rank challenge” [CC11]; “Istella Learning to Rank” dataset [DLN + 16]; and Microsoft Learning to Rank “MSLR-Web30k” dataset [QL13].

Original: We refer to them as“Yahoo”, “Istella” and “Web30k” throughout this section.
Revised: We refer to them as “Yahoo”, “Istella” and “Web30k” throughout this section.

Original: Datasets overview and preprocessing.
Revised: Datasets overview and preprocessing.

Original: The training samples in all three datasets can be viewed asquery groups , where each query group contains 1 query and multiple documents to be ranked.
Revised: The training samples in all three datasets can be viewed as query groups , where each query group contains 1 query and multiple documents to be ranked.

Original: Eachquery-document pair is represented as a real-value feature vector (e.g., user dwelling time, tf-idf of document, etc.
Revised: Each query-document pair is represented as a real-value feature vector (e.g., user dwelling time, tf-idf of document, etc.

Original: See [CC11] for detail).
Revised: See [CC11] for detail).

Original: Further, each query-document pair has a human-annotatedrelevance score r 2 { 0 , 1 , 2 , 3 , 4 } .
Revised: Further, each query-document pair has a human-annotated relevance score r ∈ { 0 , 1 , 2 , 3 , 4 } .

Original: All datasets are preprocessed by removing query groups thatcontain no positive relevance score or have less than 10 documents.
Revised: All datasets are preprocessed by removing query groups that contain no positive relevance score or have less than 10 documents.

Original: The features are transformed bythe log1p transformation as in [ZWBN20, QYZ + 21].
Revised: The features are transformed by the log1p transformation as in [ZWBN20, QYZ + 21].

Original: Binary label generation.
Revised: Binary label generation.

Original: In practice, binary label (e.g., click) is more commonly seen and easierto obtain than relevance score.
Revised: In practice, binary label (e.g., click) is more commonly seen and easier to obtain than relevance score.

Original: For our experiments, we generate a binary label y for each querydocument pair based on the human-annotated relevance score r .
Revised: For our experiments, we generate a binary label y for each querydocument pair based on the human-annotated relevance score r .

Original: Speciﬁcally:
Revised: Specifically:

Original: [Equation]
Revised: [Equation]

Original: where t is a temperature parameter and G 1 and G 0 follow the standard Gumbel distribution.
Revised: where t is a temperature parameter and G 1 and G 0 follow the standard Gumbel distribution.

Original: It canbe shown that y is 1 with probability  ( t · ( r  ⌧ target )) , where  ( · ) is the sigmoid function (see
Revised: It can be shown that y is 1 with probability σ ( t · ( r − τ target )) , where σ ( · ) is the sigmoid function (see

Original: Appendix A.1 for proof).
Revised: Appendix A.1 for proof).

Original: For the rest of our experiments, we set t = 4 and ⌧ target = 4 . 8 unlessotherwise mentioned.
Revised: For the rest of our experiments, we set t = 4 and τ target = 4 . 8 unless otherwise mentioned.

Original: We refer to the query groups that contain at least one y = 1 to be positivequery groups , and other query groups are referred to as negative query groups .
Revised: We refer to the query groups that contain at least one y = 1 to be positive query groups , and other query groups are referred to as negative query groups .

Original: Regular and privileged features split.
Revised: Regular and privileged features split.

Original: For each of the datasets, we sort the features according to themagnitude of their correlations with the binary label y and use the top 200, 50, and 40 features asprivileged features for Yahoo, Istella, and Web30k, respectively.
Revised: For each of the datasets, we sort the features according to the magnitude of their correlations with the binary label y and use the top 200, 50, and 40 features as privileged features for Yahoo, Istella, and Web30k, respectively.

Original: Other features are used as regularfeatures.
Revised: Other features are used as regular features.

Original: Please see Table 1 for dataset statistics after preprocessing and binary label generation.
Revised: Please see Table 1 for dataset statistics after preprocessing and binary label generation.

Original: [Table]
Revised: [Table]

Original: Ranking model and performance metric.
Revised: Ranking model and performance metric.

Original: The ranking model is a 5-layer fully connected neuralnetwork, which maps the query-document feature into a real-value score s 2 [0 , 1] .
Revised: The ranking model is a 5-layer fully connected neural network, which maps the query-document feature into a real-value score s ∈ [0 , 1] .

Original: The ranking b ⇡ ofdocuments is obtained by sorting the scores decreasingly, where b ⇡ ( i ) represents the ranked order ofthe i -th document.
Revised: The ranking (cid:98) π of documents is obtained by sorting the scores decreasingly, where (cid:98) π ( i ) represents the ranked order of the i -th document.

Original: The ranking performance is measured by the NDCG@ k metric:
Revised: The ranking performance is measured by the NDCG@ k metric:

Original: [Equation]
Revised: [Equation]

Original: where ⇡ ⇤ is the optimal ranking obtained by sorting y i . 
Revised: where π ∗ is the optimal ranking obtained by sorting y i . [Figure]

Original: PFD is effective for all threedatasets.
Revised: PFD is effective for all three datasets.

Original: We evaluate the efﬁcacy of PFD on all three aforementioned datasets, under bothpointwise (RankBCE) and pairwise (RankNet [BSR + 05]) lossfunctions (see deﬁnitions in Appendix A.2).
Revised: We evaluate the efficacy of PFD on all three aforementioned datasets, under both pointwise (RankBCE) and pairwise (RankNet [BSR + 05]) loss functions (see definitions in Appendix A.2).

Original: Please see the evaluated algorithms and results in
Revised: Please see the evaluated algorithms and results in

Original: Table 2 (complete results with
Revised: Table 2 (complete results with

Original: RankNet loss deferred to Table 4).
Revised: RankNet loss deferred to Table 4).

Original: Figure 2 shows the testing
Revised: Figure 2 shows the testing

Original: NDCG@8 curve on Yahoo and
Revised: NDCG@8 curve on Yahoo and

Original: Web30k with RankBCE loss.
Revised: Web30k with RankBCE loss.

Original:  Table 2 shows that PFD has the best performance on all evaluated settings.
Revised: [Table] Table 2 shows that PFD has the best performance on all evaluated settings.

Original: We remark that (1) the onlydifference between PFD and self-distillation is that the teacher in PFD additionally uses privilegedfeatures and therefore has better prediction accuracy than the teacher in self-distillation.
Revised: We remark that (1) the only difference between PFD and self-distillation is that the teacher in PFD additionally uses privileged features and therefore has better prediction accuracy than the teacher in self-distillation.

Original: Comparing
Revised: Comparing

Original: PFD with self-distillation reveals the improvement of using “privileged features” for distillation; (2)the performance of GenD is worse than no-distillation on Istella and Web30k.
Revised: PFD with self-distillation reveals the improvement of using “privileged features” for distillation; (2) the performance of GenD is worse than no-distillation on Istella and Web30k.

Original: The reason for such[Table] inferior performance is that the teacher model in GenD only uses privileged features (and not regularfeatures).
Revised: The reason for such inferior performance is that the teacher model in GenD only uses privileged features (and not regular features).

Original: For Istella and Web30k, only using privileged features is not sufﬁcient to generate goodpredictions.
Revised: For Istella and Web30k, only using privileged features is not sufficient to generate good predictions.

Original: The teachers in GenD are also worse than no-distillation, see Appendix A.4.
Revised: The teachers in GenD are also worse than no-distillation, see Appendix A.4.

Original: 4.2 Ablation study on public datasets
Revised: 4.2 Ablation study on public datasets

Original:  PFD is not sensitive to ↵ .
Revised: [Figure] PFD is not sensitive to α .

Original: In former experiments, we kept themixing ratio of teacher loss anddata loss to be ↵ = 0 .
Revised: In former experiments, we kept the mixing ratio of teacher loss and data loss to be α = 0 .

Original: Here weevaluate the sensitivity of PFDto parameter ↵ .
Revised: Here we evaluate the sensitivity of PFD to parameter α .

Original: The experimentshere use the Yahoo dataset and
Revised: The experiments here use the Yahoo dataset and

Original: RankBCE loss.
Revised: RankBCE loss.

Original: From the lefthand side of Figure 3, we see that
Revised: From the lefthand side of Figure 3, we see that

Original: PFD delivers good performanceover a large range of ↵ .
Revised: PFD delivers good performance over a large range of α .

Original: However, it is worth noting that the teacher loss is typically much larger than the data loss (e.g., about20 times larger in this set of experiments), since the teacher’s predictions are much denser learningtargets.
Revised: However, it is worth noting that the teacher loss is typically much larger than the data loss (e.g., about 20 times larger in this set of experiments), since the teacher’s predictions are much denser learning targets.

Original: The right-hand side plot of Figure 3 takes the scale of both losses into consideration.
Revised: The right-hand side plot of Figure 3 takes the scale of both losses into consideration.

Original: It showsthat PFD yields the best performance only when the teacher loss dominates the distillation loss.
Revised: It shows that PFD yields the best performance only when the teacher loss dominates the distillation loss.

Original: PFD brings a larger gain when the positive labels are sparse.
Revised: PFD brings a larger gain when the positive labels are sparse.

Original: Recall that we view negative querygroups as unlabeled data.
Revised: Recall that we view negative query groups as unlabeled data.

Original: Here we evaluate the performance of PFD under different numbers ofpositive labels.
Revised: Here we evaluate the performance of PFD under different numbers of positive labels.

Original: Speciﬁcally, by reducing ⌧ target from 4.8 to 0.4, we can increase the percentage ofpositive query groups (i.e., query groups with at least one y = 1 ).
Revised: Specifically, by reducing τ target from 4.8 to 0.4, we can increase the percentage of positive query groups (i.e., query groups with at least one y = 1 ).

Original: The relative improvement overbaseline is shown in Figure 4.
Revised: The relative improvement over baseline is shown in Figure 4.

Original: While it is known that distillation works better when there are moreunlabeled samples, Figure 4 shows that PFD further ampliﬁes such gains: the relative gain of PFDover self-distillation also increases as the positive labels become sparser.
Revised: While it is known that distillation works better when there are more unlabeled samples, Figure 4 shows that PFD further amplifies such gains: the relative gain of PFD over self-distillation also increases as the positive labels become sparser.

Original: Such beneﬁt is especiallyfavorable in recommendation systems, where the positive labels (e.g., click) are naturally very sparse.
Revised: Such benefit is especially favorable in recommendation systems, where the positive labels (e.g., click) are naturally very sparse.

Original:  Correlation 
Revised: [Figure] PFD yields larger improvement [Figure] Correlation between privileged features and target.

Original: Itis believed that privileged features that are discriminative (e.g., high correlation with the target) lead to accurate teacher predictions, and thus beneﬁt the distillation[XLG + 20].
Revised: It is believed that privileged features that are discriminative (e.g., high correlation with the target) lead to accurate teacher predictions, and thus benefit the distillation [XLG + 20].

Original: However, we show that PFD has poor performance when the privileged features are too discriminative.
Revised: However, we show that PFD has poor performance when the privileged features are too discriminative.

Original: Speciﬁcally, we modify the experiment setting such thatall the features in the datasets are used as regular features,while the privileged features z are generated accordingto z
Revised: Specifically, we modify the experiment setting such that all the features in the datasets are used as regular features, while the privileged features z are generated according to z

Original: = I ( t · r + G 1 > t · ⌧ privileged + G 0 ) , where G 1 and248 G 0 have the same values as in binary label
Revised: = I ( t · r + G 1 > t · τ privileged + G 0 ) , where G 1 and G 0 have the same values as in binary label

Original: y generation(Equation (2)).
Revised: y generation (Equation (2)).

Original: By changing ⌧ privileged , we can obtain privileged features z with different correlations with the label y .
Revised: By changing τ privileged , we can obtain privileged features z with different correlations with the label y .

Original: For instance, when ⌧ privileged = ⌧ target , then z can perfectlypredict y (since z = y by deﬁnition); and z becomes lessdiscriminative when ⌧ privileged gets smaller.
Revised: For instance, when τ privileged = τ target , then z can perfectly predict y (since z = y by definition); and z becomes less discriminative when τ privileged gets smaller.

Original: Using z as theprivileged feature, we have the PFD results in Figure 5.
Revised: Using z as the privileged feature, we have the PFD results in Figure 5.

Original: Notice that the privileged feature with the largest correlation with y does not give the best performance.
Revised: Notice that the privileged feature with the largest correlation with y does not give the best performance.

Original: We believethe reason is that as the correlation of z and y increases,the privileged feature becomes so “discriminative” that itcan explain almost all the variance in y , even the noise.
Revised: We believe the reason is that as the correlation of z and y increases, the privileged feature becomes so “discriminative” that it can explain almost all the variance in y , even the noise.

Original: As a result, teacher predictions have high variance, whichleads to high-variance student estimates and inferior testing performance.
Revised: As a result, teacher predictions have high variance, which leads to high-variance student estimates and inferior testing performance.

Original: See Section 5.2 for theoretical insights.
Revised: See Section 5.2 for theoretical insights.

Original: 4.3 Evaluation on Amazon’s dataset
Revised: 4.3 Evaluation on Amazon’s dataset

Original:  overview and ranking model.
Revised: Dataset overview and ranking model.

Original: The dataset isderived from Amazon’s logs which contains query andproduct title text, the position at which the product wasshown, and the user’s behaviors click, add-to-cart, andpurchase .
Revised: The dataset is derived from Amazon’s logs which contains query and product title text, the position at which the product was shown, and the user’s behaviors click, add-to-cart, and purchase .

Original: The ranking model is a multi-layer transformerthat maps query and product title to an estimate of thepurchase likelihood.
Revised: The ranking model is a multi-layer transformer that maps query and product title to an estimate of the purchase likelihood.

Original: The goal is to rank the products that are more likely to be purchased ﬁrst.
Revised: The goal is to rank the products that are more likely to be purchased first.

Original: Efﬁcacy of PFD.
Revised: Efficacy of PFD.

Original: Here we evaluate the performance of PFD.
Revised: Here we evaluate the performance of PFD.

Original: Notice that the position is a privilegedfeature as it is not available as an input during online serving (it is the output of the ranking modelthat becomes position).
Revised: Notice that the position is a privileged feature as it is not available as an input during online serving (it is the output of the ranking model that becomes position).

Original: Further, the click and add-to-cart are naturally privileged features, since onecannot know which of the product will be clicked or added to cart before showing the products.
Revised: Further, the click and add-to-cart are naturally privileged features, since one cannot know which of the product will be clicked or added to cart before showing the products.

Original: The baseline no-distillation model only takes query and product title as input, while the teachermodels in PFD additionally take positions or clicks or add-to-cart as privileged features.
Revised: The baseline no-distillation model only takes query and product title as input, while the teacher models in PFD additionally take positions or clicks or add-to-cart as privileged features.

Original: As in publicdatasets, we only use the positive query groups to train the teacher model and use all query groups fordistillation.
Revised: As in public datasets, we only use the positive query groups to train the teacher model and use all query groups for distillation.

Original: We additionally use pretraining then ﬁnetuning as another baseline, as predicting “click”or “add-to-cart” can serve as pretraining tasks.
Revised: We additionally use pretraining then finetuning as another baseline, as predicting “click” or “add-to-cart” can serve as pretraining tasks.

Original: The experiment results are shown in Table 3.
Revised: The experiment results are shown in Table 3.

Original: Extension: Multi-teacher distillation.
Revised: Extension: Multi-teacher distillation.

Original: Inspired by [FSK + 17, ZXHL18], we also evaluate the multiteacher distillation, where the student learns from more than one teacher.
Revised: Inspired by [FSK + 17, ZXHL18], we also evaluate the multiteacher distillation, where the student learns from more than one teacher.

Original: We adopt three privilegedteachers which take positions, clicks, and add-to-cart as input, respectively.
Revised: We adopt three privileged teachers which take positions, clicks, and add-to-cart as input, respectively.

Original: We calculate the loss w.r.t.each of the teachers’ predictions and use the average as “teacher loss” in Equation (1).
Revised: We calculate the loss w.r.t. each of the teachers’ predictions and use the average as “teacher loss” in Equation (1).

Original: Intuitively,the student model is trained to learn from an “ensemble” of teacher models.
Revised: Intuitively, the student model is trained to learn from an “ensemble” of teacher models.

Original: The multi-teacher PFDyields the best performance, an 11.2% improvement on testing NDCG@8 over the baseline model. 
Revised: The multi-teacher PFD yields the best performance, an 11.2% improvement on testing NDCG@8 over the baseline model. [Table] [Table]

Revised: teachers (i.e., “teacher loss” in Equation (1) being the averaged loss w.r.t.

Revised: each of the teachers’ predictions) further improves the PFD performance and leads to the best ranking performance.

Original:  Theoretical Insights
Revised:  Theoretical Insights

Original: In this section, we present theoretical insights on why and when PFD works via analysis on linearmodels.
Revised: In this section, we present theoretical insights on why and when PFD works via analysis on linear models.

Original: While our empirical focus is on ranking problems, our theoretical insights are more general.[Table] [Table] [Table]
Revised: While our empirical focus is on ranking problems, our theoretical insights are more general.

Original: 3) learning from multiple teachers (i.e., “teacher loss” in Equation (1) being the averaged loss w.r.t.

Original: each of the teachers’ predictions) further improves the PFD performance and leads to the best ranking performance.

Original: Consider the following learning problem: the regular feature x 2 R d x is drawn from a spherical
Revised: Consider the following learning problem: the regular feature x ∈ R d x is drawn from a spherical

Original: Gaussian distribution N (0 , I d x ) and an un-observable feature u 2 R d u is drawn from N (0 , I d u ) .
Revised: Gaussian distribution N (0 , I d x ) and an un-observable feature u ∈ R d u is drawn from N (0 , I d u ) .

Original: With two unknown parameters w ⇤ 2 R d x and v ⇤ 2 R d u , the label y is generated as following:
Revised: With two unknown parameters w ∗ ∈ R d x and v ∗ ∈ R d u , the label y is generated as following:

Original: [Equation]
Revised: [Equation]

Original: where ✏ represents the label noise.
Revised: where ϵ represents the label noise.

Original: During training time, we observe the features z = u as privilegedfeatures.
Revised: During training time, we observe the features z = u as privileged features.

Original: Suppose that the labeled training set S label =
Revised: Suppose that the labeled training set S label =

Original:  X 2 R n ⇥ d x , Z 2 R n ⇥ d z , y 2 R n  and theunlabeled set S unlabel = [Equation]
Revised: (cid:8) X ∈ R n × d x , Z ∈ R n × d z , y ∈ R n (cid:9) and the unlabeled set S unlabel = (cid:8)

Original:  are generated according to the aforementioned data generation scheme.
Revised: X ( u ) ∈ R m × d x , Z ( u ) ∈ R m × d z (cid:9) are generated according to the aforementioned data generation scheme.

Original: Let X ( a ) =
Revised: Let X ( a ) =

Original: [ X ; X ( u ) ] 2 R ( n + m ) ⇥
Revised: [ X ; X ( u ) ] ∈ R ( n + m ) ×

Original: d x and Z ( a ) =
Revised: d x and Z ( a ) =

Original: [ Z ; Z ( u ) ] [-
-]
Revised: [ Z ; Z ( u ) ] [-
-]∈

Original: R ( n + m ) ⇥
Revised: R ( n + m ) ×

Original: d z be the all the inputs from both labeled and unlabeled datasets.
Revised: d z be the all the inputs from both labeled and unlabeled datasets.

Original: The goal is to learn topredict y with only regular feature x as input.
Revised: The goal is to learn to predict y with only regular feature x as input.

Original: 5.1 PFD works by reducing estimation variance
Revised: 5.1 PFD works by reducing estimation variance

Original: Let b w reg denote the model learned by standard linear regression and b w pri be the model learned byprivileged features distillation.
Revised: Let (cid:98) w reg denote the model learned by standard linear regression and (cid:98) w pri be the model learned by privileged features distillation.

Original: For simplicity, we consider the case with ↵ = 0 , i.e., only learningfrom the teacher’s prediction during distillation.
Revised: For simplicity, we consider the case with α = 0 , i.e., only learning from the teacher’s prediction during distillation.

Original: Speciﬁcally, the standard linear regression only usesthe set S label , and b w reg is obtained by regressing y on X .
Revised: Specifically, the standard linear regression only uses the set S label , and (cid:98) w reg is obtained by regressing y on X .

Original: PFD, on the other hand, ﬁrst uses S label toregress y on [ X ; Z ] .
Revised: PFD, on the other hand, first uses S label to regress y on [ X ; Z ] .

Original: The learned model is then used to generate predictions b y 
Revised: The learned model is then used to generate predictions (cid:98) y for S label ∪ S unlabel .

Original: Finally, b y is regressed on X ( a ) , which gives b w pri .
Revised: Finally, (cid:98) y is regressed on X ( a ) , which gives (cid:98) w pri .

Original: We have the following result on the merit of PFD:
Revised: We have the following result on the merit of PFD:

Original: Theorem 1.
Revised: Theorem 1.

Original: For standard linear regression, we have that
Revised: For standard linear regression, we have that

Original: [Equation]
Revised: [Equation]

Original: For privileged features distillation, we have that
Revised: For privileged features distillation, we have that

Original: [Equation]
Revised: [Equation]

Original: Notice that var ( y | x ) =
Revised: Notice that var ( y | x ) =

Original:  2 + k v ⇤ k 22 , where  2 corresponds to the label noise and k v ⇤ k 22 correspondsto the variance that can be explained by the privileged features.
Revised: σ 2 + ∥ v ∗ ∥ 22 , where σ 2 corresponds to the label noise and ∥ v ∗ ∥ 22 corresponds to the variance that can be explained by the privileged features.

Original: The result shows that PFD canexplain a proper part of the variance in y by privileged features z .
Revised: The result shows that PFD can explain a proper part of the variance in y by privileged features z .

Original: By learning from the teacher’spredictions, PFD can therefore reduce the variance of b w pri by exploiting the privileged features andthe unlabeled samples.
Revised: By learning from the teacher’s predictions, PFD can therefore reduce the variance of (cid:98) w pri by exploiting the privileged features and the unlabeled samples.

Original: On the other hand, when learning with plain linear regression, the labelvariance corresponding to z is treated as noise, which leads to estimation with higher variance.
Revised: On the other hand, when learning with plain linear regression, the label variance corresponding to z is treated as noise, which leads to estimation with higher variance.

Original: Remark 2.
Revised: Remark 2.

Original: Why GenD has worse-than-baseline performance.
Revised: Why GenD has worse-than-baseline performance.

Original: Notice that the teacher modelin GenD uses privileged features only.
Revised: Notice that the teacher model in GenD uses privileged features only.

Original: GenD has inferior performance for two reasons: (1) theprivileged features alone are not enough for the teacher model to generate good predictions; and (2)when z is independent of x , the predictions from the GenD’s teacher are not learnable for the student.5.2 PFD has inferior performance when the privileged features are too discriminative
Revised: GenD has inferior performance for two reasons: (1) the privileged features alone are not enough for the teacher model to generate good predictions; and (2) when z is independent of x , the predictions from the GenD’s teacher are not learnable for the student. 5.2 PFD has inferior performance when the privileged features are too discriminative

Original: To understand the performance of PFD under different privileged features, consider the setting wherez 2 R d z is the ﬁrst d z coordinates of u .
Revised: To understand the performance of PFD under different privileged features, consider the setting where z ∈ R d z is the first d z coordinates of u .

Original: When d z = d u , it recovers the setting in previous subsection.
Revised: When d z = d u , it recovers the setting in previous subsection.

Original: Notice that the larger d z becomes, the better ( x ; z ) can predict y .
Revised: Notice that the larger d z becomes, the better ( x ; z ) can predict y .

Original: While one might expect thatd z = d u (i.e., when the privileged features contain the most information about y ) leads to the bestdistillation performance, our next result shows that such belief is not true in general.
Revised: While one might expect that d z = d u (i.e., when the privileged features contain the most information about y ) leads to the best distillation performance, our next result shows that such belief is not true in general.

Original: Let v ⇤ z be thepart of v ⇤ that corresponds to z (i.e., the ﬁrst d z coordinates of v ⇤ ), we have:
Revised: Let v ∗ z be the part of v ∗ that corresponds to z (i.e., the first d z coordinates of v ∗ ), we have:

Original: Theorem 2.
Revised: Theorem 2.

Original: For privileged features distillation, we have that
Revised: For privileged features distillation, we have that

Original: [Equation]
Revised: [Equation]

Original: ) , Z

Original: [Equation]
Revised: [Figure]

Original: As we increase d z from 0 to d u , k v ⇤ z k also increases.
Revised: As we increase d z from 0 to d u , ∥ v ∗ z ∥ also increases.

Original: The326 teacher, therefore, explains more variance in y and contributes to a smaller error in the student estimate b w pri .
Revised: The teacher, therefore, explains more variance in y and contributes to a smaller error in the student estimate (cid:98) w pri .

Original: However, the denominator of the ﬁrst term decreases asd z increases, which leads to a higher variance (thus lessaccurate) student parameter estimate b w pri .
Revised: However, the denominator of the first term decreases as d z increases, which leads to a higher variance (thus less accurate) student parameter estimate (cid:98) w pri .

Original: Combining thetwo effects, the privileged features z that contain the mostinformation about y do not yield the best distillation performance.
Revised: Combining the two effects, the privileged features z that contain the most information about y do not yield the best distillation performance.

Original: This matches the non-monotone observation in
Revised: This matches the non-monotone observation in

Original: Figure 5; and the results in Table 3 where using add-to-cart(i.e. , the most informative feature for predicting purchase)does not give the best PFD result.
Revised: Figure 5; and the results in Table 3 where using add-to-cart (i.e., the most informative feature for predicting purchase) does not give the best PFD result.

Original: Example 1.
Revised: Example 1.

Original: Consider the data generation as shown in
Revised: Consider the data generation as shown in

Original: Equation (3) .
Revised: Equation (3) .

Original: We set d x = 10 , d u = 10 , n = 30 , m =200 , and draw w ⇤ from a spherical Gaussian distribution
Revised: We set d x = 10 , d u = 10 , n = 30 , m = 200 , and draw w ∗ from a spherical Gaussian distribution

Original: N (0 , I d x ) .
Revised: N (0 , I d x ) .

Original: Further, we set  = 15 , and let v ⇤ =
Revised: Further, we set σ = 15 , and let v ∗ =

Original: We evaluate the performance ofthe standard linear regression and the privileged features distillation with d z from 0 to 10.
Revised: We evaluate the performance of the standard linear regression and the privileged features distillation with d z from 0 to 10.

Original: The resultsin Figure 6 shows that the most predictive z does not give the best PFD performance.
Revised: The results in Figure 6 shows that the most predictive z does not give the best PFD performance.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we take a step toward understanding PFD in learning-to-rank.
Revised: In this paper, we take a step toward understanding PFD in learning-to-rank.

Original: We ﬁrst evaluate
Revised: We first evaluate

Original: PFD on three public ranking datasets (Yahoo, Istella, and MSLR- Web30k) and an industrial-scale ranking problem derived from Amazon’s search logs.
Revised: PFD on three public ranking datasets (Yahoo, Istella, and MSLR-Web30k) and an industrial-scale ranking problem derived from Amazon’s search logs.

Original: Our evaluation shows that PFD has the best performance in all evaluated settings.
Revised: Our evaluation shows that PFD has the best performance in all evaluated settings.

Original: We further conduct comprehensive empirical ablation studies, which demonstrates the efﬁcacy and robustness of PFD and uncovers an interesting non-monotone behavior – as the predictive power of privileged features increase, the performance of PFD ﬁrstincreases and then decreases.
Revised: We further conduct comprehensive empirical ablation studies, which demonstrates the efficacy and robustness of PFD and uncovers an interesting non-monotone behavior – as the predictive power of privileged features increase, the performance of PFD first increases and then decreases.

Original: Finally, we present theoretical insights for PFD via rigorous analysisfor linear models.
Revised: Finally, we present theoretical insights for PFD via rigorous analysis for linear models.

Original: The theoretical results show that (1) PFD is effective by reducing the variance of student estimation; and (2) a too predictive privileged teacher produces high variance predictions, which lead to high variance (less accurate) student estimates and inferior testing performance
Revised: The theoretical results show that (1) PFD is effective by reducing the variance of student estimation; and (2) a too predictive privileged teacher produces high variance predictions, which lead to high variance (less accurate) student estimates and inferior testing performance.

Original: Checklist
Revised: A Appendix for Experiments A.1 Features preprocessing and binary label generation log1p transformation

Revised: The log1p [QYZ + 21] transformation is defined as following: [Equation]

Original: For all authors...
Revised: where ⊙ is the element-wise multiplication operator.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: Notice that the standard Gumbel distribution is G i = − log( − log( U i )) , where U i is a uniform random variables on [0 , 1] .

Original: [Yes] See Sections 4 and 5.
Revised: Therefore, we have [Equation]

Original: (b) Did you describe the limitations of your work?
Revised: privileged

Original: [Yes] Sections 4 and 5 present discussion
Revised: [Equation] privileged [Equation]

Original: on when PFD does not work. (c) Did you discuss any potential negative societal impacts of your work?
Revised: The σ ( · ) is the sigmoid function. This is known as the “Gumbel trick”. A.2 Training loss functions

Original: [N/A]
Revised: RankBCE The RankBCE loss views each query-document pair as a binary classification sample and applies cross entropy loss.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to 
Revised: Let D denote all query-document pairs and f be the model to be evaluated, the RankBCE loss is defined as

Original: them? [Yes]
Revised: [Equation]

Original: If you are including theoretical results...
Revised: where σ is the sigmoid function.

Original: (a) Did you state the full set of assumptions of all theoretical results? [Yes] The beginningof Section 5 speciﬁes the setting and assumption.(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.3.
Revised: RankNet RankNet is a pairwise loss function.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: For a query and two documents i and j , the RankNet loss function penalizes inconsistent predictions (i.e., f ( x i ) <

Original: mental results (either in the supplemental material or as a URL)?
Revised: f ( x j ) while y i

Original: [Yes] See supplemen-
Revised: =

Original: tal material.
Revised: and y j = 0 ).

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: Formally, the RankNet loss is defined as [Equation] D

Original: how they were chosen)?
Revised: [Equation] (cid:88) i,

Original: [Yes] See Appendix A.3.
Revised: [Equation] j

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: CrossEntropy ( σ ( f ( x i ) − f ( x j )) , ( y i − y j + 1) / 2) A.3 Detailed Experiments Setup The ranking model is a 5-layer fully connected neural network with hidden dimension 100.

Original: [Yes] All results on  the public datasets include standarddeviation.(d)
Revised: The ranking model is trained for 100 epochs with Adam optimizer with weight decay 0.005. For all three datasets, we use batch size 500 and initial learning rate 1e-3 for RankBCE loss and batch sizelearning rate 3e-4 for RankNet loss. Learning rate decays to half at every 20 epochs. For experiments on Yahoo and Istella, we repeat the experiments for 5 independent runs, and for

Original: Did you include the total amount of compute and  the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Web30k, we use the official 5-fold splits. For each run, the ranking model is trained for 100 epochs and evaluated on the test set after every training epoch. The average and standard deviation of testing NDCG of the best model checkpoints (i.e., the one that has the best testing NDCG@8 score in each run) are reported.

Original: [No]4.
Revised: For all distillation experiments, the teacher model is the model checkpoint with the highest testing NDCG@8 score. The implementation for training and evaluation is adapted from

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: PT-ranking [Yu20]. A.4 Complete results on public datasets See complete results and description in Table 4.

Original: (a) 
Revised: A.

Original: If your work uses existing assets, did you cite the creators?
Revised: About viewing

Original: [Yes] See Section 4and appendix A.3.
Revised: negative query

Original: (b) Did you mention the license of the assets?
Revised: groups as unlabeled data.

Original: [Yes] See supplemental material. (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: Recall that query groups that contain at least one y = 1 are called positive query groups, and others are called negative query groups.

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
Revised: Notice that in all previous experiments on public datasets, we only use positive query groups to compute the data loss and view the negative query groups as unlabeled data.

Original: [N/A]
Revised: An alternative is to use all query groups to compute the data loss. Here we compare two different ways of training and the results are presented in Table 5.

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: From Table 5, we see that using only positive query groups for data loss always leads to better distillation performance.

Original: [Table] information or offensive content? [N/A] [Table]
Revised: We believe the reason is that the labels y in the negative query groups are all 0 and thus those samples give little information on which document is more relevant.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: It is tempting to use PFD teacher models during testing time with imputed privileged features as input. Here we test PFD teacher models’ performance with two ways of imputation: (1) constant 0; or (2) the mean of privileged features in the training set.

Original: [Table]
Revised: The results are presented in Table 6.

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: It shows that both ways of privileged features imputation are significantly worse than the models trained without privileged features.

Original: [Table]
Revised: As an alternative, one might learn to predict the privileged features from regular features. However, the privileged features might be harder to predict than the original target and sometimes the privileged features are independent (thus not predictable) from the regular features.

Original: applicable?
Revised: Therefore it is generally hard to use the PFD teacher models during testing time. A.7 Sensitivity to α on other datasets To see PFD is generally robust to different choices of α (i.e., the mixing ratio of teacher loss and data loss, see Equation (1)), we also evaluated the sensitivity to α on Istella and Web30k.

Original: [N/A]
Revised: Specifically, we adopted a list of α : [0.3, 0.5, 0.7, 0.9, 0.95, 0.99, 0.999] and the results on Istella show that for all α ≤ 0 . 95 , PFD can deliver an over 2.5% improvement in NDCG@8; for Web30k, for all α ≤ 0 . PFD can stably deliver an over 2.6% improvement. B Proofs for theoretical insights Proof of Theorem 1.

Original: [Table]
Revised: Proof.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: We first prove the result for standard linear regression. For a matrix M , let M † be the pseudo inverse of M , i.e., M † = (cid:0) M ⊤ M (cid:1) − 1 M ⊤ . The solution of standard linear regression is

Original: [Table]
Revised: [Equation]

Original: (IRB) approvals, if applicable?
Revised: where N ∈ R n × 1 is the stack of label noise ϵ . Therefore, we have [Equation] The last equality holds because (cid:0) X ⊤ X (cid:1) − 1 follows the inverse-Wishart distribution, whose expectation is

Original: [N/A]
Revised: [Equation] [Table] [Table] [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: For privileged features distillation, for the teacher estimation (cid:98) θ ∈ R d x + d z , we have

Original: [Table]
Revised: [Equation]

Original: spent on participant compensation?
Revised: where X † Z , ⊥ is the pseudo inverse of the projection of X to the column space orthogonal to Z , and Z † X , ⊥ is defined similarly. After distillation, we have that [Equation]

Revised: Notice that X † ( a ) Z ( a ) Z † X , ⊥ N is order O (cid:0) 1 n · m (cid:1) , which is a non-dominating term.

Revised: For the other two terms, we have

Revised: [Equation]

Revised: The last equality follows as X ( a ) has n + m samples, and E (cid:20)(cid:16)

Revised: X ⊤ ( a ) X (

Revised: [Equation]

Revised: (cid:17) −

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: [N/A]
Revised: + m − [Equation] x [Equation]

Revised: x which is the expectation of inverse-Wishart distribution.

Revised: Notice that the degree of freedom of X ⊤ Z , ⊥ is n − d z , therefore the expectation of (cid:16) X ⊤ Z , ⊥ X Z , ⊥ (cid:17) − 1 is

Revised: [Equation]

Revised: − d z

Revised: [Equation]

Revised: I d

Original: [Table]
Revised: [Equation] ■ Proof of Theorem 2. Proof.

Revised: For privileged features distillation, let U ¬ z ∈ R n × ( d u − d z ) denote the features that correspond to the coordinates of U not revealed by Z and v ∗¬ z denote the corresponding coordinates of v ∗ .

Revised: For the teacher estimation (cid:98) θ ∈ R

Revised: d

Revised: x + d z , we have

Revised: [Equation]

Revised: where X † Z , ⊥ is the pseudo inverse of the projection of X to the column space orthogonal to Z , and

Revised: Z † X , ⊥ is defined similarly.

Revised: After distillation, we have that

Revised: [Equation]

Revised: Notice that X † ( a ) Z ( a ) Z † X , ⊥

Revised: ( U ¬ z v ∗¬ z + N ) has the order O (cid:0) 1 n · m (cid:1) , which is a non-dominating term.

Revised: For the other two terms, we have

Revised: [Equation]

Revised: The last equality follows as X ( a ) has n + m samples, and E (cid:20)(cid:16)

Revised: X ⊤ ( a ) X ( a ) (cid:17) − 1 (cid:21) =

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: 1 I d x which is the expectation of inverse-Wishart distribution.

Revised: Notice that the degree of freedom of X ⊤ Z , ⊥ is n − d z , therefore the expectation of (cid:16) X ⊤ Z , ⊥ X Z , ⊥ (cid:17) − 1 is

Revised: [Equation]