Original: D IMENSIONALITY R EDUCTION FOR R EPRESENTING THE K NOWLEDGE OF P ROBABILISTIC M ODELS
Revised: D IMENSIONALITY R EDUCTION FOR R EPRESENTING THE K NOWLEDGE OF P ROBABILISTIC M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most deep learning models rely on expressive high-dimensional representations to achieve good performance on tasks such as classiﬁcation.
Revised: Most deep learning models rely on expressive high-dimensional representations to achieve good performance on tasks such as classiﬁcation.

Original: However, the high dimensionality of these representations makes them difﬁcult to interpret and prone to over-ﬁtting.
Revised: However, the high dimensionality of these representations makes them difﬁcult to interpret and prone to over-ﬁtting.

Original: We propose a simple, intuitive and scalable dimension reduction framework that takes into account the soft probabilistic interpretation of standard deep models for classiﬁcation.
Revised: We propose a simple, intuitive and scalable dimension reduction framework that takes into account the soft probabilistic interpretation of standard deep models for classiﬁcation.

Original: When applying our framework to visualization, our representations more accurately reﬂect inter-class distances than standard visualization techniques such as t-SNE.
Revised: When applying our framework to visualization, our representations more accurately reﬂect inter-class distances than standard visualization techniques such as t-SNE.

Original: We show experimentally that our framework improves generalization performance to unseen categories in zero-shot learning.
Revised: We show experimentally that our framework improves generalization performance to unseen categories in zero-shot learning.

Original: We also provide a ﬁnite sample error upper bound guarantee for the method.
Revised: We also provide a ﬁnite sample error upper bound guarantee for the method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dimensionality reduction is an important problem in machine learning tasks to increase classiﬁcation performance of learned models, improve computational efﬁciency, or perform visualization.
Revised: Dimensionality reduction is an important problem in machine learning tasks to increase classiﬁcation performance of learned models, improve computational efﬁciency, or perform visualization.

Original: In the context of visualization, high-dimensional representations are typically converted to two or threedimensional representations so that the underlying relations between data points can be observed and interpreted from a scatterplot.
Revised: In the context of visualization, high-dimensional representations are typically converted to two or threedimensional representations so that the underlying relations between data points can be observed and interpreted from a scatterplot.

Original: Currently, a major source of high-dimensional representations that machine learning practitioners have trouble understanding are those generated by deep neural networks.
Revised: Currently, a major source of high-dimensional representations that machine learning practitioners have trouble understanding are those generated by deep neural networks.

Original: Techniques such as PCA or t-SNE (Van der Maaten, 2014) are typically used to visualize them ( e.g. in (Law et al., 2017; Snell et al., 2017)).
Revised: Techniques such as PCA or t-SNE (Van der Maaten, 2014) are typically used to visualize them ( e.g. in (Law et al., 2017; Snell et al., 2017)).

Original: Moreover, Schulz et al. (2015) proposed a visualization technique that represents examples based on their predicted category only.
Revised: Moreover, Schulz et al. (2015) proposed a visualization technique that represents examples based on their predicted category only.

Original: However, none of these techniques exploit the fact that deep models have soft probabilistic interpretations.
Revised: However, none of these techniques exploit the fact that deep models have soft probabilistic interpretations.

Original: For instance, the output of deep classiﬁers typically employ softmax regression, which optimizes classiﬁcation scores across categories by minimizing cross entropy.
Revised: For instance, the output of deep classiﬁers typically employ softmax regression, which optimizes classiﬁcation scores across categories by minimizing cross entropy.

Original: This results in soft probabilistic representations that reﬂect the conﬁdence of the model in assigning examples to the different categories.
Revised: This results in soft probabilistic representations that reﬂect the conﬁdence of the model in assigning examples to the different categories.

Original: Many other deep learning tasks such as semantic segmentation (Long et al., 2015) or boundary/skeleton detection (Xie & Tu, 2015) also optimize for probability distributions.
Revised: Many other deep learning tasks such as semantic segmentation (Long et al., 2015) or boundary/skeleton detection (Xie & Tu, 2015) also optimize for probability distributions.

Original: In this paper, we experimentally demonstrate that the soft probability representations learned by a neural network reveal key structure about the learned model.
Revised: In this paper, we experimentally demonstrate that the soft probability representations learned by a neural network reveal key structure about the learned model.

Original: To this end, we propose a dimensionality reduction framework that transforms probability representations into a low-dimensional space for easy visualization.
Revised: To this end, we propose a dimensionality reduction framework that transforms probability representations into a low-dimensional space for easy visualization.

Original: Furthermore, our approach improves generalization.
Revised: Furthermore, our approach improves generalization.

Original: In the context of zero-shot learning where novel categories are added at test time, deep learning approaches often learn high-dimensional representations that over-ﬁt to training categories.
Revised: In the context of zero-shot learning where novel categories are added at test time, deep learning approaches often learn high-dimensional representations that over-ﬁt to training categories.

Original: By learning low-dimensional representations that match the classiﬁcation scores of a high-dimensional pre-trained model, our approach takes into account inter-class similiarities and generalizes better to unseen categories than standard approaches.
Revised: By learning low-dimensional representations that match the classiﬁcation scores of a high-dimensional pre-trained model, our approach takes into account inter-class similiarities and generalizes better to unseen categories than standard approaches.

Original: Proposed approach: We propose to exploit as input representations the probability scores generated by a high-dimensional pre-trained model, called the teacher model or target , in order to train a lower-dimensional representation, called the student .
Revised: Proposed approach: We propose to exploit as input representations the probability scores generated by a high-dimensional pre-trained model, called the teacher model or target , in order to train a lower-dimensional representation, called the student .

Original: In detail, our approach learns low-dimensional student representations of examples such that, when applying a speciﬁc soft clustering algorithm on the student representations, the predicted clustering scores are similar to the target probability scores.
Revised: In detail, our approach learns low-dimensional student representations of examples such that, when applying a speciﬁc soft clustering algorithm on the student representations, the predicted clustering scores are similar to the target probability scores.

Original: Contributions: This paper makes the following contributions: (1) we propose the ﬁrst dimensionality reduction approach optimized to consider some soft target probabilistic representations as input.
Revised: Contributions: This paper makes the following contributions: (1) we propose the ﬁrst dimensionality reduction approach optimized to consider some soft target probabilistic representations as input.

Original: By exploiting the probability representations generated by a pre-trained model, our approach reﬂects the learned semantic structure better than standard visualization approaches.
Revised: By exploiting the probability representations generated by a pre-trained model, our approach reﬂects the learned semantic structure better than standard visualization approaches.

Original: (3) We experimentally show that our approach improves generalization performance in zero-shot learning.
Revised: (3) We experimentally show that our approach improves generalization performance in zero-shot learning.

Original: (4) We theoretically analyze the statistical properties of the approach and provide a ﬁnite sample error upper bound guarantee for it.
Revised: (4) We theoretically analyze the statistical properties of the approach and provide a ﬁnite sample error upper bound guarantee for it.

Original:  D IMENSIONALITY R EDUCTION FOR P ROBABILISTIC R EPRESENTATIONS (DRPR)
Revised:  D IMENSIONALITY R EDUCTION FOR P ROBABILISTIC R EPRESENTATIONS (DRPR)

Original: Our method, called Dimensionality Reduction of Probabilistic Representations (DRPR), is given probability representations generated from high-dimensional data as target.
Revised: Our method, called Dimensionality Reduction of Probabilistic Representations (DRPR), is given probability representations generated from high-dimensional data as target.

Original: Its goal is to learn a low-dimensional representation such that the soft clustering scores predicted by a soft clustering algorithm are similar to the target.
Revised: Its goal is to learn a low-dimensional representation such that the soft clustering scores predicted by a soft clustering algorithm are similar to the target.

Original: If the targets are probability distributions generated by a pretrained classiﬁer, we want the low-dimensional space to reﬂect the relationships between categories interpreted by the classiﬁer.
Revised: If the targets are probability distributions generated by a pretrained classiﬁer, we want the low-dimensional space to reﬂect the relationships between categories interpreted by the classiﬁer.

Original: The position of each example in the low-dimensional space should then reﬂect the ambiguity of the classiﬁer for the example (see Fig. 1).
Revised: The position of each example in the low-dimensional space should then reﬂect the ambiguity of the classiﬁer for the example (see Fig. 1).

Original: We summarize in Section 2.1 the soft clustering algorithm that is used by DRPR in the low-dimensional space, the algorithm is detailed in Banerjee et al.
Revised: We summarize in Section 2.1 the soft clustering algorithm that is used by DRPR in the low-dimensional space, the algorithm is detailed in Banerjee et al.

Original: The general learning algorithm of DRPR is introduced in Section 2.2.
Revised: The general learning algorithm of DRPR is introduced in Section 2.2.

Original: 2. F AMILY OF EFFICIENT SOFT CLUSTERING ALGORITHMS
Revised: 2. F AMILY OF EFFICIENT SOFT CLUSTERING ALGORITHMS

Original: Probability density: We consider that we are given a set of n vectors f 1 , · · · , f n ∈ V concatenated into a single matrix F =
Revised: Probability density: We consider that we are given a set of n vectors f 1 , · · · , f n ∈ V concatenated into a single matrix F =

Original: [ f 1 , · · · , f n ] (cid:62) ∈ V n .
Revised: [ f 1 , · · · , f n ] (cid:62) ∈ V n .

Original: In the following, we consider V = R d , and V n = R n × d .
Revised: In the following, we consider V = R d , and V n = R n × d .

Original: The goal is to partition n examples into k soft clusters.
Revised: The goal is to partition n examples into k soft clusters.

Original: Each cluster C c with c ∈ { 1 , · · · , k } has a center ˆ µ c ∈ V and its corresponding probability density is p c ( f i ) =
Revised: Each cluster C c with c ∈ { 1 , · · · , k } has a center ˆ µ c ∈ V and its corresponding probability density is p c ( f i ) =

Original: exp( − d ( f i , ˆ µ c ))
Revised: exp( − d ( f i , ˆ µ c ))

Original: b ( f i ) , where d is a regular Bregman divergence (Banerjee et al., 2005; Bregman, 1967) and b : V → R + is a uniquely determined function that depends on d and ensures that the integral of the density over V is 1 ( e.g. b ( f i )
Revised: b ( f i ) , where d is a regular Bregman divergence (Banerjee et al., 2005; Bregman, 1967) and b : V → R + is a uniquely determined function that depends on d and ensures that the integral of the density over V is 1 ( e.g. b ( f i )

Original: =
Revised: =

Original: 1 / (2 π ) d if d is the squared Euclidean distance).
Revised: 1 / (2 π ) d if d is the squared Euclidean distance).

Original: For simplicity, we consider that d is the squared Euclidean distance.
Revised: For simplicity, we consider that d is the squared Euclidean distance.

Original: The density p c ( f i ) decreases as the divergence between the example f i and the center ˆ µ c increases.
Revised: The density p c ( f i ) decreases as the divergence between the example f i and the center ˆ µ c increases.

Original: Bregman Soft Clustering problem (BSCP): The BSCP (Banerjee et al., 2005) is deﬁned as that of learning the maximum likelihood parameters Γ = { ˆ µ c , ˆ π c } kc =1 of a mixture model p ( f i | Γ)
Revised: Bregman Soft Clustering problem (BSCP): The BSCP (Banerjee et al., 2005) is deﬁned as that of learning the maximum likelihood parameters Γ = { ˆ µ c , ˆ π c } kc =1 of a mixture model p ( f i | Γ)

Original: = (cid:80) kc
Revised: = (cid:80) kc

Original: =

Revised: =

Original: ˆ π c exp( − d ( f i , ˆ µ c ))
Revised: ˆ π c exp( − d ( f i , ˆ µ c ))

Original: b ( f i ) where ˆ π c is the prior probability that f i is generated by C c .
Revised: b ( f i ) where ˆ π c is the prior probability that f i is generated by C c .

Original: To partition the n examples into k clusters, Banerjee et al.
Revised: To partition the n examples into k clusters, Banerjee et al.

Original: (2005) apply the EM algorithm to maximize the likelihood parameter estimation problem for mixture models formulated as: max Γ (cid:81)
Revised: (2005) apply the EM algorithm to maximize the likelihood parameter estimation problem for mixture models formulated as: max Γ (cid:81)

Original: ni =1 p ( f i | Γ)
Revised: ni =1 p ( f i | Γ)

Original: Assignment matrix:
Revised: Assignment matrix:

Original: Partitioning the n observations in F into k soft clusters is equivalent to determining some soft assignment matrix ˆ Y ∈ (0 , 1) n × k in the set Y n × k of matrices whose rows are positive and sum to 1 .
Revised: Partitioning the n observations in F into k soft clusters is equivalent to determining some soft assignment matrix ˆ Y ∈ (0 , 1) n × k in the set Y n × k of matrices whose rows are positive and sum to 1 .

Original: Formally, Y n × k is written Y n × k :=
Revised: Formally, Y n × k is written Y n × k :=

Original: { ˆ Y ∈ (0 , 1) n × k : ˆ Y 1 k = 1 n } where 1 k ∈ { 1 } k is the k -dimensional vector containing only 1.
Revised: { ˆ Y ∈ (0 , 1) n × k : ˆ Y 1 k = 1 n } where 1 k ∈ { 1 } k is the k -dimensional vector containing only 1.

Original: For a given value of Γ , the element ˆ Y ic = p ( C c | f i ) ∈ (0 , 1) is the posterior probability, or responsibility of C c for f i .
Revised: For a given value of Γ , the element ˆ Y ic = p ( C c | f i ) ∈ (0 , 1) is the posterior probability, or responsibility of C c for f i .

Original: The higher the value of ˆ Y ic , the more likely f i belongs to cluster C c .
Revised: The higher the value of ˆ Y ic , the more likely f i belongs to cluster C c .

Original: Local maximum condition:
Revised: Local maximum condition:

Original: Once the BSCP has converged to a local maximum of max Γ (cid:81) ni =1 p ( f i | Γ) , the following equations are all satisﬁed:
Revised: Once the BSCP has converged to a local maximum of max Γ (cid:81) ni =1 p ( f i | Γ) , the following equations are all satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: M-step
Revised: M-step

Original: [Equation]
Revised: [Equation]

Original: and ∀
Revised: and ∀

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Eq.
Revised: Eq.

Original: (1) corresponds to the E-step of the EM algorithm that computes p ( C c | f i )
Revised: (1) corresponds to the E-step of the EM algorithm that computes p ( C c | f i )

Original: =
Revised: =

Original: ˆ Y ic when the parameters F and Γ are given.
Revised: ˆ Y ic when the parameters F and Γ are given.

Original: Eq.
Revised: Eq.

Original: (2) corresponds to the M-step which is also simple since the likelihood is a regular exponential family function (Banerjee et al., 2005).
Revised: (2) corresponds to the M-step which is also simple since the likelihood is a regular exponential family function (Banerjee et al., 2005).

Original: The M-step may be computationally expensive for other types of exponential family distributions (Banerjee et al., 2005,
Revised: The M-step may be computationally expensive for other types of exponential family distributions (Banerjee et al., 2005,

Original: Section 5.2).
Revised: Section 5.2).

Original: It is worth noting that these optimality conditions do not depend on the function b used to deﬁne p c ( f i ) , so b can be ignored.
Revised: It is worth noting that these optimality conditions do not depend on the function b used to deﬁne p c ( f i ) , so b can be ignored.

Original: 2. L EARNING THE MAPPING TO THE LOW - DIMENSIONAL SPACE
Revised: 2. L EARNING THE MAPPING TO THE LOW - DIMENSIONAL SPACE

Original: Section 2.1 explains how to perform soft clustering on some ﬁxed representation F .
Revised: Section 2.1 explains how to perform soft clustering on some ﬁxed representation F .

Original: [Figure]
Revised: [Figure]

Original: Student y
Revised: Student y

Original: i
Revised: i

Original: i
Revised: i

Original: We now describe how to learn F so that the soft clustering scores predicted by the BSCP match those of the target.
Revised: We now describe how to learn F so that the soft clustering scores predicted by the BSCP match those of the target.

Original: We assume that we are given the probability representation y i ∈ [0 , 1] k as target for each training example f i .
Revised: We assume that we are given the probability representation y i ∈ [0 , 1] k as target for each training example f i .

Original: These representations are concatenated into a single matrix
Revised: These representations are concatenated into a single matrix

Original: Y =
Revised: Y =

Original: [ y 1 , · · · , y n ] (cid:62) ∈ Y n × k which is the target of our method for F .
Revised: [ y 1 , · · · , y n ] (cid:62) ∈ Y n × k which is the target of our method for F .

Original: DRPR learns the representation of F ∈ V n such that the soft assignment matrix obtained from applying the BSCP on F is close to
Revised: DRPR learns the representation of F ∈ V n such that the soft assignment matrix obtained from applying the BSCP on F is close to

Original: Y .
Revised: Y .

Original: We ﬁrst give the formulation of our prediction function in Eq.
Revised: We ﬁrst give the formulation of our prediction function in Eq.

Original: Our dimensionality reduction problem is given in Eq.
Revised: Our dimensionality reduction problem is given in Eq.

Original: Prediction function: Let us assume that we are given the dataset matrix F =
Revised: Prediction function: Let us assume that we are given the dataset matrix F =

Original: [ f 1 , · · · , f n ] (cid:62) ∈
Revised: [ f 1 , · · · , f n ] (cid:62) ∈

Original: V n = R n × d , the centers M = [ µ 1 , · · · , µ k ] (cid:62) ∈
Revised: V n = R n × d , the centers M = [ µ 1 , · · · , µ k ] (cid:62) ∈

Original: V k
Revised: V k

Original: = R k × d and the priors π
Revised: = R k × d and the priors π

Original: = [ π 1 , · · · , π k ] (cid:62) ∈ (0 , 1) k .
Revised: = [ π 1 , · · · , π k ] (cid:62) ∈ (0 , 1) k .

Original: As in Eq.
Revised: As in Eq.

Original: (1), we deﬁne our prediction function ψ ( F, M, π ) =
Revised: (1), we deﬁne our prediction function ψ ( F, M, π ) =

Original: Ψ ∈ Y n × k as the soft assignment matrix predicted by the BSCP given F ,
Revised: Ψ ∈ Y n × k as the soft assignment matrix predicted by the BSCP given F ,

Original: M and π .
Revised: M and π .

Original: The elements of the matrix Ψ are then computed as follows:
Revised: The elements of the matrix Ψ are then computed as follows:

Original: [Equation]
Revised: [Equation]

Original: Optimization problem: DRPR learns the representation of F so that the predicted assignment matrix ψ ( F, M, π )
Revised: Optimization problem: DRPR learns the representation of F so that the predicted assignment matrix ψ ( F, M, π )

Original: =
Revised: =

Original: Ψ is similar to Y .
Revised: Ψ is similar to Y .

Original: Given the optimal condition properties of the BSCP stated in Section 2.1, the optimal values of M and π also depend on Ψ and are therefore variables of our dimensionality reduction problem that we formulate:
Revised: Given the optimal condition properties of the BSCP stated in Section 2.1, the optimal values of M and π also depend on Ψ and are therefore variables of our dimensionality reduction problem that we formulate:

Original: [Equation]
Revised: [Equation]

Original: The function ∆ n (Ψ , Y ) is an empirical discrepancy loss between the predicted assignment matrix Ψ and the target assignment matrix Y .
Revised: The function ∆ n (Ψ , Y ) is an empirical discrepancy loss between the predicted assignment matrix Ψ and the target assignment matrix Y .

Original: Since the rows of Ψ and Y represent probability distributions, we formulate ∆ n as the average KL -divergence between the rows of Y and the rows of Ψ .
Revised: Since the rows of Ψ and Y represent probability distributions, we formulate ∆ n as the average KL -divergence between the rows of Y and the rows of Ψ .

Original: Let ψ (cid:62) i and y (cid:62) i be the i -th rows of Ψ and Y , respectively, we formulate ∆ n (Ψ , Y )
Revised: Let ψ (cid:62) i and y (cid:62) i be the i -th rows of Ψ and Y , respectively, we formulate ∆ n (Ψ , Y )

Original: = 1 n (cid:80) ni =1 D KL ( y i (cid:107) ψ i ) .
Revised: = 1 n (cid:80) ni =1 D KL ( y i (cid:107) ψ i ) .

Original: Note that the choice of the discrepancy loss ∆ n is independent of the chosen Bregman divergence d .
Revised: Note that the choice of the discrepancy loss ∆ n is independent of the chosen Bregman divergence d .

Original: Moreover, the number of classes k has an impact on the number of clusters in the low-dimensional space but is not related to the dimensionality d of the model.
Revised: Moreover, the number of classes k has an impact on the number of clusters in the low-dimensional space but is not related to the dimensionality d of the model.

Original: DRPR considers that each class c is represented by one cluster prototype µ c ∈ R d .
Revised: DRPR considers that each class c is represented by one cluster prototype µ c ∈ R d .

Original: Terminology:
Revised: Terminology:

Original: In our experiments, the target (or teacher) is the assignment matrix Y ∈ Y n × k that contains the probability scores generated by a pre-trained neural network.
Revised: In our experiments, the target (or teacher) is the assignment matrix Y ∈ Y n × k that contains the probability scores generated by a pre-trained neural network.

Original: It corresponds to the output of a classiﬁer trained with softmax regression in the visualization experiments, and to the matrices Yand Y 2 described in Section 4.2.
Revised: It corresponds to the output of a classiﬁer trained with softmax regression in the visualization experiments, and to the matrices Yand Y 2 described in Section 4.2.

Original: The goal is then to learn F , M and π so that (1) F , M and π reach the BSCP optimality conditions given in Section 2.1 and (2) Ψ = ψ ( F, M, π ) is similar to Y .
Revised: The goal is then to learn F , M and π so that (1) F , M and π reach the BSCP optimality conditions given in Section 2.1 and (2) Ψ = ψ ( F, M, π ) is similar to Y .

Original: Visualization: DRPR can be used for visualization since many models ( e.g. usual neural networks) have probabilistic interpretations w.r.t.
Revised: Visualization: DRPR can be used for visualization since many models ( e.g. usual neural networks) have probabilistic interpretations w.r.t.

Original: the k training categories.
Revised: the k training categories.

Original: In our visualization task, the matrices
Revised: In our visualization task, the matrices

Original: M and π are not provided, whereas the target matrix Y is given.
Revised: M and π are not provided, whereas the target matrix Y is given.

Original: By using the optimality conditions in Eq.
Revised: By using the optimality conditions in Eq.

Original: (2), we can write the desired values of M and π as a function of F and Y : at each iteration, for some current value of F , the optimal values M = diag( Y (cid:62) 1 n ) − 1 Y (cid:62) F and π = 1 n Y (cid:62) 1 n are computed and F is updated via gradient descent.
Revised: (2), we can write the desired values of M and π as a function of F and Y : at each iteration, for some current value of F , the optimal values M = diag( Y (cid:62) 1 n ) − 1 Y (cid:62) F and π = 1 n Y (cid:62) 1 n are computed and F is updated via gradient descent.

Original: DRPR is illustrated in Algorithm 1 in the case where F is the output of a model ϕ θ parameterized by θ ( e.g. ϕ θ can be a neural network).
Revised: DRPR is illustrated in Algorithm 1 in the case where F is the output of a model ϕ θ parameterized by θ ( e.g. ϕ θ can be a neural network).

Original: However, we represent F as non-parametric embeddings in our visualization experiments to have an equitable comparison to non-parametric baselines.
Revised: However, we represent F as non-parametric embeddings in our visualization experiments to have an equitable comparison to non-parametric baselines.

Original: The learning algorithm then modiﬁes the matrix F at each iteration.
Revised: The learning algorithm then modiﬁes the matrix F at each iteration.

Original: If the priors π c are all equal, then the priors are updated in step 4 as follows: π ← 1 k 1 k .
Revised: If the priors π c are all equal, then the priors are updated in step 4 as follows: π ← 1 k 1 k .

Original: Zero-shot learning: DRPR can be used to improve zero-shot learning generalization since highdimensional models may overﬁt to training categories and the goal of zero-shot learning is to generalize to novel categories.
Revised: Zero-shot learning: DRPR can be used to improve zero-shot learning generalization since highdimensional models may overﬁt to training categories and the goal of zero-shot learning is to generalize to novel categories.

Original: In the considered zero-shot learning task, the variable F concatenates image representations (outputs of a neural network) in the same way as step 3 of Algorithm 1, and the variable M concatenates category representations extracted from text (outputs of another neural network).
Revised: In the considered zero-shot learning task, the variable F concatenates image representations (outputs of a neural network) in the same way as step 3 of Algorithm 1, and the variable M concatenates category representations extracted from text (outputs of another neural network).

Original: Both F and M are of different nature and are therefore computed as concatenating the outputs of two distinct neural networks taking different sources as input.
Revised: Both F and M are of different nature and are therefore computed as concatenating the outputs of two distinct neural networks taking different sources as input.

Original: To optimize Eq.
Revised: To optimize Eq.

Original: (4), both neural networks are trained alternately by ﬁxing the other neural network during backpropagation.
Revised: (4), both neural networks are trained alternately by ﬁxing the other neural network during backpropagation.

Original: Choice of divergence: In our experiments, we consider the squared Euclidean distance d ( f i , ˆ µ c )
Revised: Choice of divergence: In our experiments, we consider the squared Euclidean distance d ( f i , ˆ µ c )

Original: = (cid:107) f
Revised: = (cid:107) f

Original: i − ˆ µ c (cid:107) 22 .
Revised: i − ˆ µ c (cid:107) 22 .

Original: However, DRPR can be used with any regular Bregman divergence (Banerjee et al., 2005).
Revised: However, DRPR can be used with any regular Bregman divergence (Banerjee et al., 2005).

Original: The algorithm is then identical, with the exception of the chosen divergence d to compute the prediction in Eq.
Revised: The algorithm is then identical, with the exception of the chosen divergence d to compute the prediction in Eq.

Original: Convergence and scalability: Although our problem has 3 variables ( F , M and π ), we use the optimal properties of the BSCP to write them as a function of each other (see step 4 of Algo 1).
Revised: Convergence and scalability: Although our problem has 3 variables ( F , M and π ), we use the optimal properties of the BSCP to write them as a function of each other (see step 4 of Algo 1).

Original: Eq.
Revised: Eq.

Original: (4) is then an optimization problem wrt only one variable F .
Revised: (4) is then an optimization problem wrt only one variable F .

Original: Since the problem is differentiable wrt F and unconstrained, it is easy to optimize by gradient descent (e.g. back-propagation when training neural networks).
Revised: Since the problem is differentiable wrt F and unconstrained, it is easy to optimize by gradient descent (e.g. back-propagation when training neural networks).

Original: Moreover, our loss is nonnegative, hence lower-bounded.
Revised: Moreover, our loss is nonnegative, hence lower-bounded.

Original: It is worth noting that we do not apply multiple iterations of the EM algorithm at each gradient descent iteration, as we ﬁrst use the optimal properties of BSCP to obtain closed-form formulations of M and π , and then compute ψ ( F, M, π ) =
Revised: It is worth noting that we do not apply multiple iterations of the EM algorithm at each gradient descent iteration, as we ﬁrst use the optimal properties of BSCP to obtain closed-form formulations of M and π , and then compute ψ ( F, M, π ) =

Original: Ψ to minimize our problem in Eq.
Revised: Ψ to minimize our problem in Eq.

Original: Unlike t-SNE and many iterative DR problems, the complexity of DRPR is linear in n (instead of quadratic) and linear in k , which makes it efﬁcient and scalable.
Revised: Unlike t-SNE and many iterative DR problems, the complexity of DRPR is linear in n (instead of quadratic) and linear in k , which makes it efﬁcient and scalable.

Original: Our visualization experiments take less than 5 minutes to do 10 5 iterations while t-SNE takes 1 hour to do 1000 iterations.
Revised: Our visualization experiments take less than 5 minutes to do 10 5 iterations while t-SNE takes 1 hour to do 1000 iterations.

Original: PCA, which has an efﬁcient closed-form solution, is still much faster.
Revised: PCA, which has an efﬁcient closed-form solution, is still much faster.

Original: Our approach is thus simple to optimize, hence scalable, and it generalizes to a large family of Bregman divergences.
Revised: Our approach is thus simple to optimize, hence scalable, and it generalizes to a large family of Bregman divergences.

Original: Gradient interpretation: We now interpret the gradient of our optimization problem w.r.t.
Revised: Gradient interpretation: We now interpret the gradient of our optimization problem w.r.t.

Original: examples.
Revised: examples.

Original: To simplify its formulation, we consider that all the priors π 1 , · · · , π k are equal and the matrix M does not depend on F , which is the case in our zero-shot learning task.
Revised: To simplify its formulation, we consider that all the priors π 1 , · · · , π k are equal and the matrix M does not depend on F , which is the case in our zero-shot learning task.

Original: When d is the squared Euclidean distance, all the priors are equal and M does not depend on F , the gradient of Eq.
Revised: When d is the squared Euclidean distance, all the priors are equal and M does not depend on F , the gradient of Eq.

Original: (4) w.r.t.
Revised: (4) w.r.t.

Original: f i is:
Revised: f i is:

Original: [Equation]
Revised: [Equation]

Original: One can observe that its magnitude depends on both the target scores Y ic ∈ (0 , 1) and the predicted responsibilities Ψ ic .
Revised: One can observe that its magnitude depends on both the target scores Y ic ∈ (0 , 1) and the predicted responsibilities Ψ ic .

Original: The gradient tries to make the vector f
Revised: The gradient tries to make the vector f

Original: i closer to each centroid µ c while separating it from all the centroids µ m depending on their predicted scores Ψ im ∈ (0 , 1) .
Revised: i closer to each centroid µ c while separating it from all the centroids µ m depending on their predicted scores Ψ im ∈ (0 , 1) .

Original: Statistical guarantee: We analyze the statistical property of the algorithm in Appendix A. Theoremprovides a ﬁnite sample upper bound guarantee on the quality of the minimizer of the empirical discrepancy (4).
Revised: Statistical guarantee: We analyze the statistical property of the algorithm in Appendix A. Theoremprovides a ﬁnite sample upper bound guarantee on the quality of the minimizer of the empirical discrepancy (4).

Original: We show that it is upper bounded by the minimum of the true expected discrepancy, and a O ( n − 1 / 2 ) estimation error term (under certain conditions and with high probability).
Revised: We show that it is upper bounded by the minimum of the true expected discrepancy, and a O ( n − 1 / 2 ) estimation error term (under certain conditions and with high probability).

Original: We defer the detail to the appendix.
Revised: We defer the detail to the appendix.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: This paper introduces a dimensionality reduction method that represents the relations in a dataset that has probabilistic interpretation.
Revised: This paper introduces a dimensionality reduction method that represents the relations in a dataset that has probabilistic interpretation.

Original: It can be seen as a metric learning approach.
Revised: It can be seen as a metric learning approach.

Original: Metric Learning: The most related approaches try to solve the “supervised” hard clustering problem.
Revised: Metric Learning: The most related approaches try to solve the “supervised” hard clustering problem.

Original: During training, they are given the target hard assignment matrix Y ∈ { 0 , 1 } n × k where Y
Revised: During training, they are given the target hard assignment matrix Y ∈ { 0 , 1 } n × k where Y

Original: ic
Revised: ic

Original: = 1 if the training example f i has to belong to C c , and 0 otherwise.
Revised: = 1 if the training example f i has to belong to C c , and 0 otherwise.

Original: The goal is to learn a representation such that applying a hard clustering algorithm
Revised: The goal is to learn a representation such that applying a hard clustering algorithm

Original: ( e.g. , kmeans ) on the training dataset will return the desired assignment matrix Y .
Revised: ( e.g. , kmeans ) on the training dataset will return the desired assignment matrix Y .

Original: These approaches can be decomposed in 2 groups: (1) the methods that optimize some regression problem (Lajugie et al., 2014; Law et al., 2017) and exploit orthogonal projection properties that hold only in the hard clustering context, and (2) the methods that use exponential families (Mensink et al., 2012;
Revised: These approaches can be decomposed in 2 groups: (1) the methods that optimize some regression problem (Lajugie et al., 2014; Law et al., 2017) and exploit orthogonal projection properties that hold only in the hard clustering context, and (2) the methods that use exponential families (Mensink et al., 2012;

Original: Snell et al., 2017) to describe some probability score.
Revised: Snell et al., 2017) to describe some probability score.

Original: In the latter case, the learning problem is written as a multi-class logistic regression problem where the probability of a category C c given an example f i is p ( C c | f i ) =
Revised: In the latter case, the learning problem is written as a multi-class logistic regression problem where the probability of a category C c given an example f i is p ( C c | f i ) =

Original: [Equation]
Revised: [Equation]

Original: ) and δ W is the learned dissimilarity function.
Revised: ) and δ W is the learned dissimilarity function.

Original: DRPR generalizes those approaches and can also be used for hard clustering learning.
Revised: DRPR generalizes those approaches and can also be used for hard clustering learning.

Original: For instance, Snell et al.
Revised: For instance, Snell et al.

Original: (2017) is a special case of DRPR where π = 1 k 1 k , Y ∈ { 0 , 1 } n × k is a hard assignment matrix and ∆ n is the same ∆ n as ours ( i.e. , ∆ n (Ψ , Y ) = 1 n (cid:80) ni
Revised: (2017) is a special case of DRPR where π = 1 k 1 k , Y ∈ { 0 , 1 } n × k is a hard assignment matrix and ∆ n is the same ∆ n as ours ( i.e. , ∆ n (Ψ , Y ) = 1 n (cid:80) ni

Original: =1 D KL ( y i (cid:107) ψ
Revised: =1 D KL ( y i (cid:107) ψ

Original: i ) by using the convention 0 log 0 = 0 ).
Revised: i ) by using the convention 0 log 0 = 0 ).

Original: Moreover, the optimal value of M is not implicitly written as a function of F and Y in Snell et al.
Revised: Moreover, the optimal value of M is not implicitly written as a function of F and Y in Snell et al.

Original: The approach in
Revised: The approach in

Original: Mensink et al.
Revised: Mensink et al.

Original: (2012) is similar to Snell et al.
Revised: (2012) is similar to Snell et al.

Original: but only considers linear models.
Revised: but only considers linear models.

Original: In summary, both Snell et al.
Revised: In summary, both Snell et al.

Original: (2017) and Mensink et al.
Revised: (2017) and Mensink et al.

Original: (2012) do not exploit the BSCP formulation to its full potential as they consider some restricted hard clustering case.
Revised: (2012) do not exploit the BSCP formulation to its full potential as they consider some restricted hard clustering case.

Original: DRPR generalizes these approaches to the soft clustering context at no additional algorithmic complexity.
Revised: DRPR generalizes these approaches to the soft clustering context at no additional algorithmic complexity.

Original: Dimensionality reduction: Learning models by exploiting soft probability scores predicted by another pre-trained model as supervision was proposed in Ba & Caruana (2014) for classiﬁcation.
Revised: Dimensionality reduction: Learning models by exploiting soft probability scores predicted by another pre-trained model as supervision was proposed in Ba & Caruana (2014) for classiﬁcation.

Original: It was experimentally observed in Ba & Caruana (2014) that using the output of a large pre-trained neural network as supervision, instead of ground truth labels, improves classiﬁcation performance of small neural networks.
Revised: It was experimentally observed in Ba & Caruana (2014) that using the output of a large pre-trained neural network as supervision, instead of ground truth labels, improves classiﬁcation performance of small neural networks.

Original: However, in Ba & Caruana (2014), each dimension of the student representation describes the conﬁdence score of the model for one training category, which is problematic in contexts such as zero-shot learning where categories can be added or removed at test time.
Revised: However, in Ba & Caruana (2014), each dimension of the student representation describes the conﬁdence score of the model for one training category, which is problematic in contexts such as zero-shot learning where categories can be added or removed at test time.

Original: Our approach can learn a representation with dimensionality different from the number of training categories.
Revised: Our approach can learn a representation with dimensionality different from the number of training categories.

Original: It can therefore be used in zero-shot learning.
Revised: It can therefore be used in zero-shot learning.

Original: Dimensionality reduction with neural networks has been proposed in unsupervised contexts ( e.g. to maximize variance in the latent space Hinton &
Revised: Dimensionality reduction with neural networks has been proposed in unsupervised contexts ( e.g. to maximize variance in the latent space Hinton &

Original: Salakhutdinov (2006)) and in supervised contexts ( e.g. using ground truth labels as supervision
Revised: Salakhutdinov (2006)) and in supervised contexts ( e.g. using ground truth labels as supervision

Original: Salakhutdinov & Hinton 2007).
Revised: Salakhutdinov & Hinton 2007).

Original: Instead, our approach exploits probability scores generated by a teacher pre-trained model.
Revised: Instead, our approach exploits probability scores generated by a teacher pre-trained model.

Original: These scores may be very different from ground truth labels if the pretrained model does not generalize well on the dataset.
Revised: These scores may be very different from ground truth labels if the pretrained model does not generalize well on the dataset.

Original: Our approach can then help understand what was learned by the teacher by visualizing groups of examples the teacher has trouble distinguishing.
Revised: Our approach can then help understand what was learned by the teacher by visualizing groups of examples the teacher has trouble distinguishing.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the relevance of our method in two types of experiments.
Revised: We evaluate the relevance of our method in two types of experiments.

Original: The ﬁrst learns low-dimensional representations for visualization to better interpret pre-trained deep models.
Revised: The ﬁrst learns low-dimensional representations for visualization to better interpret pre-trained deep models.

Original: The second experiment exploits the probability scores generated by a pre-trained classiﬁer in the zero-shot learning context; these probability scores are used as supervision to improve performance on novel categories.
Revised: The second experiment exploits the probability scores generated by a pre-trained classiﬁer in the zero-shot learning context; these probability scores are used as supervision to improve performance on novel categories.

Original: 4. V ISUALIZATION
Revised: 4. V ISUALIZATION

Original: Interpreting deep models is a difﬁcult task, and one of the most common tools to solve that task is visualization.
Revised: Interpreting deep models is a difﬁcult task, and one of the most common tools to solve that task is visualization.

Original: Representations are most often visualized with t-SNE which does not account for the probabilistic interpretation of the learned models.
Revised: Representations are most often visualized with t-SNE which does not account for the probabilistic interpretation of the learned models.

Original: We propose to exploit probability classiﬁcation scores as input of our dimensionality reduction framework.
Revised: We propose to exploit probability classiﬁcation scores as input of our dimensionality reduction framework.

Original: In the visualization experiments of this section, DRPR learns non-parametric low-dimensional embeddings ( i.e. our representations are not
Revised: In the visualization experiments of this section, DRPR learns non-parametric low-dimensional embeddings ( i.e. our representations are not

Original: Original 3D dataset 1 2  3 04 -2 (a) t-SNE using original dataset as input - -   100 -100 - -60 -400 2060(b)
Revised: Original 3D dataset 1 2  3 04 -2 (a) t-SNE using original dataset as input - -   100 -100 - -60 -400 2060(b)

Original: t-SNE using soft clustering scores as input -100100 -100 - -60 -400 2060 80(c)
Revised: t-SNE using soft clustering scores as input -100100 -100 - -60 -400 2060 80(c)

Original: Our 2D representation -2 -1  3 -1.5 - -0.50.5 1 1.
Revised: Our 2D representation -2 -1  3 -1.5 - -0.50.5 1 1.

Original: 2 2.5(d) outputs of neural networks but vectors) as done by non-parametric baselines ( e.g. PCA).
Revised: 2 2.5(d) outputs of neural networks but vectors) as done by non-parametric baselines ( e.g. PCA).

Original: Nonetheless,
Revised: Nonetheless,

Original: DRPR can also be learned with neural networks ( e.g. as done in Section 4.2).
Revised: DRPR can also be learned with neural networks ( e.g. as done in Section 4.2).

Original: Toy dataset: As an illustrative toy experiment, we compare the behavior of t-SNE and DRPR when applied to reduce a simple artiﬁcial 3-dimensional dataset as 2D representations.
Revised: Toy dataset: As an illustrative toy experiment, we compare the behavior of t-SNE and DRPR when applied to reduce a simple artiﬁcial 3-dimensional dataset as 2D representations.

Original: The 3D dataset illustrated in Fig.
Revised: The 3D dataset illustrated in Fig.

Original: 2 (a) contains k = 6 clusters, each generated by a Gaussian model and containing 1,000 artiﬁcial points.
Revised: 2 (a) contains k = 6 clusters, each generated by a Gaussian model and containing 1,000 artiﬁcial points.

Original: To generate target soft clustering probability scores in the 3D space, we compute the relative distances of the examples to the different cluster centers and normalize them to obtain (probability) responsibilities as done in Eq.
Revised: To generate target soft clustering probability scores in the 3D space, we compute the relative distances of the examples to the different cluster centers and normalize them to obtain (probability) responsibilities as done in Eq.

Original: In detail, let us note the original 3-dimensional dataset X =
Revised: In detail, let us note the original 3-dimensional dataset X =

Original: [ x 1 , · · · , x n ] (cid:62)
Revised: [ x 1 , · · · , x n ] (cid:62)

Original: ∈ R n × 3 (where n is the number of examples)
Revised: ∈ R n × 3 (where n is the number of examples)

Original: plotted in
Revised: plotted in

Original: Fig.
Revised: Fig.

Original: 2 (a).
Revised: 2 (a).

Original: The target soft assignment matrix Y ∈ Y n × k where k = 6 is constructed by computing
Revised: The target soft assignment matrix Y ∈ Y n × k where k = 6 is constructed by computing

Original: Y ic =
Revised: Y ic =

Original: [Equation]
Revised: [Equation]

Original: 2 ) where c c ∈ R 3 is the center of the c -th Gaussian model (deﬁned by its color) in the original space, and priors are equal.
Revised: 2 ) where c c ∈ R 3 is the center of the c -th Gaussian model (deﬁned by its color) in the original space, and priors are equal.

Original: We plot in Fig.
Revised: We plot in Fig.

Original: 2 the 2D representation of our model when using Y as input/target, and the t-SNE representations obtained when using the original dataset
Revised: 2 the 2D representation of our model when using Y as input/target, and the t-SNE representations obtained when using the original dataset

Original: X as input in Fig.
Revised: X as input in Fig.

Original: 2 (b) and Y as input in Fig.
Revised: 2 (b) and Y as input in Fig.

Original: 2 (c).
Revised: 2 (c).

Original: Two observations can be made from Fig.
Revised: Two observations can be made from Fig.

Original: 2: (1) the global structure of the original dataset is better preserved with DRPR than with t-SNE; (2)
Revised: 2: (1) the global structure of the original dataset is better preserved with DRPR than with t-SNE; (2)

Original: DRPR satisﬁes the relative distances between the different clusters better than t-SNE since DRPR tries to preserve the relative responsibilities of the different clusters.
Revised: DRPR satisﬁes the relative distances between the different clusters better than t-SNE since DRPR tries to preserve the relative responsibilities of the different clusters.

Original: We quantitatively evaluate these two observations in the following.
Revised: We quantitatively evaluate these two observations in the following.

Original: It is also worth noting that it is known that distances between clusters obtained with t-SNE may not be meaningful (Wattenberg et al., 2016) as t-SNE preserves local neighborhood instead of global similarities.
Revised: It is also worth noting that it is known that distances between clusters obtained with t-SNE may not be meaningful (Wattenberg et al., 2016) as t-SNE preserves local neighborhood instead of global similarities.

Original: In the following ( i.e. in Fig. 4 and tables of results), we only consider the case where t-SNE takes as input the logits ( i.e. , classiﬁcation scores before the softmax operation) instead of probability scores since the latter case returns bad artifacts such as the one in Fig.
Revised: In the following ( i.e. in Fig. 4 and tables of results), we only consider the case where t-SNE takes as input the logits ( i.e. , classiﬁcation scores before the softmax operation) instead of probability scores since the latter case returns bad artifacts such as the one in Fig.

Original: 2 (c).
Revised: 2 (c).

Original: Other examples of bad artifacts obtained with t-SNE exploiting probability scores are provided in the supplementary material.
Revised: Other examples of bad artifacts obtained with t-SNE exploiting probability scores are provided in the supplementary material.

Original: We also provide in the supplementary material the visualizations obtained by t-SNE when replacing the (cid:96) 2 -norm in the input space by the KL -divergence and the Jensen-Shannon divergence to compare probabilistic representations that DRPR uses as input.
Revised: We also provide in the supplementary material the visualizations obtained by t-SNE when replacing the (cid:96) 2 -norm in the input space by the KL -divergence and the Jensen-Shannon divergence to compare probabilistic representations that DRPR uses as input.

Original: These lead to worse visualizations than Fig(b) in terms of preserving original clusters and their inter-cluster distances.
Revised: These lead to worse visualizations than Fig(b) in terms of preserving original clusters and their inter-cluster distances.

Original: [Table]
Revised: [Table]

Original: Quantitative evaluation metrics: We quantify the relevance of our approach with the two following evaluation metrics: (1) an adaptation of the Neighborhood Preservation Ratio (NPR) Van der Maaten & Hinton (2012): for each image i , it counts the ratio of κ nearest neighbors of i ( i.e. that have the closest probability scores w.r.t.
Revised: Quantitative evaluation metrics: We quantify the relevance of our approach with the two following evaluation metrics: (1) an adaptation of the Neighborhood Preservation Ratio (NPR) Van der Maaten & Hinton (2012): for each image i , it counts the ratio of κ nearest neighbors of i ( i.e. that have the closest probability scores w.r.t.

Original: the KL divergence) that are also in the set of κ nearest neighbors in the learned low-dimensional space ( w.r.t.
Revised: the KL divergence) that are also in the set of κ nearest neighbors in the learned low-dimensional space ( w.r.t.

Original: the Euclidean distance).
Revised: the Euclidean distance).

Original: It is averaged over all images i .
Revised: It is averaged over all images i .

Original: This metric evaluates how much images that have similar probability scores are close to each other with the student representation.
Revised: This metric evaluates how much images that have similar probability scores are close to each other with the student representation.

Original: (2) Clustering Distance Preservation Ratio (CDPR): we randomly sample 10triplets of images ( i, i + , i − ) such that the 3 images all belong to different categories and i has closer
Revised: (2) Clustering Distance Preservation Ratio (CDPR): we randomly sample 10triplets of images ( i, i + , i − ) such that the 3 images all belong to different categories and i has closer

Original: [Table]
Revised: [Table]

Original: 2.8 LLE 0.
Revised: 2.8 LLE 0.

Original: LLE   0.
Revised: LLE   0.

Original: 1.4 ISOMAP 0.
Revised: 1.4 ISOMAP 0.

Original: ISOMAP 0.
Revised: ISOMAP 0.

Original: 3.9 t-SNE 5.
Revised: 3.9 t-SNE 5.

Original: t-SNE 2.
Revised: t-SNE 2.

Original: 16.1 DRPR(Ours) 13.
Revised: 16.1 DRPR(Ours) 13.

Original: DRPR 9.
Revised: DRPR 9.

Original: probability score to i + than to i − w.r.t.
Revised: probability score to i + than to i − w.r.t.

Original: the KL divergence.
Revised: the KL divergence.

Original: The metric counts the percentage of times that the learned representation of i is closer to i + than to i − w.r.t.
Revised: The metric counts the percentage of times that the learned representation of i is closer to i + than to i − w.r.t.

Original: the Euclidean distance in the low-dimensional representation.
Revised: the Euclidean distance in the low-dimensional representation.

Original: This evaluates how well inter-cluster distances are preserved.
Revised: This evaluates how well inter-cluster distances are preserved.

Original: We evaluate our approach on the test sets of the MNIST LeCun et al.
Revised: We evaluate our approach on the test sets of the MNIST LeCun et al.

Original: STL Coates et al.
Revised: STL Coates et al.

Original: CIFAR 10 and CIFAR 100 Krizhevsky & Hinton (2009) datasets with pre-trained models that are publicly available and optimized for cross entropy.
Revised: CIFAR 10 and CIFAR 100 Krizhevsky & Hinton (2009) datasets with pre-trained models that are publicly available and optimized for cross entropy.

Original: The dimensionality of the high-dimensional representations is equal to the number of categories in the respective datasets ( i.e. 10 except for
Revised: The dimensionality of the high-dimensional representations is equal to the number of categories in the respective datasets ( i.e. 10 except for

Original: CIFAR 100 that contains 100 categories).
Revised: CIFAR 100 that contains 100 categories).

Original: Our goal is to visualize the teacher representations with 2-dimensional representations by using their probability scores as target, not the ground truth labels.
Revised: Our goal is to visualize the teacher representations with 2-dimensional representations by using their probability scores as target, not the ground truth labels.

Original: Quantitative comparisons with standard visualization techniques such as t-SNE, ISOMAP
Revised: Quantitative comparisons with standard visualization techniques such as t-SNE, ISOMAP

Original: Tenenbaum et al.
Revised: Tenenbaum et al.

Original: Locally Linear Embedding (LLE) Roweis & Saul (2000) and PCA using the 2 leading dimensions are provided in Tables 1 and 2.
Revised: Locally Linear Embedding (LLE) Roweis & Saul (2000) and PCA using the 2 leading dimensions are provided in Tables 1 and 2.

Original: We also report the scores obtained with the logit representations which are not dimensionality reduction representations but provide an estimate of the behavior of the original dataset.
Revised: We also report the scores obtained with the logit representations which are not dimensionality reduction representations but provide an estimate of the behavior of the original dataset.

Original: DRPR outperforms the dimensionality reduction baselines w.r.t.
Revised: DRPR outperforms the dimensionality reduction baselines w.r.t.

Original: both evaluation metrics and is competitive with the logit representation.
Revised: both evaluation metrics and is competitive with the logit representation.

Original: Examples that have similar probability-based representations are closer with our approach than with other dimensionality reduction baselines.
Revised: Examples that have similar probability-based representations are closer with our approach than with other dimensionality reduction baselines.

Original: DRPR also better preserves inter-cluster distances.
Revised: DRPR also better preserves inter-cluster distances.

Original: It is worth noting that DRPR exploits as much supervision as the “unsupervised” visualization baselines.
Revised: It is worth noting that DRPR exploits as much supervision as the “unsupervised” visualization baselines.

Original: Indeed, all the compared methods use as input the same source of supervision which is included in the (classiﬁer output) representations given as input.
Revised: Indeed, all the compared methods use as input the same source of supervision which is included in the (classiﬁer output) representations given as input.

Original: Qualitative results: Visualizations of pre-trained models obtained with DRPR and t-SNE are illustrated in Fig.
Revised: Qualitative results: Visualizations of pre-trained models obtained with DRPR and t-SNE are illustrated in Fig.

Original: 4 for MNIST and STL.
Revised: 4 for MNIST and STL.

Original: The visualizations for CIFAR 10 and 100 are in the supplementary material.
Revised: The visualizations for CIFAR 10 and 100 are in the supplementary material.

Original: DRPR representations contain spiked groups at the corners to better reﬂect examples that have high conﬁdence scores for one category.
Revised: DRPR representations contain spiked groups at the corners to better reﬂect examples that have high conﬁdence scores for one category.

Original: Indeed, an example in a spike at the corner of a ﬁgure has a soft assignment score w.r.t.
Revised: Indeed, an example in a spike at the corner of a ﬁgure has a soft assignment score w.r.t.

Original: its closest center close to 1.
Revised: its closest center close to 1.

Original: This means that the pre-trained model has very high conﬁdence to assign the example to the corresponding category (see illustration in Fig.
Revised: This means that the pre-trained model has very high conﬁdence to assign the example to the corresponding category (see illustration in Fig.

Original: Examples that are between multiple clusters (usually in the middle of ﬁgures) are those that are harder to classify by the model.
Revised: Examples that are between multiple clusters (usually in the middle of ﬁgures) are those that are harder to classify by the model.

Original: Detailed visualizations and analysis are provided in the supplementary material.
Revised: Detailed visualizations and analysis are provided in the supplementary material.

Original: One can observe that representations obtained with DRPR reﬂect the semantic structure between categories.
Revised: One can observe that representations obtained with DRPR reﬂect the semantic structure between categories.

Original: On MNIST, categories that contain a curve at the bottom of the digit ( i.e. , 0, 3, 5, 6, 8 and 9) are in the bottom of Fig.
Revised: On MNIST, categories that contain a curve at the bottom of the digit ( i.e. , 0, 3, 5, 6, 8 and 9) are in the bottom of Fig.

Original: 4 (left); some pairs of digits that are often hard to differentiate between and
Revised: 4 (left); some pairs of digits that are often hard to differentiate between and

Original: -SNE is the the clusters and airplane .
Revised: -SNE is the the clusters and airplane .

Original: These categories are actually hard for the model to differentiate since they contain blue backgrounds and relatively similar objects.
Revised: These categories are actually hard for the model to differentiate since they contain blue backgrounds and relatively similar objects.

Original: In particular, the STL airplane category contains many images of seaplanes lying on the sea and can then be mistaken for ships.
Revised: In particular, the STL airplane category contains many images of seaplanes lying on the sea and can then be mistaken for ships.

Original: between both categories on the t-SNE
Revised: between both categories on the t-SNE

Original: Due to lack of space, a detailed analysis for the CIFAR 100 and STL datasets is available in the supplementary material.
Revised: Due to lack of space, a detailed analysis for the CIFAR 100 and STL datasets is available in the supplementary material.

Original: A summary of the results is that categories that belong to the same superclass ( e.g. categories hamster, mouse, rabbit, shrew, squirrel are part of the superclass small mammals ) are grouped together with DRPR.
Revised: A summary of the results is that categories that belong to the same superclass ( e.g. categories hamster, mouse, rabbit, shrew, squirrel are part of the superclass small mammals ) are grouped together with DRPR.

Original: The DRPR visualization also reﬂects some semantical structure: plants and insects are on the top left; animals are on the bottom left and categories on the right are categories.
Revised: The DRPR visualization also reﬂects some semantical structure: plants and insects are on the top left; animals are on the bottom left and categories on the right are categories.

Original: Medium mammals are also represented between small mammals and large carnivores .
Revised: Medium mammals are also represented between small mammals and large carnivores .

Original: In conclusion, the quantitative results show that the representations of DRPR are meaningful since they better preserve the cluster structure and allow observation of ambiguities between categories.
Revised: In conclusion, the quantitative results show that the representations of DRPR are meaningful since they better preserve the cluster structure and allow observation of ambiguities between categories.

Original: 4. Z ERO - SHOT LEARNING
Revised: 4. Z ERO - SHOT LEARNING

Original: We consider the same zero-shot learning scenario as Reed et al.
Revised: We consider the same zero-shot learning scenario as Reed et al.

Original: and Snell et al.
Revised: and Snell et al.

Original: In particular, we test our approach on the same datasets and splits as them.
Revised: In particular, we test our approach on the same datasets and splits as them.

Original: The main goal is to learn two mappings, one for image representations and one for category representations, in a common space V .
Revised: The main goal is to learn two mappings, one for image representations and one for category representations, in a common space V .

Original: The latter mapping takes some category descriptions as input ( e.g. from text description or visual attributes).
Revised: The latter mapping takes some category descriptions as input ( e.g. from text description or visual attributes).

Original: Image representations are then learned so that they are closer to the representative of their category than to the representative of any other category.
Revised: Image representations are then learned so that they are closer to the representative of their category than to the representative of any other category.

Original: At test time, categories that were unseen during training are considered, and their representative is obtained by using the second mapping.
Revised: At test time, categories that were unseen during training are considered, and their representative is obtained by using the second mapping.

Original: An image is assigned to the category with closest representative.
Revised: An image is assigned to the category with closest representative.

Original: Training datasets: We use the medium-scaled Caltech-UCSD Birds (CUB) dataset (Welinder et al., 2010) and Oxford Flowers-102 (Flowers) dataset (Nilsback & Zisserman, 2008).
Revised: Training datasets: We use the medium-scaled Caltech-UCSD Birds (CUB) dataset (Welinder et al., 2010) and Oxford Flowers-102 (Flowers) dataset (Nilsback & Zisserman, 2008).

Original: CUB contains 11,788 bird images from 200 different species categories split into disjoint sets: 100 categories for training, 50 for validation and 50 for test.
Revised: CUB contains 11,788 bird images from 200 different species categories split into disjoint sets: 100 categories for training, 50 for validation and 50 for test.

Original: Flowers contains 8,189 ﬂower images fromdifferent species categories: 62 categories are used for training, 20 for validation and 20 for test.
Revised: Flowers contains 8,189 ﬂower images fromdifferent species categories: 62 categories are used for training, 20 for validation and 20 for test.

Original: To represent images, Reed et al.
Revised: To represent images, Reed et al.

Original: (2016) train a GoogLeNet (Szegedy et al., 2015) model whose output dimensionality is 1,024.
Revised: (2016) train a GoogLeNet (Szegedy et al., 2015) model whose output dimensionality is 1,024.

Original: For each category, Reed et al.
Revised: For each category, Reed et al.

Original: (2016) extract some text annotations from which they learn a representative vector ( e.g. , based on Char CNN-RNN (Zhang et al., 2015)).
Revised: (2016) extract some text annotations from which they learn a representative vector ( e.g. , based on Char CNN-RNN (Zhang et al., 2015)).

Original: The image representations of examples and the text representations of categories are learned jointly so that each image is more similar to the representative vector of its own category than to any other.
Revised: The image representations of examples and the text representations of categories are learned jointly so that each image is more similar to the representative vector of its own category than to any other.

Original: Supervision: We now describe how we generate the supervision/target of our model from the models pre-trained by (Reed et al., 2016;
Revised: Supervision: We now describe how we generate the supervision/target of our model from the models pre-trained by (Reed et al., 2016;

Original: Snell et al., 2017) and provided by their respective authors.
Revised: Snell et al., 2017) and provided by their respective authors.

Original: Once its training is over, Prototypical Network (Snell et al., 2017) represents each image i by some vector ˜
Revised: Once its training is over, Prototypical Network (Snell et al., 2017) represents each image i by some vector ˜

Original: f
Revised: f

Original: i and each category c by some vector ˜ µ c .
Revised: i and each category c by some vector ˜ µ c .

Original: By concatenating the different vectors into matrices ˜
Revised: By concatenating the different vectors into matrices ˜

Original: F =
Revised: F =

Original: [˜ f 1 , · · · , ˜ f n ] (cid:62) and ˜ M =
Revised: [˜ f 1 , · · · , ˜ f n ] (cid:62) and ˜ M =

Original: [˜ µ 1 , · · · , ˜ µ k ] (cid:62) and formulating ˜ π = 1 k 1 k , DRPR considers the soft assignment matrix Y 1 =
Revised: [˜ µ 1 , · · · , ˜ µ k ] (cid:62) and formulating ˜ π = 1 k 1 k , DRPR considers the soft assignment matrix Y 1 =

Original: ψ ( ˜ F, ˜ M, ˜ π ) ∈ Y n × k as target ( i.e. supervision).
Revised: ψ ( ˜ F, ˜ M, ˜ π ) ∈ Y n × k as target ( i.e. supervision).

Original: In the case of Reed et al.
Revised: In the case of Reed et al.

Original: , we consider the same preprocessing as Snell et al.
Revised: , we consider the same preprocessing as Snell et al.

Original: Each image i is represented by some vector ˇ f i , each category c is represented by some vector ˇ µ c (that
Revised: Each image i is represented by some vector ˇ f i , each category c is represented by some vector ˇ µ c (that

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: is (cid:96) 2 -normalized in Snell et al.
Revised: is (cid:96) 2 -normalized in Snell et al.

Original: The target soft assignment matrix of DRPR is then Y 2 = ψ ( ˇ F, ˇ M, 1 k 1 k ) in this case.
Revised: The target soft assignment matrix of DRPR is then Y 2 = ψ ( ˇ F, ˇ M, 1 k 1 k ) in this case.

Original: Trained models:
Revised: Trained models:

Original: In the model that Snell et al.
Revised: In the model that Snell et al.

Original: (2017) provide and that obtains 58 .
Revised: (2017) provide and that obtains 58 .

Original: 3% accuracy on
Revised: 3% accuracy on

Original: CUB, ProtoNet trains two models that take as arguments the representations learned by Reed et al.
Revised: CUB, ProtoNet trains two models that take as arguments the representations learned by Reed et al.

Original: They train one model ˜ ϕ
Revised: They train one model ˜ ϕ

Original: ˜ θ 1 for images such that ∀ i, ˜ f
Revised: ˜ θ 1 for images such that ∀ i, ˜ f

Original: i = ˜ ϕ
Revised: i = ˜ ϕ

Original: ˜ θ 1 (ˇ f i ) , and one model ˜ ϕ
Revised: ˜ θ 1 (ˇ f i ) , and one model ˜ ϕ

Original: ˜ θ 2 for text representative vectors such that ∀ c, ˜ µ c = ˜ ϕ
Revised: ˜ θ 2 for text representative vectors such that ∀ c, ˜ µ c = ˜ ϕ

Original: ˜ θ 2 ( ˇ µ c ) .
Revised: ˜ θ 2 ( ˇ µ c ) .

Original: Following Snell et al.
Revised: Following Snell et al.

Original: , we train two (neural network) models: ϕ θ 1 for images, and ϕ θ 2 for categories.
Revised: , we train two (neural network) models: ϕ θ 1 for images, and ϕ θ 2 for categories.

Original: Both of them take as input the image and category representations used to create the target soft assignment matrix ( i.e. , we take the representations learned by Reed et al.
Revised: Both of them take as input the image and category representations used to create the target soft assignment matrix ( i.e. , we take the representations learned by Reed et al.

Original: (2016) when its probability scores Y 2 are used as supervision, and the representations learned by Snell et al.
Revised: (2016) when its probability scores Y 2 are used as supervision, and the representations learned by Snell et al.

Original: (2017) otherwise).
Revised: (2017) otherwise).

Original: In this context, we alternately optimize ϕ θ 1 by ﬁxing M (which depends on ϕ θ 2 ) and optimize ϕ θ 2 by ﬁxing F (which depends on ϕ θ 1 ).
Revised: In this context, we alternately optimize ϕ θ 1 by ﬁxing M (which depends on ϕ θ 2 ) and optimize ϕ θ 2 by ﬁxing F (which depends on ϕ θ 1 ).

Original: Implementation details: We consider that the learned models ϕ θ 1 and ϕ θ 2 have the same architecture and are multilayer perceptrons (MLP) with tanh activation functions.
Revised: Implementation details: We consider that the learned models ϕ θ 1 and ϕ θ 2 have the same architecture and are multilayer perceptrons (MLP) with tanh activation functions.

Original: The number of hidden layers λ ∈ { 0 , 1 , 2 } and output dimensionality d are hyperparameters cross-validated from the accuracy on the validation set.
Revised: The number of hidden layers λ ∈ { 0 , 1 , 2 } and output dimensionality d are hyperparameters cross-validated from the accuracy on the validation set.

Original: More details on their architecture can be found in the supplementary material.
Revised: More details on their architecture can be found in the supplementary material.

Original: Results: We report the performance of our approach on the test categories of the CUB and Flowers datasets in Tables 3 and 4, respectively.
Revised: Results: We report the performance of our approach on the test categories of the CUB and Flowers datasets in Tables 3 and 4, respectively.

Original: The performance is measured as the average classiﬁcation accuracy across all unseen classes.
Revised: The performance is measured as the average classiﬁcation accuracy across all unseen classes.

Original: We use DS-SJE (Char CNN-RNN) and Prototypical Networks as supervision for our model because they are the only approaches whose pre-trained models are publicly available.
Revised: We use DS-SJE (Char CNN-RNN) and Prototypical Networks as supervision for our model because they are the only approaches whose pre-trained models are publicly available.

Original: Our approach obtains state-of-the-art results on both CUB and Flowers datasets by signiﬁcantly improving the classiﬁcation performance of the different classiﬁers.
Revised: Our approach obtains state-of-the-art results on both CUB and Flowers datasets by signiﬁcantly improving the classiﬁcation performance of the different classiﬁers.

Original: For instance, it improves the scores of 63 .
Revised: For instance, it improves the scores of 63 .

Original: 9% obtained by ProtoNet of Flowers up to 68 .
Revised: 9% obtained by ProtoNet of Flowers up to 68 .

Original: In general, it improves zero-shot learning performance of the different classiﬁers by 2% to 4 .
Revised: In general, it improves zero-shot learning performance of the different classiﬁers by 2% to 4 .

Original: Impact of dimensionality: We report in the supplementary material the performance of our model on both the validation and test sets using different numbers of hidden layers, and ranging the output dimensionality d from 16 to the dimensionality e of the input representations.
Revised: Impact of dimensionality: We report in the supplementary material the performance of our model on both the validation and test sets using different numbers of hidden layers, and ranging the output dimensionality d from 16 to the dimensionality e of the input representations.

Original: Except for linear models ( i.e. λ = 0 ), reducing the dimensionality improves generalization.
Revised: Except for linear models ( i.e. λ = 0 ), reducing the dimensionality improves generalization.

Original: This shows that the zeroshot learning performance of a given model can be signiﬁcantly improved by taking its prediction scores as supervision of our model.
Revised: This shows that the zeroshot learning performance of a given model can be signiﬁcantly improved by taking its prediction scores as supervision of our model.

Original: To study the impact of the dimensionality reduction generated
Revised: To study the impact of the dimensionality reduction generated

Original: DRPR, we also ran the codes of Reed et al.
Revised: DRPR, we also ran the codes of Reed et al.

Original: ; Snell et al.
Revised: ; Snell et al.

Original: (2017) by learning representations with dimensionality smaller than those provided (using the same ranges as those in the tables of the supp.
Revised: (2017) by learning representations with dimensionality smaller than those provided (using the same ranges as those in the tables of the supp.

Original: material).
Revised: material).

Original: This decreased their generalization performance.
Revised: This decreased their generalization performance.

Original: Therefore, directly learning a low-dimensional representation is not a sufﬁcient condition to generalize well.
Revised: Therefore, directly learning a low-dimensional representation is not a sufﬁcient condition to generalize well.

Original: Our framework that learns representations so that examples with similar ambiguities ( i.e. similar teacher predictions) are close to each other acts as a semantic regularizer.
Revised: Our framework that learns representations so that examples with similar ambiguities ( i.e. similar teacher predictions) are close to each other acts as a semantic regularizer.

Original: This is suggested by the fact that test accuracy is improved with DRPR even when e = d (as long as the MLPs contain hidden layers).
Revised: This is suggested by the fact that test accuracy is improved with DRPR even when e = d (as long as the MLPs contain hidden layers).

Original: It is worth mentioning that one test category of the CUB dataset (Indigo bunting) belongs to the
Revised: It is worth mentioning that one test category of the CUB dataset (Indigo bunting) belongs to the

Original: ImageNet dataset (Deng et al., 2009) that was used to pretrain GoogLeNet.
Revised: ImageNet dataset (Deng et al., 2009) that was used to pretrain GoogLeNet.

Original: By using the train/val/test category splits proposed by Xian et al.
Revised: By using the train/val/test category splits proposed by Xian et al.

Original: (2017), we did not observe a change of performance of the different models on CUB.
Revised: (2017), we did not observe a change of performance of the different models on CUB.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed a dimensionality reduction approach such that the soft clustering scores obtained in the low-dimensional space are similar to those given as input.
Revised: We have proposed a dimensionality reduction approach such that the soft clustering scores obtained in the low-dimensional space are similar to those given as input.

Original: We experimentally show that our approach improves generalization performance in zero-shot learning on challenging datasets.
Revised: We experimentally show that our approach improves generalization performance in zero-shot learning on challenging datasets.

Original: It can also be used to complement t-SNE, as a visualization tool to better understand learned models.
Revised: It can also be used to complement t-SNE, as a visualization tool to better understand learned models.

Original: In particular, we show that we can give a soft clustering interpretation to models that have probabilistic interpretations.
Revised: In particular, we show that we can give a soft clustering interpretation to models that have probabilistic interpretations.

Original: Real-world applications that can be used with DRPR include distillation .
Revised: Real-world applications that can be used with DRPR include distillation .

Original: For instance, when the teacher model is too large to store on a device with small memory (e g. mobile phone), the student model which has a smaller memory footprint is used instead.
Revised: For instance, when the teacher model is too large to store on a device with small memory (e g. mobile phone), the student model which has a smaller memory footprint is used instead.

Original: Low-dimensional representations can also speed up retrieval tasks.
Revised: Low-dimensional representations can also speed up retrieval tasks.

Original: Future work includes applying our approach to the task of distillation in the standard classiﬁcation task where training categories are also test categories.
Revised: Future work includes applying our approach to the task of distillation in the standard classiﬁcation task where training categories are also test categories.

Original: A T HEORETICAL A NALYSIS
Revised: A T HEORETICAL A NALYSIS

Original: This section provides a statistical guarantee for the algorithm presented in Section 2.2.
Revised: This section provides a statistical guarantee for the algorithm presented in Section 2.2.

Original: We show that the minimizer of the empirical discrepancy (4), which uses a ﬁnite number of data points x 1 , · · · , x n ∈ X , is close to the minimizer of the true expected discrepancy measure, to be deﬁned shortly.
Revised: We show that the minimizer of the empirical discrepancy (4), which uses a ﬁnite number of data points x 1 , · · · , x n ∈ X , is close to the minimizer of the true expected discrepancy measure, to be deﬁned shortly.

Original: Let us deﬁne the setup here.
Revised: Let us deﬁne the setup here.

Original: We are given data points D n = { X 1 , . . .
Revised: We are given data points D n = { X 1 , . . .

Original: X n } .
Revised: X n } .

Original: We suppose that each X i ∈
Revised: We suppose that each X i ∈

Original: X is independent and identically distributed (i.i.d.) with the distribution ν ∈ M ( X ) , where M ( X ) is the space of all probability distributions deﬁned over X .
Revised: X is independent and identically distributed (i.i.d.) with the distribution ν ∈ M ( X ) , where M ( X ) is the space of all probability distributions deﬁned over X .

Original: The teacher is a ﬁxed function φ = [ φ ( · ; 1) , . .
Revised: The teacher is a ﬁxed function φ = [ φ ( · ; 1) , . .

Original: , φ ( · ; k )] that maps points in X to a k -dimensional simplex, and provides the target probability distributions.
Revised: , φ ( · ; k )] that maps points in X to a k -dimensional simplex, and provides the target probability distributions.

Original: That is, the target y
Revised: That is, the target y

Original: i for X i is computed as y i =
Revised: i for X i is computed as y i =

Original: φ ( X i ) .
Revised: φ ( X i ) .

Original: Consider a function space G whose domain is X and its range is a subset of R d .
Revised: Consider a function space G whose domain is X and its range is a subset of R d .

Original: This function space might be represented by a DNN, but we do not make such an assumption in our statistical analysis.
Revised: This function space might be represented by a DNN, but we do not make such an assumption in our statistical analysis.

Original: Given a function g ∈ G and the number of clusters k , we deﬁne ψ g ( x ) =
Revised: Given a function g ∈ G and the number of clusters k , we deﬁne ψ g ( x ) =

Original: [ ψ g ( x ; 1) , . . . , ψ g ( x ; k )] as
Revised: [ ψ g ( x ; 1) , . . . , ψ g ( x ; k )] as

Original: [Equation]
Revised: [Equation]

Original: , with the cluster centres
Revised: with the cluster centres

Original: [Equation]
Revised: [Equation]

Original: and the priors
Revised: and the priors

Original: [Equation]
Revised: [Equation]

Original: for c ∈ { 1 , . . .
Revised: for c ∈ { 1 , . . .

Original: , k } (cf.
Revised: , k } (cf.

Original: Section 2.1).
Revised: Section 2.1).

Original: Note that ψ g ( x ) deﬁnes a k -dimensional probability distribution, and µ  ( g ) =
Revised: Note that ψ g ( x ) deﬁnes a k -dimensional probability distribution, and µ c ( g ) is a mapping of the function g to a point in R d . Similarly, deﬁne the empirical cluster centres

Original: [ µ 1 ( g  )  , . . . , µ k ( g  )]is a mapping of the function g to a point in R d .
Revised: [Equation] and their corresponding ˆ ψ g ( x ) = [ ˆ ψ g ( x ; 1) , . . . , ˆ ψ g ( x ; k )]

Revised: [Equation]

Revised: Note that here for simplicity of analysis, we assume that the priors π c are exact, and not estimated from data.

Original: Our goal is to ﬁnd a g such that ψ g , learned by the student, is close to φ .
Revised: Our goal is to ﬁnd a g such that ψ g , learned by the student, is close to φ .

Original: In this paper, the closeness is deﬁned based on their KL divergence.
Revised: In this paper, the closeness is deﬁned based on their KL divergence.

Original: Speciﬁcally, Algorithm 1 minimizes the  empirical discrepancy (4), which can be written as
Revised: Speciﬁcally, Algorithm 1 minimizes the distorted empirical discrepancy (4), which can be written as

Original: [Equation]
Revised: [Equation]

Original: where X i s are from dataset D n .
Revised: where X i s are from dataset D n .

Revised: Notice that the distorted empirical discrepancy ∆ n ( ˆ ψ, φ ) is deﬁned based on ˆ ψ , which uses the empirical centres ˆ µ c , instead of the expected centres µ c .

Revised: We also deﬁne an empirical discrepancy w.r.t.

Revised: the true µ c is ∆ n ( ψ, φ ) , which is not directly accessible to the algorithm.

Original: We evaluate the quality of g , and its corresponding ψ g , based on how well, in average, it performs on new points x ∈ X .
Revised: We evaluate the quality of g , and its corresponding ψ g , based on how well, in average, it performs on new points x ∈ X .

Original: We consider the expected KL divergence between ψ and φ w.r.t.
Revised: We consider the expected KL divergence between ψ and φ w.r.t.

Original: distribution ν as the measure of performance.
Revised: distribution ν as the measure of performance.

Original: Therefore, the discrepancy is
Revised: Therefore, the discrepancy is

Original: [Equation]
Revised: [Equation]

Original: The output of Algorithm 1 is the minimizer 5 of ∆ n ( ψ, φ ) , which we denote by ˆ g
Revised: The output of Algorithm 1 is the minimizer 5 of ∆ n ( ψ, φ ) , which we denote by ˆ g

Original: [Equation]
Revised: [Equation]

Original: We also deﬁne the minimizer of the discrepancy by g ∗ :
Revised: We also deﬁne the minimizer of the discrepancy by g ∗ :

Original: [Equation]
Revised: [Equation]

Original: Since our focus is on the statistical analysis, we ignore the issue of the quality of

Original: We would like to compare the performance of ˆ g when evaluated according to discrepancy, that is ∆( ψ ˆ g , φ ) , and compare it with ∆( ψ g ∗ , φ ) .
Revised: We would like to compare the performance of ˆ g when evaluated according to discrepancy, that is ∆( ψ ˆ g , φ ) , and compare it with ∆( ψ g ∗ , φ ) .

Original: Before stating our results, we enlist our assumptions.
Revised: Before stating our results, we enlist our assumptions.

Original: We shall remark on them as we introduce.
Revised: We shall remark on them as we introduce.

Original: Assumption A1 (Samples)
Revised: Assumption A1 (Samples)

Original: The dataset D n = { X i } ni =1 consists of i.i.d.
Revised: The dataset D n = { X i } ni =1 consists of i.i.d.

Original: samples drawn from ν ( X ) .
Revised: samples drawn from ν ( X ) .

Original: The i.i.d.
Revised: The i.i.d.

Original: assumption simpliﬁes the analysis.
Revised: assumption simpliﬁes the analysis.

Original: With extra effort, one can provide similar results for some classes of dependent processes too.
Revised: With extra effort, one can provide similar results for some classes of dependent processes too.

Original: For example, if the dependent process comes from a time series and it gradually “forget” its past, one may still obtain similar statistical guarantees.
Revised: For example, if the dependent process comes from a time series and it gradually “forgets” its past, one may still obtain similar statistical guarantees.

Original: Forgetting can be formalized through the notion of a “mixing” of the underlying stochastic process (Doukhan, 1994).
Revised: Forgetting can be formalized through the notion of a “mixing” of the underlying stochastic process (Doukhan, 1994).

Original: One can then provide statistical guarantees for learning algorithms under various mixing conditions (Yu, 1994; Meir, 2000; Steinwart & Christmann, 2009; Mohri & Rostamizadeh, 2009; 2010; Farahmand & Szepesvári, 2012).
Revised: One can then provide statistical guarantees for learning algorithms under various mixing conditions (Yu, 1994; Meir, 2000; Steinwart & Christmann, 2009; Mohri & Rostamizadeh, 2009; 2010; Farahmand & Szepesvári, 2012).

Original: Assumption A2 (Bounded Function Space) Functions in G are L -bounded for some L > 0 , i.e., g ( x ) ∈
Revised: Assumption A2 (Bounded Function Space) Functions in G are L -bounded for some L > 0 , i.e., g ( x ) ∈

Original: [ − L, L ] d for any g ∈ G and x ∈ X .
Revised: [ − L, L ] d for any g ∈ G and x ∈ X .

Original: This is a mild and realistic assumption on the function space G , and is mainly here to simplify some steps of the analysis.
Revised: This is a mild and realistic assumption on the function space G , and is mainly here to simplify some steps of the analysis.

Original: Assumption A3 (Teacher) 
Revised: Assumption A3 (Teacher) Part I)

Original: The output of the teacher φ is a probability distribution, i.e., for any x ∈ X and c = { 1 , . . .
Revised: The output of the teacher φ is a probability distribution, i.e., for any x ∈ X and c = { 1 , . . .

Original: , k } , we have φ ( x ; c ) ≥ 0 and (cid:80) kc =1 φ ( x ; c ) = 1 . 
Revised: , k } , we have φ ( x ; c ) ≥ 0 and (cid:80) kc =1 φ ( x ; c ) = 1 . Part II) Moreover, we assume that π c = E

Revised: [ φ c ( X )] is bounded away from zero for all c , and we set π min =

Revised: min c π c .

Original: This  assumption explicitly expresses the fact that the algorithm expects to receive a probability distribution from the teacher.
Revised: This ﬁrst part assumption explicitly expresses the fact that the algorithm expects to receive a probability distribution from the teacher.

Original: If it is not, for example if φ ( x ; c ) is negative for some x and c but we treat it as a probability in the calculation of the KL divergence, the algorithm would not be well-deﬁned.
Revised: If it is not, for example if φ ( x ; c ) is negative for some x and c but we treat it as a probability in the calculation of the KL divergence, the algorithm would not be well-deﬁned.

Revised: The second part of this assumption requires that prior probability for each cluster is bounded away from zero, and has the probability at least π min .

Original: We need to make some assumptions about the function space G and its complexity (i.e., capacity).
Revised: We need to make some assumptions about the function space G and its complexity, i.e., capacity.

Original: We use covering number (and its logarithm, i.e., metric entropy) as the characterizer of the complexity.
Revised: We use covering number (and its logarithm, i.e., metric entropy) as the characterizer of the complexity.

Original: The covering number at resolution ε is the minimum number of balls with radius ε required to cover the space M according to a particular metric.
Revised: The covering number at resolution ε is the minimum number of balls with radius ε required to cover the space M according to a particular metric.

Original: We use N ( ε, G , (cid:107)·(cid:107) ) to denote the covering number of G w.r.t.
Revised: We use N ( ε, G , (cid:107)·(cid:107) ) to denote the covering number of G w.r.t.

Original: the norm (cid:107)·(cid:107) , which we shall explicitly specify.
Revised: the norm (cid:107)·(cid:107) , which we shall explicitly specify.

Original: As ε decreases, the covering number increases (or more accurately, the covering number is non-decreasing).
Revised: As ε decreases, the covering number increases (or more accurately, the covering number is non-decreasing).

Original: For example, the covering number for a p -dimensional linear function approximator with constraint on the magnitude of its functions behaves like O ( 1 ε p ) .
Revised: For example, the covering number for a p -dimensional linear function approximator with constraint on the magnitude of its functions behaves like O ( 1 ε p ) .

Original: A similar result holds when the subgraphs of a function space has a VC-dimension p .
Revised: A similar result holds when the subgraphs of a function space has a VC-dimension p .

Original: Function spaces whose covering number grows faster are more complex, and estimating a function within them is more difﬁcult.
Revised: Function spaces whose covering number grows faster are more complex, and estimating a function within them is more difﬁcult.

Original: This leads to larger estimation error.
Revised: This leads to larger estimation error.

Original: On the other hand, those function spaces are often (but not always) show better function approximation properties too.
Revised: On the other hand, those function spaces are often (but not always) show better function approximation properties too.

Original: For more detail and many examples, refer to Györﬁ et al.
Revised: For more detail and many examples, refer to Györﬁ et al.

Original: (2002); van de Geer (2000); Steinwart & Christmann (2008);
Revised: (2002); van de Geer (2000); Steinwart & Christmann (2008);

Original: Giné & Nickl (2015).
Revised: Giné & Nickl (2015).

Original: Let us deﬁne a norm for the function space G :
Revised: Let us deﬁne a norm for the function space G :

Original: [Equation]
Revised: [Equation]

Original: This is a mixed-norm where we compute the (cid:96) 2 -norm for each g ( x ) ∈ R d , and then take the supremum norm over the resulting (cid:96) 2 -norm.
Revised: This is a mixed-norm where we compute the (cid:96) 2 -norm for each g ( x ) ∈ R d , and then take the supremum norm over the resulting (cid:96) 2 -norm.

Original: We use this norm to characterize the covering number of G .
Revised: We use this norm to characterize the covering number of G .

Original: Assumption A4 (Metric Entropy)
Revised: Assumption A4 (Metric Entropy)

Original: There exists constants B > 0 and 0 < α < 1 such that for any ε , the following covering number (i.e., metric entropy) condition is satisﬁed:
Revised: There exists constants B > 0 and 0 < α < 1 such that for any ε , the following covering number (i.e., metric entropy) condition is satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: The logarithm of the covering number of G is O ( 1 ε 2 α ) .
Revised: The logarithm of the covering number of G is O ( 1 ε 2 α ) .

Original: It grows much faster than the metric entropy of linear models, which is O ( p log( 1 ε )) .
Revised: It grows much faster than the metric entropy of linear models, which is O ( p log( 1 ε )) .

Original: This behaviour is suitable to capture the complexity of large function spaces such as the Sobolev space W k ( R d ) and many reproducing kernel Hilbert spaces (RKHS).
Revised: This behaviour is suitable to capture the complexity of large function spaces such as the Sobolev space W k ( R d ) and many reproducing kernel Hilbert spaces (RKHS).

Original: 6 Note that have we use a mixed-norm to deﬁne the covering number.
Revised: 6 Note that have we use a mixed-norm to deﬁne the covering number.

Original: The use of supremum norm in the deﬁnition might be considered conservative.
Revised: The use of supremum norm in the deﬁnition might be considered conservative.

Original: Using a more relaxed norm, for example based on the empirical L p ( P X 1: n ) -norm for some 1 ≤ p < ∞ , is a technical open question for future work.
Revised: Using a more relaxed norm, for example based on the empirical L p ( P X 1: n ) -norm for some 1 ≤ p < ∞ , is a technical open question for future work.

Original: Finally let us deﬁne the pointwise loss function
Revised: Finally let us deﬁne the pointwise loss function

Original: [Equation]
Revised: [Equation]

Original: Notice that ∆ n ( ψ g , φ ) = 1 n (cid:80) ni =1 l ( X i ; g ) and ∆( ψ g , φ ) =
Revised: Notice that ∆ n ( ψ g , φ ) = 1 n (cid:80) ni =1 l ( X i ; g ) and ∆( ψ g , φ ) =

Original: E ν [ l ( X ; g )] .
Revised: E ν [ l ( X ; g )] .

Original: We deﬁne the following function space:
Revised: We deﬁne the following function space:

Original: [Equation]
Revised: [Equation]

Revised: We also deﬁne the entropy integral (Dudley integral)

Revised: [Equation]

Original: We are now ready to state the main theoretical result of this section.
Revised: We are now ready to state the main theoretical result of this section.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose that Assumptions A1, A2, and A3 hold.
Revised: Suppose that Assumptions A1, A2, and A3 hold.

Original: Consider ˆ g obtained by solving (6) .
Revised: Consider ˆ g obtained by solving (6) .

Original: Deﬁne the entropy integral

Original: [Equation]

Original: There exists a ﬁnite c 1 > 0 such that for any δ > 0 , with probability at least 1 − δ , we have
Revised: There exists a ﬁnite c 1 > 0 such that for any δ > 0 , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if the metric entropy satisﬁes Assumption A4, there exist a constant c 2  > 0 and a function [Equation] > 0 , which depends only on α , such that  any δ > 0 ,with probability at least 1 − δ , we have
Revised: Furthermore, if the metric entropy satisﬁes Assumption A4, there exist a constant c 2 , c 4 > 0 and a function c 3 ( α ) > 0 , which depends only on α , such that for any δ > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: n [Equation] This theorem provides a ﬁnite sample error upper bound on the true discrepancy of ˆ g , and relates it to the expressiveness and complexity of the function space G , the number of samples n , and some other properties.
Revised: This theorem provides a ﬁnite sample error upper bound on the true discrepancy of ˆ g , and relates it to the expressiveness and complexity of the function space G , the number of samples n , and some other properties.

Original: The term min g ∈G ∆( ψ g , φ ) =
Revised: The term min g ∈G ∆( ψ g , φ ) =

Original: ∆( ψ g ∗ , φ ) is the function approximation error, and reﬂects the expressiveness of G .
Revised: ∆( ψ g ∗ , φ ) is the function approximation error, and reﬂects the expressiveness of G .

Original: This is the minimum achievable error.
Revised: This is the minimum achievable error.

Original: Let us focus on the second part of the theorem, which is under the particular choice of covering number according to Assumption A4.
Revised: Let us focus on the second part of the theorem, which is under the particular choice of covering number according to Assumption A4.

Original: In that case, the estimation error shows n − 1 / 2 dependence on the number of samples, and hence decreases as we have more training data.
Revised: In that case, the estimation error shows n − 1 / 2 dependence on the number of samples, and hence decreases as we have more training data.

Original: We observe that the upper bound increases as the range L of the function space G , the dimension d of the low-dimensional space, and the number of clusters k increases.
Revised: We observe that the upper bound increases as the range L of the function space G , the dimension d of the low-dimensional space, and the number of clusters k increases.

Revised: The effect of using the distorted empirical discrepancy shows itself in the last term.

Original: Note, however, that  the dependence on k  is logarithmic, and grows slowly.
Revised: Note that the appearance of π min in the denominator of the last term. This causes a large estimation error if one of the classes is not very probable. Also note that π min ≤ 1 k (equality under the uniform distribution over classes), so we have a linear in k -term in the upper bound. We believe it is possible to improve this upper bound.

Original: Proof.
Revised: Proof.

Original: To simplify the notation, we denote ∆ n ( g ) =
Revised: To simplify the notation, we denote ∆ n ( g ) =

Original: ∆ n ( ψ g , φ )  and ∆( g ) =
Revised: ∆ n ( ψ g , φ ) , ˆ∆ n ( g ) = ∆ n ( ˆ ψ g , φ ) , and ∆( g ) =

Original: ∆ n ( ψ g , φ ) .
Revised: ∆ n ( ψ g , φ ) .

Original: We ﬁrst relate ∆(ˆ g ) to ∆( g ∗ )  and the supremum of the empirical process ∆( g ) − ∆ n ( g ) , as follows
Revised: We want to relate ∆(ˆ g ) to ∆( g ∗ ) , and the supremum of the empirical process ∆( g ) − ∆ n ( g ) and the supremum of the distortion of the empirical loss. We have the following relations:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is because of the the optimizer property of ˆ g , i.e., ∆ n (ˆ g ) ≤ ∆ n ( g ) for any g ∈ G , including g ∗ .
Revised: The ﬁrst inequality is because of the the optimizer property of ˆ g , i.e., ˆ∆ n (ˆ g ) ≤ ˆ∆ n ( g ) for any g ∈ G , including g ∗ .

Revised: We need to upper bound the supremum of the empirical process, sup g ∈G | ∆( g ) − ∆ n ( g ) | , and the supremum of the distortion caused by using ˆ ψ in minimizing ∆ n instead of ψ , see (6).

Original:  We use Lemma 5 in Appendix B.3 in order to upper bound the supremum of the empirical process sup g ∈G | ∆( g ) − ∆ n ( g ) | , which is equivalent to sup l ∈L (cid:12)(cid:12) 1 n (cid:80) ni
Revised: Upper Bounding sup g ∈G | ∆( g ) − ∆ n ( g ) | . We use Lemma 7 in Appendix B.3 in order to upper bound the supremum of the empirical process, sup g ∈G | ∆( g ) − ∆ n ( g ) | , which is equivalent to sup l ∈L (cid:12)(cid:12) 1 n (cid:80) ni

Original: =1 l ( X i ) − E
Revised: =1 l ( X i ) − E

Original: [ l ( X )] (cid:12)(cid:12) .
Revised: [ l ( X )] (cid:12)(cid:12) .

Original: That lemma, which is originally Theorem 2.1 of Bartlett et al.
Revised: That lemma, which is originally Theorem 2.1 of Bartlett et al.

Original: (2005), relates the supremum of the empirical process to the Rademacher complexity of L , deﬁned in the same appendix.
Revised: (2005), relates the supremum of the empirical process to the Rademacher complexity of L , deﬁned in the same appendix.

Original: To apply the lemma, we ﬁrst provide upper bound on l ( x ) and Var [ l ( X )] .
Revised: To apply the lemma, we ﬁrst provide upper bound on l ( x ) and Var [ l ( X )] .

Original: By assumption, g ( x ) is within [ − L, L ] d .
Revised: By assumption, g ( x ) is within [ − L, L ] d .

Original: It is not difﬁcult to see that µ c ( g ) is also within [ − L, L ] d (for example, see the proof leading to (16)).
Revised: It is not difﬁcult to see that µ c ( g ) is also within [ − L, L ] d (for example, see the proof leading to (19)).

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: We evoke Proposition 4 with the choice of f ( x ; c ) = (cid:107) g
Revised: We evoke Proposition 4 with the choice of f ( x ; c ) = (cid:107) g

Original: ( x ) − µ c (cid:107) 22 and L (cid:48)
Revised: ( x ) − µ c (cid:107) 22 and L (cid:48)

Original: = 4 dL 2 to obtain that
Revised: = 4 dL 2 to obtain that

Original: [Equation]
Revised: [Equation]

Original: Since l ( x ; g ) is bounded, we have
Revised: Since l ( x ; g ) is bounded, we have

Original: [Equation]
Revised: [Equation]

Original: By the choice of β = 1 , B = (8 dL 2 + log 2 k ) , and r = (8 dL 2 + log 2 k ) 2 in Lemma5, we get that for any δ > 0 ,
Revised: By the choice of β = 1 , B = (8 dL 2 + log 2 k ) , and r = (8 dL 2 + log 2 k ) 2 in Lemma

Original: [Equation]
Revised: [Equation]

Revised: we get that for any δ 1 > 0 ,

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ  .
Revised: with probability at least 1 − δ 1 .

Original: It remains to provide an upper bound on E [ R n ( L )] .
Revised: It remains to provide an upper bound on E [ R n ( L )] .

Original: This can be done by using Dudley’s integral to relate the Rademacher complexity of L to the covering number of L .
Revised: This can be done by using Dudley’s integral to relate the Rademacher complexity of L to the covering number of L .

Original: Afterwards, we use Lemmato relate the covering number of L to the covering number of G .
Revised: Afterwards, we use Lemmato relate the covering number of L to the covering number of G .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: In the second inequality, we used two observations:
Revised: In the second inequality, we used two observations:

Original: ﬁrst, we use l ( x ) ≤ 8 dL 2 + log 2 k for any l ∈ L (9) to upper bound diam ( L ) ; second, the covering number w.r.t.
Revised: ﬁrst, we use l ( x ) ≤ 8 dL 2 + log 2 k for any l ∈ L (11) to upper bound diam ( L ) ; second, the covering number w.r.t.

Original: L 2 ( P X 1: n ) can be upper bounded by the covering number w.r.t.
Revised: L 2 ( P X 1: n ) can be upper bounded by the covering number w.r.t.

Original: the supremum norm.
Revised: the supremum norm.

Revised: Upper Bounding sup g ∈G (cid:12)(cid:12)(cid:12) ∆ n ( g ) − ˆ∆ n ( g ) (cid:12)(cid:12)(cid:12) .

Revised: We use Proposition 5 in Appendix A.1, which states that for any δ 2 > 0 ,

Revised: [Equation]

Revised: with probability at least 1 − δ 2 .

Original: Plugging (10) in (8) and using the entropy integral upper bound (11) lead to the desired result of the ﬁrst part.
Revised: Plugging (12) and (14) in (10) and using the entropy integral upper bound (13) lead to the desired result of the ﬁrst part.

Original: To prove the second part of the theorem, we use log N (cid:16) ε, G , (cid:107)·(cid:107) ∞ , [-2 (cid:17) ≤ (cid:0) Bε (cid:1) 
-]
Revised: To prove the second part of the theorem, we use log N (cid:16) [Equation] , G , [-2 (cid:17) ≤ (cid:0) Bε (cid:1) 
-][Equation]

Original:  α to calculate J ( ε ) , which results in
Revised: 2 α to calculate J ( ε ) , which results in

Original: [Equation]
Revised: [Equation]

Original: By plugging in ε = 16 dL 2 + 2 log
Revised: By plugging in ε = 16 dL 2 + 2 log

Original: k , we get 
Revised: k ,we get that

Revised: [Equation]

Revised: We upper bound J (2 L ) in (14) to obtain:

Original: [Equation]
Revised: [Equation]

Original: which after some simpliﬁcations leads to the desired result of the second part.
Revised: After some simpliﬁcations these lead to the desired result of the second part.

Original: In the rest of this section, we  some tools required in the proof of Theorem 1.
Revised: In the rest of this section, we develop some tools required in the proof of Theorem 1.

Original: Proposition 2 provides Lipschitz constant for some functions.
Revised: Propositionprovides Lipschitz constant for some functions.

Original: Those functions are used in the proof of Lemma 3 to relate the covering number of L (7) to that of G .
Revised: Those functions are used in the proof of Lemmato relate the covering number of L (8) to that of G .

Original: This was a key step of the proof of the theorem.
Revised: This was a key step of the proof of the theorem.

Original: Finally, Proposition 4 provides an upper bound on the magnitude of l ( x ; f ) .
Revised: Finally, Proposition 4 provides an upper bound on the magnitude of l ( x ; f ) .

Original: We introduce a few more notations to reduce the clutter.
Revised: We introduce a few more notations to reduce the clutter.

Original: Let d g ( x ; c ) = (cid:107) g
Revised: Let d g ( x ; c ) = (cid:107) g

Original: ( x ) − µ c ( g ) (cid:107) 22 , so we can write
Revised: ( x ) − µ c ( g ) (cid:107) 22 , so we can write

Original: [Equation]
Revised: [Equation]

Original: In the rest of this section, we denote d g ( x, c ) by f ( x, c ) .
Revised: We denote d g ( x, c ) by f ( x, c ) .

Original: Wealso deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: We overload the pointwise loss function l ( x ; g )  and deﬁne a similar deﬁnition for l ( x ; f ) as follows
Revised: We overload the pointwise loss function l ( x ; g ) , deﬁned in (7), and deﬁne a similar deﬁnition for l ( x ; f ) as follows

Original: [Equation]
Revised: [Equation]

Original: It is clear that with the choice of f = d g , the probability distribution p f is the same as ψ g .
Revised: It is clear that with the choice of f = d g , the probability distribution p f is the same as ψ g .

Original: The following proposition provides Lipschitzness properties of d g and l ( x ; f ) .
Revised: The following proposition provides Lipschitzness properties of d g and l ( x ; f ) .

Original: Proposition 2.
Revised: Proposition 2.

Original: Suppose that Assumption A3 hold.
Revised: Suppose that Assumption A3 hold.

Original: Part I) Consider g 1 , g 2 ∈ G and let Assumption A2 hold.
Revised: Part I) Consider g 1 , g 2 ∈ G and let Assumption A2 hold.

Original: For any x ∈ X and c ∈ { 1 , . . . , k } , we have
Revised: For any x ∈ X and c ∈ { 1 , . . . , k } , we have

Original: [Equation]
Revised: [Equation]

Original: Part II) Consider two functions f 1 , f 2 : X × { 1 , . .
Revised: Part II) Consider two functions f 1 , f 2 : X × { 1 , . .

Original: . , k }
Revised: . , k }

Original: → Rd .
Revised: → R .

Original: For any x ∈ X we have
Revised: For any x ∈ X we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Part I) Consider g 1 , g 2 ∈ G , and their corresponding d g 1 and d g 2 .
Revised: Part I) Consider g 1 , g 2 ∈ G [Equation] and their corresponding d g 1 and d g 2 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where we used Jensen’s inequality and the fact that φ c ( x ) ≥ 0 .
Revised: where we used Jensen’s inequality and the fact that φ c ( x ) ≥ 0 .

Original: As µ c (0) =
Revised: As µ c (0) =

Original: [Equation]
Revised: [Equation]

Original: we also obtain that
Revised: we also obtain that

Original: [Equation]
Revised: [Equation]

Original: [Equation] As g ( x ) ∈ [ − L, L ] d , it holds that (cid:107) g
Revised: As g ( x ) ∈ [ − L, L ] d , it holds that (cid:107) g

Original: ( x ) (cid:107) 2 ≤ L √ d .
Revised: ( x ) (cid:107) 2 ≤ L √ d .

Original: Therefore by (14), (15), and (16),
Revised: Therefore by (17), (18), and (19),

Original: [Equation]
Revised: [Equation]

Original: Part II) For functions f 1 , f 2 , using the deﬁnition of l ( x ; f ) (13), we get that
Revised: Part II) For functions f 1 , f 2 , using the deﬁnition of l ( x ; f ) (16), we get that

Original: [Equation]
Revised: [Equation]

Original: By substituting the deﬁnition of (12) and some simpliﬁcations
Revised: By substituting the deﬁnition of (15) and some simpliﬁcations

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Revised: u

Revised: [Equation]

Revised: [Equation]

Original: where u ∈ R k .
Revised: where u ∈ R k .

Original: We study the Lipschitz property of ρ ( u ) as a function of u in order toprovide and upper bound the second term on the right-hand side (RHS).
Revised: We study the Lipschitz property of ρ ( u ) as a function of u in order to upper bound the second term on the right-hand side (RHS).

Original: We take the derivative of ρ ( u ) w.r.t.
Revised: We take the derivative of ρ ( u ) w.r.t.

Original: the c -th component of u to obtain that
Revised: the c -th component of u to obtain that

Original: [Equation]
Revised: [Equation]

Original: Notice that q c ( u ) is a probability distribution.
Revised: Notice that q c ( u ) is a probability distribution.

Original: By Taylor’s theorem, for u, u (cid:48) ∈ R d , we have
Revised: By Taylor’s theorem, for u, u (cid:48) ∈ R d , we have

Original: [Equation]
Revised: [Equation]

Original: for some ˜ u
Revised: for some ˜ u

Original: = (1 − λ ) u + λu (cid:48) with 0 ≤ λ ≤ 1 .
Revised: = (1 − λ ) u + λu (cid:48) with 0 ≤ λ ≤ 1 .

Original: By Hölder inequality, for any Hölder conjugater + 1 s = 1 ( r, s ∈ [1 , ∞ ] ), we get
Revised: By Hölder inequality, for any Hölder conjugater + 1 s = 1 ( r, s ∈ [1 , ∞ ] ), we get

Original: [Equation]
Revised: [Equation]

Original: In particular, 
Revised: In particular, |

Original: [Equation]
Revised: [Equation]

Original: Here we used the fact that q c ( u ) is a probability distribution and its sum is equal to 1 , for any choice of ˜ u .
Revised: Here we used the fact that q c ( u ) is a probability distribution and its sum is equal to 1 , for any choice of ˜ u .

Original: We substitute (18) in (17) and use the upper bound | ρ ( − f 2 ( x ; · )) − ρ ( − f 1 ( x ; · )) | ≤ (cid:107) f 1 ( x ; · ) − f 2 ( x ; · ) (cid:107) ∞ , which is just shown, to get that
Revised: We substitute (21) in (20) and use the upper bound | ρ ( − f 2 ( x [Equation] [Equation] [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: as desired.
Revised: as desired.

Revised: The following lemma relates the covering number of the function space L to the covering number of the function space G .

Original: Lemma 3.
Revised: Lemma 3.

Original: Consider the function space G and its induced function space L (7) .
Revised: Consider the function space G and its induced function space L (8) .

Original: Let Assumptions A and A3 hold.
Revised: Let Assumptions A and A3 hold.

Original: For any ε > 0 , we have
Revised: For any ε > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By choosing f ( x ; c ) =
Revised: By choosing f ( x ; c ) =

Original: d g ( x ; c ) = (cid:107) g( x ) − µ c
Revised: d g ( x ; c ) = (cid:107) g

Original: [Equation] g ) [Equation] 22 and using both parts of Proposition 2, we get that
Revised: ( x ) − µ c ( g ) (cid:107) 22 and using both parts of Proposition 2, we get that

Original: [Equation]
Revised: [Equation]

Original: If we have an ε -cover of G w.r.t.
Revised: If we have an ε -cover of G w.r.t.

Original: (cid:107) g (cid:107) ∞ , 2 = sup x ∈X (cid:107) g
Revised: (cid:107) g (cid:107) ∞ , 2 = sup x ∈X [Equation] g

Original: ( x ) (cid:107) 2 , it induces an 8 L √ dε -cover on w.r.t.
Revised: [Equation] x ) [Equation] 2 , it induces an 8 L √ dε -cover on w.r.t.

Original: the supremum norm.
Revised: the supremum norm.

Original: The following proposition is used to provide an upper bound on the magnitude of l ( x ; f ) .
Revised: The following proposition is used to provide an upper bound on the magnitude of l ( x ; f ) .

Original: Proposition 4.
Revised: Proposition 4.

Original: Suppose that Assumption A3 hold and | f
Revised: Suppose that Assumption A3 hold and | f

Original: ( x ; c ) | ≤ L (cid:48) for any x ∈ X and c ∈ { 1 , . .
Revised: ( x ; c ) | ≤ L (cid:48) for any x ∈ X and c ∈ { 1 , . .

Original: . , k } .
Revised: . , k } .

Revised: Consider p f deﬁned in (15) .

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For simplicity, we ignore the dependence on x .
Revised: For simplicity, we ignore the dependence on x .

Original: We use the deﬁnition of p f (12) to getk
Revised: We use the deﬁnition of p f (15) to get

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let us consider each term on the RHS.
Revised: Let us consider each term on the RHS.

Original: • As log φ ( c ) ≤ 0 , we have (cid:80) c φ ( c ) log φ ( c ) ≤ 0 .
Revised: • As log φ ( c ) ≤ 0 , we have (cid:80) c φ ( c ) log φ ( c ) ≤ 0 .

Original: • The priors ( π 1 , . . .
Revised: • The priors ( π 1 , . . .

Original: , π k ) indeed deﬁnes a probability distribution, as each π c is nonnegative and (cid:80)
Revised: , π k ) indeed deﬁnes a probability distribution, as each π c is nonnegative and (cid:80)

Original: c
Revised: c

Original: π c = (cid:80) c (cid:82) φ c ( x )d ν ( x ) = (cid:82) (cid:80)
Revised: π c = (cid:80) c (cid:82) φ c ( x )d ν ( x ) = (cid:82) (cid:80)

Original: c
Revised: c

Original: φ c ( x )d ν ( x ) =
Revised: φ c ( x )d ν ( x ) =

Original: (cid:82) 1 × d ν ( x ) = 1 .
Revised: (cid:82) 1 × d ν ( x ) = 1 .

Original: So − (cid:80) c
Revised: So − (cid:80) c

Original: φ ( c ) log π c is the entropy of a probability distribution over an alphabet with size k , which is at most log 2 k .
Revised: φ ( c ) log π c is the entropy of a probability distribution over an alphabet with size k , which is at most log 2 k .

Original: • The summation (cid:80) c
Revised: • The summation (cid:80) c

Original: φ ( c ) f ( c ) is upper bounded by L (cid:48) because of the boundedness of f ( c ) ≤ L (cid:48) and the fact that φ is a probability distribution and sums to one.
Revised: φ ( c ) f ( c ) is upper bounded by L (cid:48) because of the boundedness of f ( c ) ≤ L (cid:48) and the fact that φ is a probability distribution and sums to one.

Original: • Consider the term (cid:80) c φ ( c ) log ( (cid:80)
Revised: • Consider the term (cid:80) c φ ( c ) log ( (cid:80)

Original: b π b exp( − f ( b ))) .
Revised: b π b exp( − f ( b ))) .

Original: By the boundedness of f ( b ) , we have (cid:80) b π b exp( − f ( b )) ≤ (cid:80) b π b exp(+ L (cid:48) )
Revised: By the boundedness of f ( b ) , we have (cid:80) b π b exp( − f ( b )) ≤ (cid:80) b π b exp(+ L (cid:48) )

Original: ≤ exp( L (cid:48) ) .
Revised: ≤ exp( L (cid:48) ) .

Original: Therefore, the term is upper bounded by (cid:80) c
Revised: Therefore, the term is upper bounded by (cid:80) c

Original: φ ( c ) log(exp
Revised: φ ( c ) log(exp

Original: ( L (cid:48)))
Revised: ( L (cid:48)

Original:  ≤ L (cid:48) .
Revised: )) ≤ L (cid:48) .

Original: Collecting all these terms leads to the upper bound of 2 L (cid:48)
Revised: Collecting all these terms leads to the upper bound of 2 L (cid:48)

Original: + log 2 k .
Revised: + log 2 k .

Original: B A UXILIARY R ESULTS
Revised: A. C ONTROLLING THE D ISTORTED L OSS

Revised: We provide an upper bound on sup g ∈G (cid:12) (cid:12) (cid:12) ∆ n ( g ) − ˆ∆ n ( g ) (cid:12) (cid:12) (cid:12) . Proposition 5. Suppose that Assumptions A1, A2, and A3 hold.

Revised: For any δ > 0 , there exists a constant c 1 > 0 such that with probability at least 1 − δ , we have

Revised: [Equation]

Revised: Proof.

Revised: Recall that d g ( x ; c ) =

Revised: (cid:107) g

Revised: ( x ) − µ c ( g ) (cid:107) 22 , ∆ n ( g ) =

Revised: ∆ n ( ψ g , φ ) , and ˆ∆ n ( g ) =

Revised: ∆ n ( ˆ ψ g , φ ) .

Revised: Let us we deﬁne ˆ d g ( x ; c )

Revised: = (cid:107) g

Revised: ( x ) − ˆ µ c ( g ) (cid:107) 22 .

Revised: Recall the deﬁnition of l ( x, f ) (16) and that we can write ∆ n ( ψ g , φ )

Revised: = 1 n (cid:80) ni

Revised: =1 l ( X i ; d g ) .

Revised: Similarly we have ˆ∆ n ( g ) =

Revised: ∆ n ( ˆ ψ g , φ ) = 1 n (cid:80) ni =1 l ( X i ; ˆ d g ) .

Revised: Therefore, for any g ∈ G and by the application of Proposition 2 (Part II), we have

Revised: [Equation]

Revised: where the supremum norm is taken over the centres c = { 1 , . .

Revised: . , k } .

Revised: For any x ∈ X and any c = { 1 , . . .

Revised: , k } , we have

Revised: [Equation]

Revised: It was shown in (19) that (cid:107) µ c ( x ) (cid:107) 2 is upper bounded by sup x ∈X (cid:107) g

Revised: ( x ) (cid:107) 2 .

Revised: One may similarly show the same for (cid:107) ˆ µ c ( x ) (cid:107) 2 :

Revised: [Equation]

Revised: As g ( x ) ∈ [ − L, + L ] d , we have that (cid:107) g

Revised: ( x ) (cid:107) 2 , (cid:107) µ c ( x ) (cid:107) 2 , and (cid:107) ˆ µ c ( x ) (cid:107) 2 are all L √ d

Revised: -bounded, so

Revised: [Equation]

Revised: This along (22) show that

Revised: [Equation]

Revised: Proposition 6, which we prove soon, upper bounds sup g ∈G (cid:107) µ c ( g ) − ˆ µ c ( g ) (cid:107) 2 .

Revised: By a union bound argument over c ∈ { 1 , . . . , k } , we get that for any ﬁxed δ > 0 , there exists a constant c 1 > 0 such that

Revised: [Equation]

Revised: with probability at least 1 − δ .

Revised: Proposition 6.

Revised: Suppose that Assumptions A1,

Revised: A

Revised: [Equation]

Revised: and A3 hold.

Revised: Consider a ﬁxed c ∈ {

Revised: [Equation]

Revised: . , k } .

Revised: For any δ >

Revised: [Equation]

Revised: there exists a constant c 1 > 0 such that with probability at least 1 − δ , we have

Revised: [Equation]

Revised: Proof.

Revised: To shorten the formula, we use the notation E n [ f ( X i )]

Revised: = 1 n (cid:80) ni

Revised: =1 f ( X i ) , for a function f : X → R .

Revised: We can decompose µ c ( g ) − ˆ µ ( g ) as follows

Revised: [Equation]

Revised: In the rest of the proof, we provide an upper bound for Term (I) and Term (II).

Revised: Term (I): Fix δ 1 > 0 .

Revised: We denote 1 d as the d -dimensional vector with all components equal to 1 .

Revised: As g ( x ) ∈ [ − L, + L ] d and φ c ( x ) ≥ 0 , we have

Revised: [Equation]

Revised: where the comparison is dimension-wise.

Revised: Therefore,

Revised: [Equation]

Revised: We provide a probabilistic upper bound on | E n [ φ c ( X )] − E [ φ c ( X )] | .

Revised: Since φ c is a ﬁxed 1 -bounded function, we use Hoeffding’s inequality to upper bound it.

Revised: After some manipulations, we obtain that

Revised: [Equation]

Revised: with probability at least 1 − δ 1 .

Revised: We use this along (24) and the assumption that E [ φ c ( X )] ≥ π min to get

Revised: [Equation]

Revised: with probability at least 1 − δ 1 .

Revised: Term (II): We want to upper bound the supremum of the norm of second term (II).

Revised: Fix δ 2 > 0 .

Revised: To simplify the notation, let us ﬁrst deﬁne functions f ( x ) =

Revised: φ c ( x ) g ( x ) based on g .

Revised: Functions f are mapping from X to R d .

Revised: Furthermore, we deﬁne the function space F = {

Revised: x (cid:55)→ φ c ( x ) g ( x ) : g ∈ G } .

Revised: For each dimension s ∈ { 1 , . . .

Revised: , d } , we also deﬁne F s = (cid:8) x

Revised: (cid:55)→ φ c ( x ) g ( s ) ( x ) : g ∈ G (cid:9) , where g s is the s -th dimension of g .

Revised: With this notation, we can write

Revised: [Equation]

Revised: Let us focus on the supremum over F term.

Revised: We have

Revised: [Equation]

Revised: We take the square root of both sides and use the fact that (cid:112) a 21 + . . .

Revised: + a 2 d ≤ a 1 + . . .

Revised: + a d (for non-negative a i ) to get

Revised: [Equation]

Revised: Notice that as φ c ( x ) is bounded by 1 , each f s ( x ) is also L -bounded.

Revised: We use Lemma 7 on each term to obtain a high probability upper bound.

Revised: We choose the parameters of that lemma as β = 1 , B

Revised: = L , r =

Revised: L 2 , and δ =

Revised: δ 2 /d .

Revised: This leads to

Revised: [Equation]

Revised: with probability at least 1 − δ 2 .

Revised: In order to control E [ R n ( F s )] , we relate it to the covering number of F s .

Revised: We see that the covering number of F s can be upper bounded by the covering number of F , which in turn can be upper bounded by the covering number of G .

Revised: First, we use Dudley’s integral to get

Revised: [Equation]

Revised: The covering number argument is as follows.

Revised: Choose any functions f s , f (cid:48) s ∈ F s .

Revised: For any sequence x 1: n , the empirical (cid:96) 2 -norm is

Revised: [Equation]

Revised: where in the last step we used the fact that φ c ( x ) ≤ 1 .

Revised: If we have an ε -cover of G w.r.t.

Revised: (cid:107) g (cid:107) ∞ , 2 = sup x ∈X (cid:107) g

Revised: ( x ) (cid:107) 2 , it induces an ε -cover on F

Revised: [Equation]

Revised: Therefore, by (27) and setting diam ( F s ) =

Revised: 2 L

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: where we used the deﬁnition of the entropy integral of G (9).

Revised: After plug-in this upper bound in (26) and use the upper bound (25) obtained for Term (I)

Revised: [Equation]

Revised: we see that

Revised: [Equation]

Revised: with probability at least 1 − ( δ 1 + δ 2 ) .

Revised: We set δ 1 = δ 2 = δ/ 2 and simplify the upper bound to obtain the desired result.

Revised: B A UXILIARY R ESULTS

Original: For the convenience of the reader, we collect some auxiliary deﬁnitions and results that are used in the our proofs.
Revised: For the convenience of the reader, we collect some auxiliary deﬁnitions and results that are used in the our proofs.

Original: These are proven elsewhere.
Revised: These are proven elsewhere.

Original: B. F UNCTION S PACE AND N ORMS
Revised: B. F UNCTION S PACE AND N ORMS

Original: Here we brieﬂy deﬁne some of the notations that we use throughout the paper.
Revised: Here we brieﬂy deﬁne some of the notations that we use throughout the paper.

Original: Consider the domain
Revised: Consider the domain

Original: X and a function space F : X → R .
Revised: X and a function space F : X → R .

Original: We do not deal with measure theoretic considerations, so we just assume that all functions involved are measurable w.r.t.
Revised: We do not deal with measure theoretic considerations, so we just assume that all functions involved are measurable w.r.t.

Original: an appropriate σ -algebra for that space.
Revised: an appropriate σ -algebra for that space.

Original: We use M ( X ) to refer to the set of all probability distributions deﬁned over that space.
Revised: We use M ( X ) to refer to the set of all probability distributions deﬁned over that space.

Original: We use symbols such as ν ∈ M ( X ) to refer to probability distributions deﬁned over that space.
Revised: We use symbols such as ν ∈ M ( X ) to refer to probability distributions deﬁned over that space.

Original: We use (cid:107) f (cid:107) p,ν to denote the L p ( ν ) -norm (1 ≤ p < ∞ ) of a measurable function f : X → R , i.e., 
Revised: We use (cid:107) f (cid:107) p,ν to denote the L p ( ν ) -norm (1 ≤ p < ∞ ) of a measurable function f : X → R , i.e., (cid:107) f (cid:107) pp,ν (cid:44) (cid:90) X | f ( x ) | p d ν ( x

Original: [Equation]
Revised: [Equation]

Original: The supremum norm is deﬁned as usual:
Revised: The supremum norm is deﬁned as usual:

Original: [Equation]
Revised: [Equation]

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x n be a sequence of points in X .
Revised: , x n be a sequence of points in X .

Original: We use x 1: n to refer to this sequence.
Revised: We use x 1: n to refer to this sequence.

Original: The empirical measure P n is the probability measure that puts a mass of 1 n at each x
Revised: The empirical measure P n is the probability measure that puts a mass of 1 n at each x

Original: i , i.e.,
Revised: i , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where δ x is the Dirac’s delta function.
Revised: where δ x is the Dirac’s delta function.

Original: For D n = z 1: x , the empirical L 2 ( P n ) -norm of function f : X → R is
Revised: For D n = z 1: x , the empirical L 2 ( P n ) -norm of function f : X → R is

Original: [Equation]
Revised: [Equation]

Original: We can also deﬁne other L p ( P n ) -norms similarly.
Revised: We can also deﬁne other L p ( P n ) -norms similarly.

Original: 8 When there is no chance of confusion about D n , we may denote the empirical norm simply by (cid:107) f (cid:107) n .
Revised: 8 When there is no chance of confusion about D n , we may denote the empirical norm simply by (cid:107) f (cid:107) n .

Original: B. T HE M ETRIC E NTROPY AND THE C OVERING N UMBER
Revised: B. T HE M ETRIC E NTROPY AND THE C OVERING N UMBER

Original: We quote the deﬁnition of the covering number from Györﬁ et al.
Revised: We quote the deﬁnition of the covering number from Györﬁ et al.

Original: Deﬁnition 1 (Deﬁnition 9.3 of Györﬁ et al. 2002) .
Revised: Deﬁnition 1 (Deﬁnition 9.3 of Györﬁ et al. 2002) .

Original: Let ε > 0 , F be a set of real-valued functions deﬁned on X , and
Revised: Let ε > 0 , F be a set of real-valued functions deﬁned on X , and

Original: ν X be a probability measure on X .
Revised: ν X be a probability measure on X .

Original: Every ﬁnite collection of N ε = { f 1 , . . .
Revised: Every ﬁnite collection of N ε = { f 1 , . . .

Original: , f N ε } deﬁned on X with the property that for every f ∈ F , there is a function f (cid:48) ∈
Revised: , f N ε } deﬁned on X with the property that for every f ∈ F , there is a function f (cid:48) ∈

Original: N ε such that (cid:107) f − f (cid:48) (cid:107) p,ν
Revised: N ε such that (cid:107) f − f (cid:48) (cid:107) p,ν

Original: X < ε is called an ε -cover of F w.r.t.
Revised: X < ε is called an ε -cover of F w.r.t.

Original: (cid:107)·(cid:107) p,ν X .
Revised: (cid:107)·(cid:107) p,ν X .

Original: Let N ( ε, F , (cid:107)·(cid:107) p,ν X ) be the size of the smallest ε -cover of F w.r.t.
Revised: Let N ( ε, F , (cid:107)·(cid:107) p,ν X ) be the size of the smallest ε -cover of F w.r.t.

Original: (cid:107)·(cid:107) p,ν X .
Revised: (cid:107)·(cid:107) p,ν X .

Original: If no ﬁnite ε -cover exists, take N ( ε, F , (cid:107)·(cid:107) p,ν X )
Revised: If no ﬁnite ε -cover exists, take N ( ε, F , (cid:107)·(cid:107) p,ν X )

Original: = ∞ .
Revised: = ∞ .

Original: Then N ( ε, F , (cid:107)·(cid:107) p,ν X ) is called an ε -covering number of F and log N ( ε, F , (cid:107)·(cid:107) p,ν X ) is called the metric entropy of F w.r.t.
Revised: Then N ( ε, F , (cid:107)·(cid:107) p,ν X ) is called an ε -covering number of F and log N ( ε, F , (cid:107)·(cid:107) p,ν X ) is called the metric entropy of F w.r.t.

Original: the same norm.
Revised: the same norm.

Original: Given a x 1: n = ( x 1 , . . .
Revised: Given a x 1: n = ( x 1 , . . .

Original: , x n ) ⊂ X and its corresponding empirical measure P n = P x 1: n , we can deﬁne the empirical covering number of F w.r.t.
Revised: , x n ) ⊂ X and its corresponding empirical measure P n = P x 1: n , we can deﬁne the empirical covering number of F w.r.t.

Original: the empirical norm (cid:107)·(cid:107) p,x 1: n and is denoted by
Revised: the empirical norm (cid:107)·(cid:107) p,x 1: n and is denoted by

Original: N p ( ε, F , x 1: n ) =
Revised: N p ( ε, F , x 1: n ) =

Original: N ( ε, F , L p ( x 1: n )) .
Revised: N ( ε, F , L p ( x 1: n )) .

Original: B. R ADEMACHER C OMPLEXITY
Revised: B. R ADEMACHER C OMPLEXITY

Original: We deﬁne Rademacher complexity and quote a result from Bartlett & Mendelson (2002).
Revised: We deﬁne Rademacher complexity and quote a result from Bartlett & Mendelson (2002).

Original: For more information about Rademacher complexity, we refer the reader to Bartlettet al. ; Bartlett & Mendelson (2002).
Revised: For more information about Rademacher complexity, we refer the reader to Bartlett & Mendelson (2002); Bartlett et al.

Original: Let σ 1 , . . .
Revised: Let σ 1 , . . .

Original: , σ n be independent random variables with P { σ i = 1 }
Revised: , σ n be independent random variables with P { σ i = 1 }

Original: = P { σ i = − 1 } = 1 / 2 .
Revised: = P { σ i = − 1 } = 1 / 2 .

Original: For a function space F : X → R , deﬁne R n F = sup f ∈F 1 n (cid:80) ni
Revised: For a function space F : X → R , deﬁne R n F = sup f ∈F 1 n (cid:80) ni

Original: =

Revised: =

Original: σ i f ( X i ) with X i ∼ ν .
Revised: σ i f ( X i ) with X i ∼ ν .

Original: The
Revised: The

Original: Rademacher complexity (or average) of F is E
Revised: Rademacher complexity (or average) of F is E

Original: [ R n G ] , in which the expectation is w.r.t.
Revised: [ R n G ] , in which the expectation is w.r.t.

Original: both σ and
Revised: both σ and

Original: X i .
Revised: X i .

Original: Rademacher complexity appears in the analysis of the supremum of an empirical process right after the application of the symmetrization technique.
Revised: Rademacher complexity appears in the analysis of the supremum of an empirical process right after the application of the symmetrization technique.

Original: As such, its behaviour is closely related to the behaviour of the empirical process.
Revised: As such, its behaviour is closely related to the behaviour of the empirical process.

Original: One may interpret the Rademacher complexity as a complexity measure that quantiﬁes the extent that a function from F can ﬁt a noise sequence of length n (Bartlett & Mendelson, 2002).
Revised: One may interpret the Rademacher complexity as a complexity measure that quantiﬁes the extent that a function from F can ﬁt a noise sequence of length n (Bartlett & Mendelson, 2002).

Original: The following is a simpliﬁed (and slightly reworded) version of Theorem 2.1 of Bartlett et al.
Revised: The following is a simpliﬁed (and slightly reworded) version of Theorem 2.1 of Bartlett et al.

Original: Lemma 5.
Revised: Lemma 7.

Original: Let F : X → R be a measurable function space with B -bounded functions.
Revised: Let F : X → R be a measurable function space with B -bounded functions.

Original: Let
Revised: Let

Original: X 1 , . . .
Revised: X 1 , . . .

Original: , X n ∈ X be independent random variables.
Revised: , X n ∈ X be independent random variables.

Original: Assume that for some r > 0 , Var [ f ( X i )] ≤ r for every f ∈ F .
Revised: Assume that for some r > 0 , Var [ f ( X i )] ≤ r for every f ∈ F .

Original: Then for every δ > 0 , with probability at least 1 − δ ,
Revised: Then for every δ > 0 , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: C E XPERIMENTAL RESULTS
Revised: C E XPERIMENTAL RESULTS

Original: We give implementation details about the experiments of our submitted paper in Section C.1.
Revised: We give implementation details about the experiments of our submitted paper in Section C.1.

Original: We report the detailed performance of our model as a function of the output dimensionality and the number of hidden layers in the zero-shot learning context in Section C.2.
Revised: We report the detailed performance of our model as a function of the output dimensionality and the number of hidden layers in the zero-shot learning context in Section C.2.

Original: We show in Section C. that our method can be generalized to hard clustering by using implicit centers.
Revised: We show in Section C. that our method can be generalized to hard clustering by using implicit centers.

Original: We give additional visualization results in Section C.4.
Revised: We give additional visualization results in Section C.4.

Original: C. I MPLEMENTATION DETAILS
Revised: C. I MPLEMENTATION DETAILS

Original: We coded our method in PyTorch and ran all our experiments on a single Nvidia GeForce GTXwhich has 6GB of RAM.
Revised: We coded our method in PyTorch and ran all our experiments on a single Nvidia GeForce GTXwhich has 6GB of RAM.

Original: PyTorch automatically calculates the gradient w.r.t.
Revised: PyTorch automatically calculates the gradient w.r.t.

Original: the mini-batch representation F .
Revised: the mini-batch representation F .

Original: Nonetheless, it is worth mentioning that both the ﬁrst and second arguments of our prediction function ψ ( F, M, π ) depend on F in the case where the centers are implicit ( i.e. , when we write M = diag( Y (cid:62) 1 n ) − 1 Y (cid:62) F ).
Revised: Nonetheless, it is worth mentioning that both the ﬁrst and second arguments of our prediction function ψ ( F, M, π ) depend on F in the case where the centers are implicit ( i.e. , when we write M = diag( Y (cid:62) 1 n ) − 1 Y (cid:62) F ).

Original: In this case, the gradient of our loss function w.r.t.
Revised: In this case, the gradient of our loss function w.r.t.

Original: F depends on both the ﬁrst and second arguments of ψ .
Revised: F depends on both the ﬁrst and second arguments of ψ .

Original: C.1. Z ERO - SHOT LEARNING EXPERIMENTS
Revised: C.1. Z ERO - SHOT LEARNING EXPERIMENTS

Original: We now give details speciﬁc to the zero-shot experiments.
Revised: We now give details speciﬁc to the zero-shot experiments.

Original: In the zero-shot learning experiment where F and M are computed from different souces ( i.e. , images and text) and are the output of two different networks, the optimization is performed by alternately optimizing one variable while ﬁxing the other.
Revised: In the zero-shot learning experiment where F and M are computed from different souces ( i.e. , images and text) and are the output of two different networks, the optimization is performed by alternately optimizing one variable while ﬁxing the other.

Original: Mini-batch size: The training datasets of CUB and Flowers contain 5894 and 5878 images, respectively.
Revised: Mini-batch size: The training datasets of CUB and Flowers contain 5894 and 5878 images, respectively.

Original: In order to ﬁt into memory, we set our mini-batch sizes as 421 ( = 5894 / 14 ) and( ≈ 5878 / 8 ) for CUB and Flowers, respectively.
Revised: In order to ﬁt into memory, we set our mini-batch sizes as 421 ( = 5894 / 14 ) and( ≈ 5878 / 8 ) for CUB and Flowers, respectively.

Original: Reed et al.
Revised: Reed et al.

Original: and Snell et al.
Revised: and Snell et al.

Original: (2017) use 10 different views per image during training (middle, upper left, upper right, lower left and lower right crops for the original and horizontally-ﬂipped image), and 1 view for test which is the middle crop of the original image.
Revised: (2017) use 10 different views per image during training (middle, upper left, upper right, lower left and lower right crops for the original and horizontally-ﬂipped image), and 1 view for test which is the middle crop of the original image.

Original: On the other hand, we use only 1 view per image ( i.e. , the middle crop of the original image) during training and test.
Revised: On the other hand, we use only 1 view per image ( i.e. , the middle crop of the original image) during training and test.

Original: Optimizer: We use the Adam optimizer with a learning rate of 10 − 5 to train both models ϕ θ 1 and ϕ θ 2 .
Revised: Optimizer: We use the Adam optimizer with a learning rate of 10 − 5 to train both models ϕ θ 1 and ϕ θ 2 .

Original: Target soft assignment matrix: When we use the representations provided by Snell et al.
Revised: Target soft assignment matrix: When we use the representations provided by Snell et al.

Original: (2017), our target soft assignment matrix is Y 1 = ψ ( ˜ F, ˜ M, ˜ π ) =
Revised: (2017), our target soft assignment matrix is Y 1 = ψ ( ˜ F, ˜ M, ˜ π ) =

Original: ˜Ψ ∈ Y n × k where the matrices ˜ F and ˜ M are provided and ˜ π
Revised: ˜Ψ ∈ Y n × k where the matrices ˜ F and ˜ M are provided and ˜ π

Original: = 1 k 1 k .
Revised: = 1 k 1 k .

Original: The elements of ˜Ψ are written ˜Ψ
Revised: The elements of ˜Ψ are written ˜Ψ

Original: ic =
Revised: ic =

Original: [Equation]

Original: , ˜

Original: [Equation]

Original: ke

Original: [Equation]

Original: [Equation]

Original: ˜

Original: [Equation]

Original: ˜ µ

Original: [Equation]
Revised: [Equation]

Original: In this case, we formulate:
Revised: In this case, we formulate:

Original: [Equation]
Revised: [Equation]

Original: Using a temperature of 10 made the optimization more stable as it avoided gradients with high values.
Revised: Using a temperature of 10 made the optimization more stable as it avoided gradients with high values.

Original: We use a 
Revised: We use a temperature of 2 when using the representations provided by Reed et al.

Original: Initial temperature of our model: To make our optimization framework stable, we start with a temperature of 50.
Revised: Initial temperature of our model: To make our optimization framework stable, we start with a temperature of 50.

Original: We then formulate our Bregman divergence as:
Revised: We then formulate our Bregman divergence as:

Original: [Equation]
Revised: [Equation]

Original: where f i and µ c are the representations learned by our model.
Revised: where f i and µ c are the representations learned by our model.

Original: We decrease our temperature by 10% ( i.e. , temp t +1 = 0 .
Revised: We decrease our temperature by 10% ( i.e. , temp t +1 = 0 .

Original: 9 temp t ) every 3000 epochs until the algorithm stops training.
Revised: 9 temp t ) every 3000 epochs until the algorithm stops training.

Original: We stop training at 10k epochs on CUB and 1k epochs on Flowers.
Revised: We stop training at 10k epochs on CUB and 1k epochs on Flowers.

Original: C.1. V ISUALIZATION EXPERIMENTS
Revised: C.1. V ISUALIZATION EXPERIMENTS

Original: We now give details speciﬁc to the visualization experiments.
Revised: We now give details speciﬁc to the visualization experiments.

Original:  Dataset size: To be comparable to t-SNE, we directly learn two-dimensional embeddings instead of neural networks.
Revised: [Table] [Table] [Table] Dataset size: To be comparable to t-SNE, we directly learn two-dimensional embeddings instead of neural networks.

Original: Our mini-batch size is the size of the test set ( i.e. , the number of examples is n = 10 4 for most datasets except STL that contains n = 8000 test examples).
Revised: Our mini-batch size is the size of the test set ( i.e. , the number of examples is n = 10 4 for most datasets except STL that contains n = 8000 test examples).

Original: Optimizer: We use the RMSprop optimizer with a learning rate of 10 − 3 , α = 0 .
Revised: Optimizer: We use the RMSprop optimizer with a learning rate of 10 − 3 , α = 0 .

Original: 99 , (cid:15) =
Revised: 99 , (cid:15) =

Original: 10 − 6 , the weight decay and momentum are both 0, and the data is not centered.
Revised: 10 − 6 , the weight decay and momentum are both 0, and the data is not centered.

Original: Target soft assignment matrix: Let us note z i =
Revised: Target soft assignment matrix: Let us note z i =

Original: [ z i, 1 , · · · , z i,k ] (cid:62)
Revised: [ z i, 1 , · · · , z i,k ] (cid:62)

Original: ∈ R k the vector containing the logits of the learned representation of the i -th test example.
Revised: ∈ R k the vector containing the logits of the learned representation of the i -th test example.

Original: We formulate y i
Revised: We formulate y i

Original: =
Revised: =

Original: [ y i, 1 , · · · , y i,k ] (cid:62) ∈ R k our target assignment vector for the i -th test example as follows:
Revised: [ y i, 1 , · · · , y i,k ] (cid:62) ∈ R k our target assignment vector for the i -th test example as follows:

Original: [Equation]
Revised: [Equation]

Original: where τ = 5 .
Revised: where τ = 5 .

Original: Initial temperature of our model: We learned our representation by using a ﬁxed temperature of( i.e. , using the standard squared Euclidean distance).
Revised: Initial temperature of our model: We learned our representation by using a ﬁxed temperature of( i.e. , using the standard squared Euclidean distance).

Original: We stop the algorithm after 8000 iterations.
Revised: We stop the algorithm after 8000 iterations.

Original: Tuning t-SNE: we tested different ranges of scaling (1/1, 1/10, 1/100) and perplexity ( i.e. , 1, 10,(default) and 100) and reported the representations that obtained the best quantitative resutls.
Revised: Tuning t-SNE: we tested different ranges of scaling (1/1, 1/10, 1/100) and perplexity ( i.e. , 1, 10,(default) and 100) and reported the representations that obtained the best quantitative resutls.

Original: C. I MPACT OF DIMENSIONALITY IN ZERO - SHOT LEARNING
Revised: C. I MPACT OF DIMENSIONALITY IN ZERO - SHOT LEARNING

Original: Let e ∈ N be the dimensionality of the representations taken as input and d the output dimensionality of the models ϕ θ 1 and ϕ θ 2 , the architecture of the models is e - d and e - e - d in the 1 and 2 hidden layer cases, respectively.
Revised: Let e ∈ N be the dimensionality of the representations taken as input and d the output dimensionality of the models ϕ θ 1 and ϕ θ 2 , the architecture of the models is e - d and e - e - d in the 1 and 2 hidden layer cases, respectively.

Original: The hyperparameter d ∈ { 16 , 32 , 64 , · · · , e } is also a hyperparameter cross-validated on the validation set.
Revised: The hyperparameter d ∈ { 16 , 32 , 64 , · · · , e } is also a hyperparameter cross-validated on the validation set.

Original: We give the detailed accuracy performance of our model in 
Revised: We give the detailed accuracy performance of our model in Tables 5 to 9.

Original: Tables
Revised: [Table]

Original: 5 to • Table 5 reports the test performance of our model on CUB when using the features provided by Reed et al.
Revised: [Table] • Table 5 reports the test performance of our model on CUB when using the features provided by Reed et al.

Original: (2016) as supervision for different numbers of hidden layers and values of outputdimensionality of our model.
Revised: (2016) as supervision for different numbers of hidden layers and values of output of our model

Original: • Table 6 (resp.
Revised: • Table 6 (resp.

Original: Table 7) reports the validation (resp.
Revised: Table 7) reports the validation (resp.

Original: test) performance of our model on CUB when using the features provided by Snell et al.
Revised: test) performance of our model on CUB when using the features provided by Snell et al.

Original: (2017) as supervision.
Revised: (2017) as supervision.

Original: Table 8 (resp.
Revised: Table 8 (resp.

Original: Table 9) reports the validation (resp.
Revised: Table 9) reports the validation (resp.

Original: test) performance of our model on Flowers when using the features provided by Snell et al.
Revised: test) performance of our model on Flowers when using the features provided by Snell et al.

Original: (2017) as supervision.
Revised: (2017) as supervision.

Original: Dimensionality reduction improves performance, though the optimal dimensionality is dataset speciﬁc.
Revised: Dimensionality reduction improves performance, though the optimal dimensionality is dataset speciﬁc.

Original: In general, increasing the number of hidden layers also helps.
Revised: In general, increasing the number of hidden layers also helps.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: C. G ENERALIZATION TO HARD CLUSTERING
Revised: C. G ENERALIZATION TO HARD CLUSTERING

Original: We validate that DRPR can be used to perform hard clustering as in Snell et al.
Revised: We validate that DRPR can be used to perform hard clustering as in Snell et al.

Original: but with implicit centers.
Revised: but with implicit centers.

Original: To this end [Table] we train a neural network with 2 convolutional layers on MNIST (LeCun et al., 1998) followed by a fully connected layer.
Revised: To this end, we train a neural network with 2 convolutional layers on MNIST (LeCun et al., 1998) followed by a fully connected layer.

Original: Its output dimensionality is d = 2 or d = 3 , the mini-batch size is n = 1000 , the number of categories is k = 10 and the target hard assignment matrix Y ∈ { 0 , 1 } n × k contains category membership information ( i [Table] e. , Y ic is 1 if the example f i belongs to category c , 0 otherwise).
Revised: Its output dimensionality is d = 2 or d = 3 , the mini-batch size is n = 1000 , the number of categories is k = 10 and the target hard assignment matrix Y ∈ { 0 , 1 } n × k contains category membership information ( i.e. , Y ic is 1 if the example f i belongs to category c , 0 otherwise).

Original: We train the model on the training set of MNIST and plot in
Revised: We train the model on the training set of MNIST and plot in

Original: Fig.
Revised: Fig.

Original: 6 the representations of the test set.
Revised: 6 the representations of the test set.

Original: By assigning each test example to the category with closest centroid (obtained from the training set), the model obtains 98% (resp.
Revised: By assigning each test example to the category with closest centroid (obtained from the training set), the model obtains 98% (resp.

Original: 99% ) accuracy when d =(resp.
Revised: 99% ) accuracy when d =(resp.

Original: d = 3 ).
Revised: d = 3 ).

Original: DRPR can then be learned for hard clustering when the centers are implicitly written as a function of the mini-batch matrix representation F and the target hard assignment matrix Y .[Figure]
Revised: DRPR can then be learned for hard clustering when the centers are implicitly written as a function of the mini-batch matrix representation F and the target hard assignment matrix Y .

Original: C. V ISUALIZATION RESULTS
Revised: C. V ISUALIZATION RESULTS

Original: We now present visualization results.
Revised: We now present visualization results.

Original: C.4. A RTIFACTS WITH T -SNE
Revised: C.4. A RTIFACTS WITH T -SNE

Original: Fig.
Revised: Fig.

Original: 5 illustrates the CIFAR 100 representation learned by t-SNE when its input data is the target probability distribution that we give as supervision/input of our algorithm.
Revised: 5 illustrates the CIFAR 100 representation learned by t-SNE when its input data is the target probability distribution that we give as supervision/input of our algorithm.

Original: Following the recommendations mentioned in https://lvdmaaten.github.io/tsne/ when the representation form a strange ball with uniformly distributed points and obtaining very low error, we decreased the perplexity from 30 (which is the value by default) to 10 and 1 and divided our data by 10, 100 and 1000.
Revised: Following the recommendations mentioned in https://lvdmaaten.github.io/tsne/ when the representation form a strange ball with uniformly distributed points and obtaining very low error, we decreased the perplexity from 30 (which is the value by default) to 10 and 1 and divided our data by 10, 100 and 1000.

Original: Nonetheless, we still obtained the same type of representation as in Fig.
Revised: Nonetheless, we still obtained the same type of representation as in Fig.

Original: This kind of artifact is the reason why we only report results obtained with logits.
Revised: This kind of artifact is the reason why we only report results obtained with logits.

Original: C.4. A DAPTING T -SNE WITH OTHER 
Revised: C.4. A DAPTING T -SNE WITH OTHER DIVERGENCES

Original: We plot in Fig.
Revised: We plot in Fig.

Original: 7 the visualization obtained by t-SNE when using the KL or JS divergences to compare pairs of probability distribution representations.
Revised: 7 the visualization obtained by t-SNE when using the KL or JS divergences to compare pairs of probability distribution representations.

Original: The representations obtained in this case are still worse than using the original 3-dimensional representations as the cluster structures are not preserved, nor the inter-cluster distances.
Revised: The representations obtained in this case are still worse than using the original 3-dimensional representations as the cluster structures are not preserved, nor the inter-cluster distances.

Original: This suggests that comparing pairs of examples, as done by t-SNE, is less appropriate than our method that considers similarities between examples and the different k = 6 clusters.
Revised: This suggests that comparing pairs of examples, as done by t-SNE, is less appropriate than our method that considers similarities between examples and the different k = 6 clusters.

Original: C.4. A DDITIONAL RESULTS
Revised: C.4. A DDITIONAL RESULTS

Original: Fig.
Revised: Fig.

Original: 8 illustrates the DRPR and t-SNE representations of CIFAR 10.
Revised: 8 illustrates the DRPR and t-SNE representations of CIFAR 10.

Original: Animal categories are illustrated on the right whereas machines are on the left.
Revised: Animal categories are illustrated on the right whereas machines are on the left.

Original: Fig.
Revised: Fig.

Original: 9 illustrates the DRPR and t-SNE representations of CIFAR 100.
Revised: 9 illustrates the DRPR and t-SNE representations of CIFAR 100.

Original: We learned the representations by exploiting 100 clusters but plot only 20 colors (one for each superclass of CIFAR 100), which is why multiple spikes have the same color.
Revised: We learned the representations by exploiting 100 clusters but plot only 20 colors (one for each superclass of CIFAR 100), which is why multiple spikes have the same color.

Original: Groups with same colors are better deﬁned with our approach than with t-SNE, this means that different categories from the same superclass ( e.g. hamster, mouse, rabbit, shrew, squirrel which are small mammals ) are grouped together with DRPR.
Revised: Groups with same colors are better deﬁned with our approach than with t-SNE, this means that different categories from the same superclass ( e.g. hamster, mouse, rabbit, shrew, squirrel which are small mammals ) are grouped together with DRPR.

Original: One can observe a semantic structure in the 2D representation of DRPR: plants and insects are on the top left; animals are on the bottom left and categories on the right are outdoor categories.
Revised: One can observe a semantic structure in the 2D representation of DRPR: plants and insects are on the top left; animals are on the bottom left and categories on the right are outdoor categories.

Original: Medium mammals are also represented between small mammals and large carnivores .
Revised: Medium mammals are also represented between small mammals and large carnivores .

Original: Figures 10, 11, 12 and 13 illustrate the representations learned by our model for the STL, MNIST,
Revised: Figures 10, 11, 12 and 13 illustrate the representations learned by our model for the STL, MNIST,

Original: CIFAR 100 and CIFAR 10 datasets, respectively.
Revised: CIFAR 100 and CIFAR 10 datasets, respectively.

Original: Instead of using colors that represent the categories of the embeddings as done in the submitted paper, we directly plot the images.
Revised: Instead of using colors that represent the categories of the embeddings as done in the submitted paper, we directly plot the images.

Original: In general, we observe that images towards the end of spikes consist of a clearly visible object in a standard viewpoint on a simple background.
Revised: In general, we observe that images towards the end of spikes consist of a clearly visible object in a standard viewpoint on a simple background.

Original: Those closer to the center often have objects with a non-standard viewpoint or have a complex textured background.
Revised: Those closer to the center often have objects with a non-standard viewpoint or have a complex textured background.

Original: At a high-level, the classes appear to be organized by their backgrounds.
Revised: At a high-level, the classes appear to be organized by their backgrounds.

Original: Taking the STL-10 visualization as an example, deers and horses are close together since they both tend to be found in the presence of green vegation.
Revised: Taking the STL-10 visualization as an example, deers and horses are close together since they both tend to be found in the presence of green vegation.

Original: These classes are far from boats and planes , which often have solid blue backgrounds.
Revised: These classes are far from boats and planes , which often have solid blue backgrounds.

Original: Looking more closely, the ordering of classes is sensible.
Revised: Looking more closely, the ordering of classes is sensible.

Original: Planes are neighbors to both boats (similar background) and birds (similar silhouette).
Revised: Planes are neighbors to both boats (similar background) and birds (similar silhouette).

Original: And trucks neighbor both cars (similar background) and horses, which appear visually similar, particularly for images in which the horse is pulling a cart.
Revised: And trucks neighbor both cars (similar background) and horses, which appear visually similar, particularly for images in which the horse is pulling a cart.

Original: Taking the MNIST visualization as another example, one can observe that written characters in spikes are easy to recognize as they correspond to examples for which the learned model has high conﬁdence in its scores.
Revised: Taking the MNIST visualization as another example, one can observe that written characters in spikes are easy to recognize as they correspond to examples for which the learned model has high conﬁdence in its scores.

Original: On the other hand, ambiguous examples are between multiple spikes ( e.g. , the characters 0 and 6 between spikes are more ambiguous than their neighbors in spikes).
Revised: On the other hand, ambiguous examples are between multiple spikes ( e.g. , the characters 0 and 6 between spikes are more ambiguous than their neighbors in spikes).