Original: E XTREME C LASSIFICATION VIA A DVERSARIAL S OFTMAX A PPROXIMATION
Revised: E XTREME C LASSIFICATION VIA A DVERSARIAL S OFTMAX A PPROXIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training a classiﬁer over a large number of classes, known as ’extreme classiﬁcation’, has become a topic of major interest with applications in technology, science, and e-commerce.
Revised: Training a classiﬁer over a large number of classes, known as ’extreme classiﬁcation’, has become a topic of major interest with applications in technology, science, and e-commerce.

Original: Traditional softmax regression induces a gradient cost proportional to the number of classes C , which often is prohibitively expensive.
Revised: Traditional softmax regression induces a gradient cost proportional to the number of classes C , which often is prohibitively expensive.

Original: A popular scalable softmax approximation relies on uniform negative sampling, which suffers from slow convergence due a poor signal-to-noise ratio.
Revised: A popular scalable softmax approximation relies on uniform negative sampling, which suffers from slow convergence due a poor signal-to-noise ratio.

Original: In this paper, we propose a simple training method for drastically enhancing the gradient signal by drawing negative samples from an adversarial model that mimicks the data distribution.
Revised: In this paper, we propose a simple training method for drastically enhancing the gradient signal by drawing negative samples from an adversarial model that mimicks the data distribution.

Original: Our contributions are three-fold: (i) an adversarial sampling mechanism that produces negative samples at a cost only logarithmic in C , thus still resulting in cheap gradient updates; (ii) a mathematical proof that this adversarial sampling minimizes the gradient variance while any bias due to non-uniform sampling can be removed; (iii) experimental results on large scale data sets that show a reduction of the training time by an order of magnitude relative to several competitive baselines.
Revised: Our contributions are three-fold: (i) an adversarial sampling mechanism that produces negative samples at a cost only logarithmic in C , thus still resulting in cheap gradient updates; (ii) a mathematical proof that this adversarial sampling minimizes the gradient variance while any bias due to non-uniform sampling can be removed; (iii) experimental results on large scale data sets that show a reduction of the training time by an order of magnitude relative to several competitive baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In many problems in science, healthcare, or e-commerce, one is interested in training classiﬁers over an enormous number of classes: a problem known as ‘extreme classiﬁcation’
Revised: In many problems in science, healthcare, or e-commerce, one is interested in training classiﬁers over an enormous number of classes: a problem known as ‘extreme classiﬁcation’

Original: (Agrawal et al., 2013; Jain et al., 2016; Prabhu & Varma, 2014; Siblini et al., 2018).
Revised: (Agrawal et al., 2013; Jain et al., 2016; Prabhu & Varma, 2014; Siblini et al., 2018).

Original: For softmax (aka multinomial) regression, each gradient step incurs a cost proportional to the number of classes C .
Revised: For softmax (aka multinomial) regression, each gradient step incurs a cost proportional to the number of classes C .

Original: As this may be prohibitively expensive for large C , recent research has explored more scalable softmax approximations which circumvent the linear scaling in C .
Revised: As this may be prohibitively expensive for large C , recent research has explored more scalable softmax approximations which circumvent the linear scaling in C .

Original: Progress in accelerating the training procedure and thereby scaling up extreme classiﬁcation promises to dramatically improve, e.g., advertising (Prabhu et al., 2018), recommender systems, ranking algorithms (Bhatia et al., 2015; Jain et al., 2016), and medical diagnostics (Bengio et al., 2019; Lippert et al., 2017; Baumel et al., 2018)
Revised: Progress in accelerating the training procedure and thereby scaling up extreme classiﬁcation promises to dramatically improve, e.g., advertising (Prabhu et al., 2018), recommender systems, ranking algorithms (Bhatia et al., 2015; Jain et al., 2016), and medical diagnostics (Bengio et al., 2019; Lippert et al., 2017; Baumel et al., 2018)

Original: While scalable softmax approximations have been proposed, each one has its drawbacks.
Revised: While scalable softmax approximations have been proposed, each one has its drawbacks.

Original: The most popular approach due to its simplicity is ‘negative sampling’
Revised: The most popular approach due to its simplicity is ‘negative sampling’

Original: (Mnih & Hinton, 2009; Mikolov et al., 2013), which turns the problem into a binary classiﬁcation between so-called ‘positive samples’ from the data set and ‘negative samples’ that are drawn at random from some (usually uniform) distribution over the class labels.
Revised: (Mnih & Hinton, 2009; Mikolov et al., 2013), which turns the problem into a binary classiﬁcation between so-called ‘positive samples’ from the data set and ‘negative samples’ that are drawn at random from some (usually uniform) distribution over the class labels.

Original: While negative sampling makes the updates cheaper since computing the gradient no longer scales with C , it induces additional gradient noise that leads to a poor signalto-noise ratio of the stochastic gradient estimate.
Revised: While negative sampling makes the updates cheaper since computing the gradient no longer scales with C , it induces additional gradient noise that leads to a poor signalto-noise ratio of the stochastic gradient estimate.

Original: Improving the signal-to-noise ratio in negative sampling while still enabling cheap gradients would dramatically enhance the speed of convergence.
Revised: Improving the signal-to-noise ratio in negative sampling while still enabling cheap gradients would dramatically enhance the speed of convergence.

Original: In this paper, we present an algorithm that inherits the cheap gradient updates from negative sampling while still preserving much of the gradient signal of the original softmax regression problem.
Revised: In this paper, we present an algorithm that inherits the cheap gradient updates from negative sampling while still preserving much of the gradient signal of the original softmax regression problem.

Original: Our approach rests on the insight that the signal-to-noise ratio in negative sampling is poor since there is no association between input features and their artiﬁcial labels.
Revised: Our approach rests on the insight that the signal-to-noise ratio in negative sampling is poor since there is no association between input features and their artiﬁcial labels.

Original: If negative samples were harder to discriminate from positive ones, a learning algorithm would obtain a better gradient signal close to the optimum.
Revised: If negative samples were harder to discriminate from positive ones, a learning algorithm would obtain a better gradient signal close to the optimum.

Original: Here, we make these arguments mathematically rigorous and propose a non-uniform sampling scheme for scalably approximating a softmax classiﬁcation scheme.
Revised: Here, we make these arguments mathematically rigorous and propose a non-uniform sampling scheme for scalably approximating a softmax classiﬁcation scheme.

Original: Instead of sampling labels uniformly, our algorithm uses an adversarial auxiliary model to draw ‘fake’ labels that are more realistic by taking the input features of the data into account.
Revised: Instead of sampling labels uniformly, our algorithm uses an adversarial auxiliary model to draw ‘fake’ labels that are more realistic by taking the input features of the data into account.

Original: We prove that such procedure reduces the gradient noise of the algorithm, and in fact minimizes the gradient variance in the limit where the auxiliary model optimally mimics the data distribution.
Revised: We prove that such procedure reduces the gradient noise of the algorithm, and in fact minimizes the gradient variance in the limit where the auxiliary model optimally mimics the data distribution.

Original: A useful adversarial model should require only little overhead to be ﬁtted to the data, and it needs to be able to generate negative samples quickly in order to enable inexpensive gradient updates.
Revised: A useful adversarial model should require only little overhead to be ﬁtted to the data, and it needs to be able to generate negative samples quickly in order to enable inexpensive gradient updates.

Original: We propose a probabilistic version of a decision tree that has these properties.
Revised: We propose a probabilistic version of a decision tree that has these properties.

Original: As a side result of our approach, we show how such an auxiliary model can be constructed and efﬁciently trained.
Revised: As a side result of our approach, we show how such an auxiliary model can be constructed and efﬁciently trained.

Original: Since it is almost hyperparameter-free, it does not cause extra complications when tuning models.
Revised: Since it is almost hyperparameter-free, it does not cause extra complications when tuning models.

Original: The ﬁnal problem that we tackle is to remove the bias that the auxiliary model causes relative to our original softmax classiﬁcation.
Revised: The ﬁnal problem that we tackle is to remove the bias that the auxiliary model causes relative to our original softmax classiﬁcation.

Original: Negative sampling is typically described as a softmax approximation; however, only uniform negative sampling correctly approximates the softmax.
Revised: Negative sampling is typically described as a softmax approximation; however, only uniform negative sampling correctly approximates the softmax.

Original: In this paper, we show that the bias due to non-uniform negative sampling can be easily removed at test time.
Revised: In this paper, we show that the bias due to non-uniform negative sampling can be easily removed at test time.

Original: The stucture of our paper reﬂects our main contributions as follows:
Revised: The stucture of our paper reﬂects our main contributions as follows:

Original: We present a new scalable softmax approximation (Section 2).
Revised: We present a new scalable softmax approximation (Section 2).

Original: We show that non-uniform sampling from an auxiliary model can improve the signal-to-noise ratio.
Revised: We show that non-uniform sampling from an auxiliary model can improve the signal-to-noise ratio.

Original: The best performance is achieved when this sampling mechanism is adversarial , i
Revised: The best performance is achieved when this sampling mechanism is adversarial , i

Original: e., when it generates fake labels that are hard to discriminate from the true ones.
Revised: e., when it generates fake labels that are hard to discriminate from the true ones.

Original: To allow for efﬁcient training, such adversarial samples need to be generated at a rate sublinear (e
Revised: To allow for efﬁcient training, such adversarial samples need to be generated at a rate sublinear (e

Original: g., logarithmic) in C . 2.
Revised: g., logarithmic) in C . 2.

Original: We design a new, simple adversarial auxiliary model that satisﬁes the above requirements (Section 3).
Revised: We design a new, simple adversarial auxiliary model that satisﬁes the above requirements (Section 3).

Original: The model is based on a probabilistic version of a decision tree.
Revised: The model is based on a probabilistic version of a decision tree.

Original: It can be efﬁciently pre-trained and included into our approach, and requires only minimal tuning.
Revised: It can be efﬁciently pre-trained and included into our approach, and requires only minimal tuning.

Original: We present mathematical proofs that (i) the best signal-to-noise ratio in the gradient is obtained if the auxiliary model best reﬂects the true dependencies between input features and labels, and that (ii) the involved bias to the softmax approximation can be exactly quantiﬁed and cheaply removed at test time (Section 4).
Revised: We present mathematical proofs that (i) the best signal-to-noise ratio in the gradient is obtained if the auxiliary model best reﬂects the true dependencies between input features and labels, and that (ii) the involved bias to the softmax approximation can be exactly quantiﬁed and cheaply removed at test time (Section 4).

Original: We present experiments onseveral two classiﬁcation data sets that show that our method outperforms all 
Revised: We present experiments on two classiﬁcation data sets that show that our method outperforms all baselines

Original: baselines by at least one order of magnitude in training speed (Section 5).
Revised: by at least one order of magnitude in training speed (Section 5).

Original: We discuss related work in Section 6 and summarize our approach in Section 7.
Revised: We discuss related work in Section 6 and summarize our approach in Section 7.

Original:  A N A DVERSARIAL S OFTMAX A PPROXIMATION
Revised:  A N A DVERSARIAL S OFTMAX A PPROXIMATION

Original: We propose an efﬁcient algorithm to train a classiﬁer over a large set of classes, using an asymptotic equivalence between softmax classiﬁcation and negative sampling (Subsection 2.1).
Revised: We propose an efﬁcient algorithm to train a classiﬁer over a large set of classes, using an asymptotic equivalence between softmax classiﬁcation and negative sampling (Subsection 2.1).

Original: To speed up convergence, we generalize this equivalence to model-based negative sampling in Subsection 2.2.
Revised: To speed up convergence, we generalize this equivalence to model-based negative sampling in Subsection 2.2.

Original:  A SYMPTOTIC E QUIVALENCE OF S OFTMAX C LASSIFICATION AND N EGATIVE S AMPLING
Revised:  A SYMPTOTIC E QUIVALENCE OF S OFTMAX C LASSIFICATION AND N EGATIVE S AMPLING

Original: Softmax Classiﬁcation (Notation).
Revised: Softmax Classiﬁcation (Notation).

Original: We consider a training data set D
Revised: We consider a training data set D

Original: = { ( x i , y i ) } i =1: N of N data points with K -dimensional feature vectors x i ∈ R K .
Revised: = { ( x i , y i ) } i =1: N of N data points with K -dimensional feature vectors x i ∈ R K .

Original: Each data point has a single label y i ∈ Y from a discrete label set Y .
Revised: Each data point has a single label y i ∈ Y from a discrete label set Y .

Original: A softmax classiﬁer is deﬁned by a set of functions { ξ y } y ∈Y that map a feature vector x and model parameters θ to a score ξ y ( x, θ ) ∈ R for each label y .
Revised: A softmax classiﬁer is deﬁned by a set of functions { ξ y } y ∈Y that map a feature vector x and model parameters θ to a score ξ y ( x, θ ) ∈ R for each label y .

Original: Its loss function is
Revised: Its loss function is

Original: [Equation]
Revised: [Equation]

Original: While the ﬁrst term encourages high scores ξ y ( x, θ ) for the correct labels y , the second term encourages low scores for all labels y (cid:48) ∈ Y , thus preventing degenerate solutions that set all scores to inﬁnity.
Revised: While the ﬁrst term encourages high scores ξ y ( x, θ ) for the correct labels y , the second term encourages low scores for all labels y (cid:48) ∈ Y , thus preventing degenerate solutions that set all scores to inﬁnity.

Original: Unfortunately, the sum over y (cid:48)
Revised: Unfortunately, the sum over y (cid:48)

Original: ∈ Y makes gradient based minimization of (cid:96) softmax ( θ ) expensive if the label set Y is large.
Revised: ∈ Y makes gradient based minimization of (cid:96) softmax ( θ ) expensive if the label set Y is large.

Original: Assuming that evaluating a single score ξ y (cid:48) ( x, θ ) takes O ( K ) time, each gradient step costs O ( KC ) , where C = |Y| is the size of the label set.
Revised: Assuming that evaluating a single score ξ y (cid:48) ( x, θ ) takes O ( K ) time, each gradient step costs O ( KC ) , where C = |Y| is the size of the label set.

Original: Negative Sampling.
Revised: Negative Sampling.

Original: Negative sampling turns classiﬁcation over a large label set Y into binary classiﬁcation between so-called positive and negative samples.
Revised: Negative sampling turns classiﬁcation over a large label set Y into binary classiﬁcation between so-called positive and negative samples.

Original: One draws positive samples ( x, y ) from the training set and constructs negative samples ( x, y (cid:48) ) by drawing random labels y (cid:48) from some noise distribution p n .
Revised: One draws positive samples ( x, y ) from the training set and constructs negative samples ( x, y (cid:48) ) by drawing random labels y (cid:48) from some noise distribution p n .

Original: One then trains a logistic regression by minimizing the stochastic loss function (cid:96)
Revised: One then trains a logistic regression by minimizing the stochastic loss function (cid:96)

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: with the sigmoid function σ ( z ) = 1 / (1 + e − z ) .
Revised: with the sigmoid function σ ( z ) = 1 / (1 + e − z ) .

Original: Here, we used the same score functions ξ y as in Eq.but introduced different model parameters φ so that we can distinguish the two models.
Revised: Here, we used the same score functions ξ y as in Eq.but introduced different model parameters φ so that we can distinguish the two models.

Original: Gradient steps for (cid:96) neg.sampl ( φ ) cost only O ( K ) time as there is no sum over all labels y (cid:48)
Revised: Gradient steps for (cid:96) neg.sampl ( φ ) cost only O ( K ) time as there is no sum over all labels y (cid:48)

Original: ∈ Y .
Revised: ∈ Y .

Original: Asymptotic Equivalence.
Revised: Asymptotic Equivalence.

Original: The models in Eqs.
Revised: The models in Eqs.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: 2 are exactly equivalent in the nonparametric limit , i.e., if the function class x (cid:55)→ ξ y ( x, θ ) is ﬂexible enough to map x to any possible score.
Revised: 2 are exactly equivalent in the nonparametric limit , i.e., if the function class x (cid:55)→ ξ y ( x, θ ) is ﬂexible enough to map x to any possible score.

Original: A further requirement is that p n in Eq.
Revised: A further requirement is that p n in Eq.

Original: 2 is the uniform distribution over Y .
Revised: 2 is the uniform distribution over Y .

Original: If both conditions hold, it follows that if θ ∗ and φ ∗ minimize Eq. 1 and Eq. 2, respectively, they learn identical scores,
Revised: If both conditions hold, it follows that if θ ∗ and φ ∗ minimize Eq. 1 and Eq. 2, respectively, they learn identical scores,

Original: [Equation]
Revised: [Equation]

Original: As a consequence, one is free to choose the loss function that is easier to minimize.
Revised: As a consequence, one is free to choose the loss function that is easier to minimize.

Original: While gradient steps are cheaper by a factor of O ( C ) for negative sampling, the randomly drawn negative samples increase the variance of the stochastic gradient estimator and worsen the signal-to-noise ratio of the gradient, slowing-down convergence.
Revised: While gradient steps are cheaper by a factor of O ( C ) for negative sampling, the randomly drawn negative samples increase the variance of the stochastic gradient estimator and worsen the signal-to-noise ratio of the gradient, slowing-down convergence.

Original: The next section combines the strengths of both approaches.
Revised: The next section combines the strengths of both approaches.

Original: 2. A DVERSARIAL N EGATIVE S AMPLING
Revised: 2. A DVERSARIAL N EGATIVE S AMPLING

Original: Overview.
Revised: Overview.

Original: We propose a generalized variant of negative sampling that reduces the gradient noise.
Revised: We propose a generalized variant of negative sampling that reduces the gradient noise.

Original: The main idea is to train with negative samples y (cid:48) that are hard to distinguish from positive samples.
Revised: The main idea is to train with negative samples y (cid:48) that are hard to distinguish from positive samples.

Original: We draw y (cid:48) from a conditional noise distribution p n ( y (cid:48) | x ) using an auxiliary model.
Revised: We draw y (cid:48) from a conditional noise distribution p n ( y (cid:48) | x ) using an auxiliary model.

Original: This introduces a bias, which we remove at prediction time.
Revised: This introduces a bias, which we remove at prediction time.

Original: In summary our proposed approach consists of three steps:
Revised: In summary our proposed approach consists of three steps:

Original: Parameterize the noise distribution p n ( y (cid:48) | x ) by an
Revised: Parameterize the noise distribution p n ( y (cid:48) | x ) by an

Original: auxiliary
Revised: auxiliary

Original: model and ﬁt it
Revised: model and ﬁt it

Original: to
Revised: to

Original: the data
Revised: the data

Original: set.
Revised: set.

Original: Train a classiﬁer via negative sampling (Eq. 2) using adversarial negative samples from the auxiliary model ﬁtted in Step 1 above.
Revised: Train a classiﬁer via negative sampling (Eq. 2) using adversarial negative samples from the auxiliary model ﬁtted in Step 1 above.

Original: For our proposed auxiliary model, drawing a negative sample costs only O ( k log C ) time with some k < K , i.e., it is sublinear in C . 3.
Revised: For our proposed auxiliary model, drawing a negative sample costs only O ( k log C ) time with some k < K , i.e., it is sublinear in C . 3.

Original: The resulting model has a bias.
Revised: The resulting model has a bias.

Original: When making predictions, remove the bias by mapping it to an unbiased softmax classiﬁer using the generalized asymptotic equivalence in Eq.
Revised: When making predictions, remove the bias by mapping it to an unbiased softmax classiﬁer using the generalized asymptotic equivalence in Eq.

Original: 5 below.
Revised: 5 below.

Original: We elaborate on the above Step 1 in Section 3.
Revised: We elaborate on the above Step 1 in Section 3.

Original: In the present section, we focus instead on Step 2 and its dependency on the choice of noise distribution p n , and on the bias removal (Step 3).
Revised: In the present section, we focus instead on Step 2 and its dependency on the choice of noise distribution p n , and on the bias removal (Step 3).

Original: Why Adversarial Noise Improves Learning.
Revised: Why Adversarial Noise Improves Learning.

Original: We ﬁrst provide some intuition why uniform negative sampling is not optimal, and how sampling from a non-uniform noise distribution may improve the gradient signal.
Revised: We ﬁrst provide some intuition why uniform negative sampling is not optimal, and how sampling from a non-uniform noise distribution may improve the gradient signal.

Original: We argue that the poor gradient signal is caused by the fact that negative samples are too easy to distinguish from positive samples.
Revised: We argue that the poor gradient signal is caused by the fact that negative samples are too easy to distinguish from positive samples.

Original: A data set with many categories is typically comprised of several hierarchical clusters, with large clusters of generic concepts and small sub-clusters of specialized concepts.
Revised: A data set with many categories is typically comprised of several hierarchical clusters, with large clusters of generic concepts and small sub-clusters of specialized concepts.

Original: When drawing negative samples uniformly across the data set, the correct label will likely belong to a different generic concept than the negative sample.
Revised: When drawing negative samples uniformly across the data set, the correct label will likely belong to a different generic concept than the negative sample.

Original: For example, an image classiﬁer will therefore quickly learn to distinguish, e.g., dogs from bicycles, but since negative samples from the same cluster are rare, it takes much longer to learn the differences between a Boston
Revised: For example, an image classiﬁer will therefore quickly learn to distinguish, e.g., dogs from bicycles, but since negative samples from the same cluster are rare, it takes much longer to learn the differences between a Boston

Original: Terrier and a French Bulldog.
Revised: Terrier and a French Bulldog.

Original: The model quickly learns to assign very low scores ξ y (cid:48)
Revised: The model quickly learns to assign very low scores ξ y (cid:48)

Original: ( x, φ ) (cid:28) 0 to such ‘obviously wrong’ labels, making their contribution to the gradient exponentially small,
Revised: ( x, φ ) (cid:28) 0 to such ‘obviously wrong’ labels, making their contribution to the gradient exponentially small,

Original: [Equation]
Revised: [Equation]

Original: A similar vanishing gradient problem was pointed out for word embeddings by Chen et al.
Revised: A similar vanishing gradient problem was pointed out for word embeddings by Chen et al.

Original: Here, the vanishing gradient is due to different word frequencies, and a popular solution is therefore to draw negative samples from a nonuniform but unconditional noise distribution p n ( y (cid:48) )
Revised: Here, the vanishing gradient is due to different word frequencies, and a popular solution is therefore to draw negative samples from a nonuniform but unconditional noise distribution p n ( y (cid:48) )

Original: based on the empirical word frequencies (Mikolov et al., 2013).
Revised: based on the empirical word frequencies (Mikolov et al., 2013).

Original: This introduces a bias which does not matter for word embeddings since the focus is not on classiﬁcation but rather on learning useful representations.
Revised: This introduces a bias which does not matter for word embeddings since the focus is not on classiﬁcation but rather on learning useful representations.

Original: Going beyond frequency-adjusted negative sampling, we show that one can drastically improve the procedure by generating negative samples from an auxiliary model.
Revised: Going beyond frequency-adjusted negative sampling, we show that one can drastically improve the procedure by generating negative samples from an auxiliary model.

Original: We therefore propose to generate negative samples y (cid:48) ∼ p n ( y (cid:48)
Revised: We therefore propose to generate negative samples y (cid:48) ∼ p n ( y (cid:48)

Original: | x ) conditioned on the input feature x .
Revised: | x ) conditioned on the input feature x .

Original: This has the advantage that the distribution of negative samples can be made much more similar to the distribution of positive samples, leading to a better signal-to-noise ratio.
Revised: This has the advantage that the distribution of negative samples can be made much more similar to the distribution of positive samples, leading to a better signal-to-noise ratio.

Original: One consequence is that the introduced bias can no longer be ignored, which is what we address next.
Revised: One consequence is that the introduced bias can no longer be ignored, which is what we address next.

Original: Bias Removal.
Revised: Bias Removal.

Original: Negative sampling with a nonuniform noise distribution introduces a bias.
Revised: Negative sampling with a nonuniform noise distribution introduces a bias.

Original: For a given input feature vector x , labels y (cid:48) with a high noise probability p n ( y (cid:48)
Revised: For a given input feature vector x , labels y (cid:48) with a high noise probability p n ( y (cid:48)

Original: | x ) are frequently drawn as negative samples, causing the model to learn a low score ξ y (cid:48)
Revised: | x ) are frequently drawn as negative samples, causing the model to learn a low score ξ y (cid:48)

Original: ( x, φ ∗ ) .
Revised: ( x, φ ∗ ) .

Original: Conversely, a low p n ( y (cid:48) | x ) leads to an inﬂated score ξ y (cid:48)
Revised: Conversely, a low p n ( y (cid:48) | x ) leads to an inﬂated score ξ y (cid:48)

Original: ( x, φ ∗ ) .
Revised: ( x, φ ∗ ) .

Original: It turns out that this bias can be easily quantiﬁed via a generalization of Eq.
Revised: It turns out that this bias can be easily quantiﬁed via a generalization of Eq.

Original: We prove in Theorem 1 (Section 4) that in the nonparametric limit for arbitrary p n ( y (cid:48) | x ) ,
Revised: We prove in Theorem 1 (Section 4) that in the nonparametric limit for arbitrary p n ( y (cid:48) | x ) ,

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 5 is an asymptotic equivalence between softmax classiﬁcation (Eq. 1) and generalized negative sampling
Revised: 5 is an asymptotic equivalence between softmax classiﬁcation (Eq. 1) and generalized negative sampling

Original: (Eq. 2).
Revised: (Eq. 2).

Original: While strict equality holds only in the nonparametric limit, many models are ﬂexible enough that Eq.
Revised: While strict equality holds only in the nonparametric limit, many models are ﬂexible enough that Eq.

Original: 5 holds approximately in practice.
Revised: 5 holds approximately in practice.

Original: Eq.
Revised: Eq.

Original: 5 allows us to make unbiased predictions by mapping biased negative sampling scores ξ y ( x, φ ∗ ) to unbiased softmax scores ξ y ( x, θ ∗ ) .
Revised: 5 allows us to make unbiased predictions by mapping biased negative sampling scores ξ y ( x, φ ∗ ) to unbiased softmax scores ξ y ( x, θ ∗ ) .

Original: There is no need to solve for the corresponding model parameters θ ∗ , the scores ξ y ( x, θ ∗ ) sufﬁce for predictions.
Revised: There is no need to solve for the corresponding model parameters θ ∗ , the scores ξ y ( x, θ ∗ ) sufﬁce for predictions.

Original: Regularization.
Revised: Regularization.

Original: In practice, softmax classiﬁcation typically requires a regularizer with some strength λ > 0 to prevent overﬁtting.
Revised: In practice, softmax classiﬁcation typically requires a regularizer with some strength λ > 0 to prevent overﬁtting.

Original: With the asymptotic equivalence in Eq. 5, regularizing the softmax scores ξ y ( x, θ ) is similar to regularizing ξ y ( x, φ ) + log p n ( y | x ) in the proposed generalized negative sampling method.
Revised: With the asymptotic equivalence in Eq. 5, regularizing the softmax scores ξ y ( x, θ ) is similar to regularizing ξ y ( x, φ ) + log p n ( y | x ) in the proposed generalized negative sampling method.

Original: We thus propose to use the following regularized variant of Eq. 2,
Revised: We thus propose to use the following regularized variant of Eq. 2,

Original: [Equation]
Revised: [Equation]

Original: Comparison to GANs.
Revised: Comparison to GANs.

Original: The use of adversarial negative samples, i.e., negative samples that are designed to ‘confuse’ the logistic regression in Eq. 2, bears some resemblance to generative adversarial networks (GANs) (Goodfellow et al., 2014).
Revised: The use of adversarial negative samples, i.e., negative samples that are designed to ‘confuse’ the logistic regression in Eq. 2, bears some resemblance to generative adversarial networks (GANs) (Goodfellow et al., 2014).

Original: The crucial difference is that GANs are generative models, whereas we train a discriminative model over a discrete label space Y .
Revised: The crucial difference is that GANs are generative models, whereas we train a discriminative model over a discrete label space Y .

Original: The ‘generator’ p n in our setup only needs to ﬁnd a rough approximation of the (conditional) label distribution because the ﬁnal predictive scores in Eq.
Revised: The ‘generator’ p n in our setup only needs to ﬁnd a rough approximation of the (conditional) label distribution because the ﬁnal predictive scores in Eq.

Original: 5 combine the ‘generator scores’ log p n ( y | x ) with the more expressive ‘discriminator scores’ ξ y ( x, φ ∗ ) .
Revised: 5 combine the ‘generator scores’ log p n ( y | x ) with the more expressive ‘discriminator scores’ ξ y ( x, φ ∗ ) .

Original: This allows us to use a very restrictive but efﬁcient generator model (see Section 3 below) that we can keep constant while training the discriminator.
Revised: This allows us to use a very restrictive but efﬁcient generator model (see Section 3 below) that we can keep constant while training the discriminator.

Original: By contrast, the focus in GANs is on ﬁnding the best possible generator, which requires concurrent training of a generator and a discriminator via a potentially unstable nested min-max optimization.
Revised: By contrast, the focus in GANs is on ﬁnding the best possible generator, which requires concurrent training of a generator and a discriminator via a potentially unstable nested min-max optimization.

Original:  C ONDITIONAL G ENERATION OF A DVERSARIAL S AMPLES
Revised:  C ONDITIONAL G ENERATION OF A DVERSARIAL S AMPLES

Original: Having proposed a general approach for improved negative sampling with an adversarial auxiliary model p n (Section 2), we now describe a simple construction for such  model that satisﬁes all requirements.
Revised: Having proposed a general approach for improved negative sampling with an adversarial auxiliary model p n (Section 2), we now describe a simple construction for such a model that satisﬁes all requirements.

Original: The model is essentially a probabilistic version of a decision tree which is able to conditionally generate negative samples by ancestral sampling.
Revised: The model is essentially a probabilistic version of a decision tree which is able to conditionally generate negative samples by ancestral sampling.

Original: Readers who prefer to proceed can skip this section without loosing the main thread of the paper.
Revised: Readers who prefer to proceed can skip this section without loosing the main thread of the paper.

Original: Our auxiliary model has the following properties: (i) it can be efﬁciently ﬁtted to the training data D requiring minimal hyperparameter tuning and subleading computational overhead over the training of the main model; (ii) drawing negative samples y (cid:48) ∼ p n ( y (cid:48)
Revised: Our auxiliary model has the following properties: (i) it can be efﬁciently ﬁtted to the training data D requiring minimal hyperparameter tuning and subleading computational overhead over the training of the main model; (ii) drawing negative samples y (cid:48) ∼ p n ( y (cid:48)

Original: | x ) scales only as O (log |Y| ) , thus improving over the linear scaling of the softmax loss function (Eq. 1); and (iii) the log likelihood log p n ( y | x ) can be evaluated explicitly so that we can apply the bias removal in Eq.
Revised: | x ) scales only as O (log |Y| ) , thus improving over the linear scaling of the softmax loss function (Eq. 1); and (iii) the log likelihood log p n ( y | x ) can be evaluated explicitly so that we can apply the bias removal in Eq.

Original: Satisfying requirements (i) and (ii) on model efﬁciency comes at the cost of some model performance.
Revised: Satisfying requirements (i) and (ii) on model efﬁciency comes at the cost of some model performance.

Original: This is an acceptable trade-off since the performance of p n affects only the quality of negative samples.
Revised: This is an acceptable trade-off since the performance of p n affects only the quality of negative samples.

Original: Model.
Revised: Model.

Original: Our auxiliary model for p n is inspired by the Hierarchical Softmax model due to Morin &
Revised: Our auxiliary model for p n is inspired by the Hierarchical Softmax model due to Morin &

Original: Bengio (2005).
Revised: Bengio (2005).

Original: It is a balanced probabilistic binary decision tree, where each leaf node is mapped uniquely to a label y ∈ Y .
Revised: It is a balanced probabilistic binary decision tree, where each leaf node is mapped uniquely to a label y ∈ Y .

Original: A decision tree imposes a hierarchical structure on Y , which can impede performance if it does not reﬂect any semantic structure in Y .
Revised: A decision tree imposes a hierarchical structure on Y , which can impede performance if it does not reﬂect any semantic structure in Y .

Original: Morin & Bengio (2005) rely on an explicitly provided semantic hierarchical structure, or ‘ontology’.
Revised: Morin & Bengio (2005) rely on an explicitly provided semantic hierarchical structure, or ‘ontology’.

Original: Since an ontology is often not available, we instead construct a hierarchical structure in a data driven way.
Revised: Since an ontology is often not available, we instead construct a hierarchical structure in a data driven way.

Original: Our method has some similarity to the approach by Mnih & Hinton (2009), but it is more principled in that we ﬁt both the model parameters and the hierarchical structure by maximizing a single log likelihood function.
Revised: Our method has some similarity to the approach by Mnih & Hinton (2009), but it is more principled in that we ﬁt both the model parameters and the hierarchical structure by maximizing a single log likelihood function.

Original: To sample from the model, one walks from the tree’s root to some leaf.
Revised: To sample from the model, one walks from the tree’s root to some leaf.

Original: At each node ν , one makes a binary decision ζ ∈ {± 1 } whether to continue to the right child ( ζ = 1 ) or to the left child ( ζ = − 1 ).
Revised: At each node ν , one makes a binary decision ζ ∈ {± 1 } whether to continue to the right child ( ζ = 1 ) or to the left child ( ζ = − 1 ).

Original: Given a feature vector x , we model the likelihood of these decisions as σ (cid:0) ζ ( w (cid:62) ν x + b ν ) (cid:1) , where the weight vector w ν and the scalar bias b ν are model parameters associated with node ν .
Revised: Given a feature vector x , we model the likelihood of these decisions as σ (cid:0) ζ ( w (cid:62) ν x + b ν ) (cid:1) , where the weight vector w ν and the scalar bias b ν are model parameters associated with node ν .

Original: Denoting the unique path π y from the root node ν 0 to the leaf node associated with label y as a sequence of nodes and binary decisions, π y ≡ (cid:0) ( ν 0 , ζ 0 ) , ( ν 1 , ζ 1 ) , . . .
Revised: Denoting the unique path π y from the root node ν 0 to the leaf node associated with label y as a sequence of nodes and binary decisions, π y ≡ (cid:0) ( ν 0 , ζ 0 ) , ( ν 1 , ζ 1 ) , . . .

Original: (cid:1) , the log likelihood of the training set D is thus
Revised: (cid:1) , the log likelihood of the training set D is thus

Original: [Equation]
Revised: [Equation]

Original: Greedy Model Fitting.
Revised: Greedy Model Fitting.

Original: We maximize the likelihood L in Eq.
Revised: We maximize the likelihood L in Eq.

Original: 7 over (i) the model parameters w ν
Revised: 7 over (i) the model parameters w ν

Original: and b ν of all nodes ν , and (ii) the hierarchical structure, i.e., the mapping between labels y and leaf nodes.
Revised: and b ν of all nodes ν , and (ii) the hierarchical structure, i.e., the mapping between labels y and leaf nodes.

Original: The latter involves an exponentially large search space, making exact maximization intractable.
Revised: The latter involves an exponentially large search space, making exact maximization intractable.

Original: We use a greedy approximation where we recursively split the label set Y into halves and associate each node ν with a subset Y ν ⊆ Y .
Revised: We use a greedy approximation where we recursively split the label set Y into halves and associate each node ν with a subset Y ν ⊆ Y .

Original: We start at the root node ν 0 with Y ν 

Revised: We start at the root node ν 0 with Y ν

Original: = Y and ﬁnishing at the leaves with a single label per leaf.
Revised: = Y and ﬁnishing at the leaves with a single label per leaf.

Original: For each node ν , we maximize the terms in L that depend on w ν and b ν .
Revised: For each node ν , we maximize the terms in L that depend on w ν and b ν .

Original: These terms correspond to data points with a label y ∈ Y ν , leading to the objective
Revised: These terms correspond to data points with a label y ∈ Y ν , leading to the objective

Original: [Equation]
Revised: [Equation]

Original: We alternate between a continuous maximization of L ν over w ν and b ν , and a discrete maximization over the binary indicators ζ y ∈ {± 1 } that deﬁne how we split Y ν into two equally sized halves.
Revised: We alternate between a continuous maximization of L ν over w ν and b ν , and a discrete maximization over the binary indicators ζ y ∈ {± 1 } that deﬁne how we split Y ν into two equally sized halves.

Original: The continuous optimization is over a convex function and it converges quickly to machine precision with
Revised: The continuous optimization is over a convex function and it converges quickly to machine precision with

Original: Newton ascent, which is free of hyperparameters like learning rates.
Revised: Newton ascent, which is free of hyperparameters like learning rates.

Original: For the discrete optimization, we note that changing ξ y for any y ∈ Y ν from − 1 to 1 (or from 1 to − 1 ) increases (or decreases)
Revised: For the discrete optimization, we note that changing ξ y for any y ∈ Y ν from − 1 to 1 (or from 1 to − 1 ) increases (or decreases)

Original: L ν by
Revised: L ν by

Original: [Equation]
Revised: [Equation]

Original: Here, the sums over D y run over all data points in D with label y , and the second equality is an algebraic identity of the sigmoid function.
Revised: Here, the sums over D y run over all data points in D with label y , and the second equality is an algebraic identity of the sigmoid function.

Original: We maximize L ν over all ζ y under the boundary condition that the split be into equally sized halves by setting ζ y ← 1 for the half of y ∈ Y ν with largest ∆ y and ζ y ← − 1 for the other half.
Revised: We maximize L ν over all ζ y under the boundary condition that the split be into equally sized halves by setting ζ y ← 1 for the half of y ∈ Y ν with largest ∆ y and ζ y ← − 1 for the other half.

Original: If this changes any ζ y then we switch back to the continuous optimization.
Revised: If this changes any ζ y then we switch back to the continuous optimization.

Original: Otherwise, we have reached a local optimum for node ν , and we proceed to the next node.
Revised: Otherwise, we have reached a local optimum for node ν , and we proceed to the next node.

Original: Technical Details.
Revised: Technical Details.

Original: In the interest of clarity, the above description left out the following details.
Revised: In the interest of clarity, the above description left out the following details.

Original: Most importantly, to prioritize efﬁciency over accuracy, we preprocess the feature vectors x and project them to a smaller space R k with k < K using principal component analysis (PCA).
Revised: Most importantly, to prioritize efﬁciency over accuracy, we preprocess the feature vectors x and project them to a smaller space R k with k < K using principal component analysis (PCA).

Original: Sampling from p n thus costs only O ( k log |Y| ) time.
Revised: Sampling from p n thus costs only O ( k log |Y| ) time.

Original: This dimensionality reduction only affects the quality of negative samples.
Revised: This dimensionality reduction only affects the quality of negative samples.

Original: The main model (Eq. 2) still operates on the full feature space R K .
Revised: The main model (Eq. 2) still operates on the full feature space R K .

Original: Second, we add a quadratic regularizer − λ n ( || w ν
Revised: Second, we add a quadratic regularizer − λ n ( || w ν

Original: || 

Revised: ||

Original: + b 2 ν ) to L ν , with strength λ n set by cross validation.
Revised: + b 2 ν ) to L ν , with strength λ n set by cross validation.

Original: Third, we introduce uninhabited padding labels if |Y| is not a power of two.
Revised: Third, we introduce uninhabited padding labels if |Y| is not a power of two.

Original: We ensure that p n (˜ y | x ) = 

Revised: We ensure that p n (˜ y | x ) =

Original: for all padding labels ˜
Revised: for all padding labels ˜

Original: y by setting b ν to a very large positive or negative value if either of ν ’s children contains only padding labels.
Revised: y by setting b ν to a very large positive or negative value if either of ν ’s children contains only padding labels.

Original: Finally, we initialize the optimization with b ν ← 0 and by setting w ν ∈ R k to the dominant eigenvector of the covariance matrix of the set of vectors { (cid:80) x ∈D y x } y ∈Y ν .
Revised: Finally, we initialize the optimization with b ν ← 0 and by setting w ν ∈ R k to the dominant eigenvector of the covariance matrix of the set of vectors { (cid:80) x ∈D y x } y ∈Y ν .

Original:  T HEORETICAL A SPECTS
Revised:  T HEORETICAL A SPECTS

Original: We formalize and proof the two main premises of the algorithm proposed in Section 2.2.
Revised: We formalize and prove the two main premises of the algorithm proposed in Section 2.2.

Original: Theorembelow states the equivalence between softmax classiﬁcation and negative sampling (Eq. 5), and
Revised: Theorembelow states the equivalence between softmax classiﬁcation and negative sampling (Eq. 5), and

Original: Theorem 2 formalizes the claim that adversarial negative samples maximize the signal-to-noise ratio.
Revised: Theorem 2 formalizes the claim that adversarial negative samples maximize the signal-to-noise ratio.

Original: Theorem 1.
Revised: Theorem 1.

Original: In the nonparametric limit, the optimal model parameters θ ∗ and φ ∗ that minimize (cid:96) softmax ( θ ) in Eq. 1 and (cid:96) neg.sampl.
Revised: In the nonparametric limit, the optimal model parameters θ ∗ and φ ∗ that minimize (cid:96) softmax ( θ ) in Eq. 1 and (cid:96) neg.sampl.

Original: ( φ ) in Eq. 2, respectively, satisfy Eq.
Revised: ( φ ) in Eq. 2, respectively, satisfy Eq.

Original: 5 for all x in the data set and all y ∈ Y .
Revised: 5 for all x in the data set and all y ∈ Y .

Original: Here, the “const.” term on the right-hand side of Eq.
Revised: Here, the “const.” term on the right-hand side of Eq.

Original: 5 is independent of y .
Revised: 5 is independent of y .

Original: Proof.
Revised: Proof.

Original: Minimizing (cid:96) softmax ( θ ) ﬁts the maximum likelihood estimate of a model with likelihood p θ ( y | x ) =
Revised: Minimizing (cid:96) softmax ( θ ) ﬁts the maximum likelihood estimate of a model with likelihood p θ ( y | x ) =

Original: e ξ y ( x,θ ) /Z
Revised: e ξ y ( x,θ ) /Z

Original: θ ( x ) with normalization Z θ ( x ) =
Revised: θ ( x ) with normalization Z θ ( x ) =

Original: (cid:80) y (cid:48)
Revised: (cid:80) y (cid:48)

Original: ∈Y e ξ y (cid:48)
Revised: ∈Y e ξ y (cid:48)

Original: ( x,θ ) .
Revised: ( x,θ ) .

Original: In the nonparametric limit, the score functions ξ y ( x, θ ) are arbitrarily ﬂexible, allowing for a perfect ﬁt, thus
Revised: In the nonparametric limit, the score functions ξ y ( x, θ ) are arbitrarily ﬂexible, allowing for a perfect ﬁt, thus

Original: [Equation]
Revised: [Equation]

Original: Similarly, (cid:96) neg.sampl.
Revised: Similarly, (cid:96) neg.sampl.

Original: ( φ ) is the maximum likelihood objective of a binomial model that discriminates positive from negative samples.
Revised: ( φ ) is the maximum likelihood objective of a binomial model that discriminates positive from negative samples.

Original: The nonparametric limit admits again a perfect ﬁt so that the learned ratio of positive rate σ ( ξ y ( x, φ )) to negative rate σ ( − ξ y
Revised: The nonparametric limit admits again a perfect ﬁt so that the learned ratio of positive rate σ ( ξ y ( x, φ )) to negative rate σ ( − ξ y

Original: ( x, φ )) equals the empirical ratio,
Revised: ( x, φ )) equals the empirical ratio,

Original: [Equation]
Revised: [Equation]

Original: where we used the identity σ ( z ) /σ ( − z )
Revised: where we used the identity σ ( z ) /σ ( − z )

Original: =
Revised: =

Original: e z .
Revised: e z .

Original: Inserting Eq.
Revised: Inserting Eq.

Original: 10 for p D ( y | x ) and taking the logarithm leads to Eq.
Revised: 10 for p D ( y | x ) and taking the logarithm leads to Eq.

Original: Here, the “const.”
Revised: Here, the “const.”

Original: term works out to log Z θ ∗ ( x ) , which is indeed independent of y .
Revised: term works out to log Z θ ∗ ( x ) , which is indeed independent of y .

Original: Signal-to-Noise Ratio.
Revised: Signal-to-Noise Ratio.

Original: In preparation for Theorem 2 below, we deﬁne a quantitative measure for the signal-to-noise ratio (SNR) in stochastic gradient descent (SGD).
Revised: In preparation for Theorem 2 below, we deﬁne a quantitative measure for the signal-to-noise ratio (SNR) in stochastic gradient descent (SGD).

Original: In the vicinity of the minimum φ ∗ of the loss function (cid:96) ( φ ) , thetrue gradient g ≈ H (cid:96) ( φ − φ ∗ ) is approximately proportional to the 
Revised: In the vicinity of the minimum φ ∗ of a loss function (cid:96) ( φ ) , the gradient g ≈ H (cid:96) ( φ − φ ∗ ) is approximately proportional to the Hessian

Original: Hessian H (cid:96) of (cid:96) at φ ∗ .
Revised: H (cid:96) of (cid:96) at φ ∗ .

Original: SGD estimates g via stochastic gradient estimates ˆ g , whose noise is measured by the covariance matrix
Revised: SGD estimates g via stochastic gradient estimates ˆ g , whose noise is measured by the covariance matrix

Original: Cov [ˆ g, ˆ g ] .
Revised: Cov [ˆ g, ˆ g ] .

Original: Thus, the eigenvalues { η i } of the matrix A :
Revised: Thus, the eigenvalues { η i } of the matrix A :

Original: = H (cid:96) Cov
Revised: = H (cid:96) Cov

Original: [ˆ g, ˆ g ] −measure the SNR along different directions in parameter space.
Revised: [ˆ g, ˆ g ] − 1 measure the SNR along different directions in parameter space.

Original: We deﬁne an overall scalar SNR ¯ η as
Revised: We deﬁne an overall scalar SNR ¯ η as

Original: [Equation]
Revised: [Equation]

Original: Here, we sum over the inverses 1 /η i rather than η i so that ¯ η ≤ min i η i and thus maximizing ¯ η encourages large values for all η i .
Revised: Here, we sum over the inverses 1 /η i rather than η i so that ¯ η ≤ min i η i and thus maximizing ¯ η encourages large values for all η i .

Original: The deﬁnition in Eq.
Revised: The deﬁnition in Eq.

Original: 12 has the additional nice property that ¯ η is invariant under arbitrary invertible reparameterization of φ .
Revised: 12 has the useful property that ¯ η is invariant under arbitrary invertible reparameterization of φ .

Original: Expressing φ in terms of new model parameters φ (cid:48) maps H (cid:96) to J (cid:62) H (cid:96)
Revised: Expressing φ in terms of new model parameters φ (cid:48) maps H (cid:96) to J (cid:62) H (cid:96)

Original: J and Cov [ˆ g, ˆ g ] to J (cid:62) Cov [ˆ g, ˆ g ] J , where J :=
Revised: J and Cov [ˆ g, ˆ g ] to J (cid:62) Cov [ˆ g, ˆ g ] J , where J :=

Original: ∂φ/∂φ (cid:48) is the Jacobian.
Revised: ∂φ/∂φ (cid:48) is the Jacobian.

Revised: Inserting into Eq.

Original: Using the cyclic property of the trace,
Revised: 12 and using the cyclic property of the trace,

Original: Tr [ PQ ] =
Revised: Tr [ PQ ] =

Original: Tr [ QP ] , all Jacobiansin Eq. 12 cancel.
Revised: Tr [ QP ] , all Jacobians cancel.

Original: Theorem 2.
Revised: Theorem 2.

Original: For negative sampling (Eq. 2) in the nonparametric limit, the signal-to-noise ratio ¯ η deﬁned in Eq.
Revised: For negative sampling (Eq. 2) in the nonparametric limit, the signal-to-noise ratio ¯ η deﬁned in Eq.

Original: 12 is maximal if p n = p D , i.e., for adversarial negative samples.
Revised: 12 is maximal if p n = p D , i.e., for adversarial negative samples.

Original: Proof.
Revised: Proof.

Original: In the nonparametric limit, the scores ξ y ( x, φ ) can be regarded as independent variables for all x and y .
Revised: In the nonparametric limit, the scores ξ y ( x, φ ) can be regarded as independent variables for all x and y .

Original: We therefore treat the scores directly as model parameters, using the invariance of ¯ η under reparameterization.
Revised: We therefore treat the scores directly as model parameters, using the invariance of ¯ η under reparameterization.

Original: Using only Eq. 2, Eq. 11, and properties of the σ -function, we show in the
Revised: Using only Eq. 2, Eq. 11, and properties of the σ -function, we show in the

Original: Appendix that the Hessian of the loss function is diagonal in this coordinate system, and given by
Revised: Appendix that the Hessian of the loss function is diagonal in this coordinate system, and given by

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and that the noise covariance matrix is block diagonal,
Revised: and that the noise covariance matrix is block diagonal,

Original: [Equation]
Revised: [Equation]

Original: 14) where α x, : ≡ ( α x,y )
Revised: blocks [Equation] [Equation] where α x, : ≡ ( α x,y )

Original: y ∈Y denotes a |Y| -dimensional column vector.
Revised: y ∈Y denotes a |Y| -dimensional column vector.

Original: Thus, the trace in Eq.
Revised: Thus, the trace in Eq.

Original: 12 is
Revised: 12 is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We thus have to maximize (cid:80) y ∈Y α x,y for each x in the training set.
Revised: We thus have to maximize (cid:80) y ∈Y α x,y for each x in the training set.

Original: We ﬁnd from Eq.
Revised: We ﬁnd from Eq.

Original: 13 and Eq.
Revised: 13 and Eq.

Original: 11, (cid:88)
Revised: 11, (cid:88)

Revised: y ∈Y

Original: [Equation]
Revised: [Equation]

Original: with f ( z ) :
Revised: with f ( z ) :

Original: = 1 / (1 + 1 /z ) .
Revised: = 1 / (1 + 1 /z ) .

Original: Using Jensen’s inequality for the concave function f , we ﬁnd that the right-hand side of Eq.
Revised: Using Jensen’s inequality for the concave function f , we ﬁnd that the right-hand side of Eq.

Original: 16 has the upper bound f (cid:0) E p n ( y | x ) [ p D ( y | x ) /p
Revised: 16 has the upper bound f (cid:0) E p n ( y | x ) [ p D ( y | x ) /p

Original: n ( y | x )] (cid:1) =
Revised: n ( y | x )] (cid:1) =

Original: f (1) = 12 , which it reaches precisely if the argument of f in Eq.
Revised: f (1) = 12 , which it reaches precisely if the argument of f in Eq.

Original: 16 is a constant, i.e., iff p n ( y | x ) =
Revised: 16 is a constant, i.e., iff p n ( y | x ) =

Original: p D ( y | x ) ∀ y ∈ Y .
Revised: p D ( y | x ) ∀ y ∈ Y .

Original: [Table]

Original:  R ESULTS
Revised:  R ESULTS

Original: We evaluated the proposed adversarial negative sampling method on two data sets from the Extreme Classiﬁciation Repository (Bhatia et al.) [Table] and we speed of convergence and predictive performance againstdifferent ﬁve baselines on each data set.
Revised: We evaluated the proposed adversarial negative sampling method on two established benchmarks by comparing speed of convergence and predictive performance against ﬁve different baselines.

Original: Both data sets contain more than 200,000 labels, making direct training of a softmax classiﬁer unfeasible. 
Revised: Both data sets contain more than 200,000 labels, making direct training of a softmax classiﬁer unfeasible. [Table]

Original: Datasets, Preprocessing and Model.
Revised: Datasets, Preprocessing and Model.

Original: We used the Wikipedia-500K and Amazon-670K data sets from the Extreme Classiﬁcation Repository (Bhatia et al.) with preprocessed XML-CNN features (Liu et al., 2017) downloaded from (Saxena).
Revised: We used the Wikipedia-500K and Amazon-670K data sets from the Extreme Classiﬁcation Repository (Bhatia et al.) with preprocessed XML-CNN features (Liu et al., 2017) downloaded from (Saxena).

Original: Both data sets contain multiple labels per data point.
Revised: Both data sets contain multiple labels per data point.

Original: To obtain data sets for single-class classiﬁcation, we follow the approach in (Ruiz et al., 2018) and keep only the ﬁrst label for each data point.
Revised: To obtain data sets for single-class classiﬁcation, we follow the approach in (Ruiz et al., 2018) and keep only the ﬁrst label for each data point.

Original: Table 1 shows the resulting sizes.
Revised: Table 1 shows the resulting sizes.

Original: Both data sets contain
Revised: Both data sets contain

Original: K = 512 -dimensional feature vectors.
Revised: K = 512 -dimensional feature vectors.

Original: We ﬁt a liner model with scores ξ y ( x, φ ) =
Revised: We ﬁt a liner model with scores ξ y ( x, φ ) =

Original: x (cid:62) w y + b y , where the model parameters φ are the weight vectors w y ∈ R K and biases b y ∈ R for each label y .
Revised: x (cid:62) w y + b y , where the model parameters φ are the weight vectors w y ∈ R K and biases b y ∈ R for each label y .

Original: Baselines.
Revised: Baselines.

Original: We compare our proposed method to ﬁve baselines: (i) standard negative sampling with a uniform noise distribution; (ii) negative sampling with an unconditional noise distribution p n ( y (cid:48) ) set to the empirical label frequencies; (iii) noise contrastive estimation (NCE, see below); (iv) ‘Augment and Reduce’ (Ruiz et al., 2018); and (v) ‘
Revised: We compare our proposed method to ﬁve baselines: (i) standard negative sampling with a uniform noise distribution; (ii) negative sampling with an unconditional noise distribution p n ( y (cid:48) ) set to the empirical label frequencies; (iii) noise contrastive estimation (NCE, see below); (iv) ‘Augment and Reduce’ (Ruiz et al., 2018); and (v) ‘

Original: One vs. Each’ (Titsias, 2016).
Revised: One vs. Each’ (Titsias, 2016).

Original: We do not compare to explicit softmax approximations classiﬁcation via Eq. 1, since both data setsare far too big (see Table 1) for it to be feasible (a single epoch  would scale as O ( NCK ) ).
Revised: We do not compare to full softmax classiﬁcation, which would be unfeasible on the large data sets (see Table 1; a single epoch of optimizing the full softmax loss would scale as O ( NCK ) ).

Original: NCE (Gutmann & Hyvärinen, 2010) is sometimes used as a synonym for negative sampling in the literature, but the original proposal of NCE is more general and allows for a nonuniform base distribution.
Revised: NCE (Gutmann & Hyvärinen, 2010) is sometimes used as a synonym for negative sampling in the literature, but the original proposal of NCE is more general and allows for a nonuniform base distribution.

Original: We use our trained auxiliary model (Section 3) for the base distribution of NCE.
Revised: We use our trained auxiliary model (Section 3) for the base distribution of NCE.

Original: Compared to our proposed method, NCE uses the base distribution only during training and not for predictions.
Revised: Compared to our proposed method, NCE uses the base distribution only during training and not for predictions.

Original: Therefore, NCE has to re-learn everything that is already captured by the base distribution.
Revised: Therefore, NCE has to re-learn everything that is already captured by the base distribution.

Original: This is less of an issue in the original setup for which NCE was proposed, namely unsupervised density estimation over a continuous space.
Revised: This is less of an issue in the original setup for which NCE was proposed, namely unsupervised density estimation over a continuous space.

Original: By contrast, training a supervised classiﬁer effectively means training a separate model for each label y ∈ Y , which is expensive if Y is large.
Revised: By contrast, training a supervised classiﬁer effectively means training a separate model for each label y ∈ Y , which is expensive if Y is large.

Original: Thus, having to re-learn what the base distribution already captures is potentially wasteful.
Revised: Thus, having to re-learn what the base distribution already captures is potentially wasteful.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: We tuned the hyperparameters for each method individually using the validation set.
Revised: We tuned the hyperparameters for each method individually using the validation set.

Original: Table 1 shows the resulting hyperparameters.
Revised: Table 1 shows the resulting hyperparameters.

Original: For the proposed method and baselines (i)-(iii) we used an Adagrad optimizer (Duchi et al., 2011) and considered learning rates
Revised: For the proposed method and baselines (i)-(iii) we used an Adagrad optimizer (Duchi et al., 2011) and considered learning rates

Original: ρ ∈ { 0 . 0003 , 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 } and regularizer strengths (see Eq. 6) λ ∈ { 10 − 5 , 3 × 10 − 5 , . . . , 0 . 03 } .
Revised: ρ ∈ { 0 . 0003 , 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 } and regularizer strengths (see Eq. 6) λ ∈ { 10 − 5 , 3 × 10 − 5 , . . . , 0 . 03 } .

Original: For ‘Augment and Reduce’ and ‘One vs. Each’ we used the implementation published by the authors (Ruiz), and tuned the learning rate ρ and prior variance σ 20 .
Revised: For ‘Augment and Reduce’ and ‘One vs. Each’ we used the implementation published by the authors (Ruiz), and tuned the learning rate ρ and prior variance σ 20 .

Original: For the auxiliary model, we used  feature dimension of k = 16 and  λ n = 0 .
Revised: For the auxiliary model, we used a feature dimension of k = 16 and regularizer strength λ n = 0 .

Original: 1 for both data sets.
Revised: 1 for both data sets.

Original: Results.
Revised: Results.

Original: Figure 1 shows our results on the Wikipedia-500K data set (left two plots) and the Amazon670K data set (right two plots).
Revised: Figure 1 shows our results on the Wikipedia-500K data set (left two plots) and the Amazon670K data set (right two plots).

Original: For each data set, we plot the learning curves for the predictive log likelihood per test data point (ﬁrst and third plot) and the predictive accuracy, i.e., the percentage of test points for which the model predicts the correct label (second and fourth plot).
Revised: For each data set, we plot the learning curves for the predictive log likelihood per test data point (ﬁrst and third plot) and the predictive accuracy, i.e., the percentage of test points for which the model predicts the correct label (second and fourth plot).

Original: The green curve in each plot shows our proposed adversarial negative sampling methods.
Revised: The green curve in each plot shows our proposed adversarial negative sampling methods.

Original: The curves for our method and for NCE (orange) start slightly shifted to the right to account for the time to ﬁt the auxiliary model.
Revised: The curves for our method and for NCE (orange) start slightly shifted to the right to account for the time to ﬁt the auxiliary model.

Original: Our main observation is that the proposed method converges orders of magnitude faster and reaches better accuracies (second and third plot in Figure 1) than all baselines.
Revised: Our main observation is that the proposed method converges orders of magnitude faster and reaches better accuracies (second and third plot in Figure 1) than all baselines.

Original: On the (smaller) Amazon-670K data set, standard uniform and frequency based negative sampling reach a slightly higher predictive log likelihood, but our method performs considerably better in terms of predictive accuracy.
Revised: On the (smaller) Amazon-670K data set, standard uniform and frequency based negative sampling reach a slightly higher predictive log likelihood, but our method performs considerably better in terms of predictive accuracy.

Original: This is consistent with the intuition that even with a poorly chosen noise distribution, negative sampling can quickly learn to assign high probabilities to a correct label y for a given input x .
Revised: This is consistent with the intuition that the predictive accuracy is very sensitive to the precise scores that the model assigns to the highest ranked labels y for a test input x .

Original: But it will typically also assign high probabilities to a few incorrect labels ˜ y .
Revised: Even a small change in the scores of the top ranked labels can affect which label is ranked highest and therefore predicted. On the other hand,

Original: Whether ˜ y ’s predictive probability is slightly lower or slightly higher than y ’s hardly affects the predictive likelihood, but it makes the entire difference between a correct or a wrong prediction.
Revised: [Figure] the predictive accuracy is independent of the precise scores for low ranked labels.

Original: To get the precise rankings among the most likely predictions right, the training procedure needs to compare the correct labels to likely competitors. This is precisely what adversarial negative samples do.
Revised: With adversarial negative sampling, the training procedure focuses on getting the scores of the highest ranked labels right, thus improving in particular the predictive accuracy.

Original: The improvements in predictive accuracy are even more pronounced in the (larger) Wikipedia-500K dat set (second plot in Figure 1).
Revised: The improvements in predictive accuracy are even more pronounced in the (larger) Wikipedia-500K dat set (second plot in Figure 1).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Efﬁcient Evaluation of the Softmax Loss Function.
Revised: Efﬁcient Evaluation of the Softmax Loss Function.

Original: Methods to speed up evaluation of Eq.include augmenting the model by adding auxiliary latent variables that can be marginalized over analytically (Galy-Fajou et al., 2019; Wenzel et al., 2019; Ruiz et al., 2018; Titsias, 2016).
Revised: Methods to speed up evaluation of Eq.include augmenting the model by adding auxiliary latent variables that can be marginalized over analytically (Galy-Fajou et al., 2019; Wenzel et al., 2019; Ruiz et al., 2018; Titsias, 2016).

Original: More closely related to our work are methods based on negative sampling (Mnih & Hinton, 2009; Mikolov et al., 2013) and noise contrastive estimation (Gutmann & Hyvärinen, 2010).
Revised: More closely related to our work are methods based on negative sampling (Mnih & Hinton, 2009; Mikolov et al., 2013) and noise contrastive estimation (Gutmann & Hyvärinen, 2010).

Original: Generalizations of negative sampling to non-uniform noise distributions have been proposed, e.g., in (Zhang &
Revised: Generalizations of negative sampling to non-uniform noise distributions have been proposed, e.g., in (Zhang &

Original: Zweigenbaum, 2018; Chen et al., 2018; Wang et al., 2014; Gutmann & Hyvärinen, 2010).
Revised: Zweigenbaum, 2018; Chen et al., 2018; Wang et al., 2014; Gutmann & Hyvärinen, 2010).

Original: Our method differs from these proposals by drawing the negative samples from a conditional distribution that takes the input feature into account, and by requiring the model to learn only correlations that are not already captured by the noise distribution.
Revised: Our method differs from these proposals by drawing the negative samples from a conditional distribution that takes the input feature into account, and by requiring the model to learn only correlations that are not already captured by the noise distribution.

Original: We further derive the optimal distribution for negative samples, and we propose an efﬁcient way to approximate it via an auxiliary model.
Revised: We further derive the optimal distribution for negative samples, and we propose an efﬁcient way to approximate it via an auxiliary model.

Original: Adversarial training (Miyato et al., 2017) is a popular method for training deep generative models (Tu, 2007;
Revised: Adversarial training (Miyato et al., 2017) is a popular method for training deep generative models (Tu, 2007;

Original: Goodfellow et al., 2014).
Revised: Goodfellow et al., 2014).

Original: By contrast, our method trains a discriminative model over a discrete set of labels (see also our comparison to GANs at the end of Section 2.2).
Revised: By contrast, our method trains a discriminative model over a discrete set of labels (see also our comparison to GANs at the end of Section 2.2).

Revised: A different sampling-based approximation of softmax classiﬁcation is ‘sampled softmax’ (Bengio et al., 2003).

Revised: It directly approximates the sum over classes y (cid:48) in the loss (Eq. 1) by sampling, which is biased even for a uniform sampling distribution.

Revised: A nonuniform sampling distribution can remove or reduce the bias (Bengio & Senécal, 2008; Blanc & Rendle, 2018; Rawat et al., 2019).

Revised: By contrast, our method uses negative sampling, and it uses a nonuniform distribution to reduce the gradient variance.

Original: Decision Trees.
Revised: Decision Trees.

Original: Decision trees (Somvanshi & Chavan, 2016) are popular in the extreme classiﬁcation literature (Agrawal et al., 2013;
Revised: Decision trees (Somvanshi & Chavan, 2016) are popular in the extreme classiﬁcation literature (Agrawal et al., 2013;

Original: Jain et al., 2016; Prabhu & Varma, 2014;
Revised: Jain et al., 2016; Prabhu & Varma, 2014;

Original: Siblini et al., 2018;
Revised: Siblini et al., 2018;

Original: Weston et al., 2013; Bhatia et al., 2015; Jasinska et al., 2016).
Revised: Weston et al., 2013; Bhatia et al., 2015; Jasinska et al., 2016).

Original: Our proposed method employs a probabilistic decision tree that is similar to Hierarchical Softmax (Morin & Bengio, 2005; Mikolov et al., 2013).
Revised: Our proposed method employs a probabilistic decision tree that is similar to Hierarchical Softmax (Morin & Bengio, 2005; Mikolov et al., 2013).

Original: While decision trees allow for efﬁcient training and sampling in O (log C ) time, their hierarchical architecture imposes a structural bias.
Revised: While decision trees allow for efﬁcient training and sampling in O (log C ) time, their hierarchical architecture imposes a structural bias.

Original: Our proposed method trains a more expressive model without such a structural bias on top of the decision tree to correct for any structural bias.
Revised: Our proposed method trains a more expressive model without such a structural bias on top of the decision tree to correct for any structural bias.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We proposed a simple method to train a classiﬁer over a large set of labels.
Revised: We proposed a simple method to train a classiﬁer over a large set of labels.

Original: Our method is based on a scalable approximation to the softmax loss function via a generalized form of negative sampling.
Revised: Our method is based on a scalable approximation to the softmax loss function via a generalized form of negative sampling.

Original: By generating adversarial negative samples from an auxiliary model, we proved that we maximize the signal-to-noise ratio of the stochastic gradient estimate.
Revised: By generating adversarial negative samples from an auxiliary model, we proved that we maximize the signal-to-noise ratio of the stochastic gradient estimate.

Original: We further show that, while the auxiliary model introduces a bias, we can remove the bias at test time.
Revised: We further show that, while the auxiliary model introduces a bias, we can remove the bias at test time.

Original: We believe that due to its simplicity, our method can be widely used, and we publish the code of both the main and the auxiliary model.
Revised: We believe that due to its simplicity, our method can be widely used, and we publish the code of both the main and the auxiliary model.

Original: A PPENDIX A. D ETAILS OF THE P ROOF OF T HEOREM
Revised: A PPENDIX A. D ETAILS OF THE P ROOF OF T HEOREM

Original: In the nonparametric limit, the score functions ξ y ( x, φ ) are so ﬂexible that they can take arbitrary values for all x in the data set and all y ∈ Y .
Revised: In the nonparametric limit, the score functions ξ y ( x, φ ) are so ﬂexible that they can take arbitrary values for all x in the data set and all y ∈ Y .

Original: Taking advantage of the invariance of ¯ η under reparameterization, we parameterize the model directly by its scores.
Revised: Taking advantage of the invariance of ¯ η under reparameterization, we parameterize the model directly by its scores.

Original: We use the shorthand ξ x,y ≡ ξ y ( x, φ ) , and we denote the collection of all scores over all x and y ∈ Y by boldface ξ ≡ ( ξ x,y ) x,y .
Revised: We use the shorthand ξ x,y ≡ ξ y ( x, φ ) , and we denote the collection of all scores over all x and y ∈ Y by boldface ξ ≡ ( ξ x,y ) x,y .

Original: Hessian.
Revised: Hessian.

Original: Eq.
Revised: Eq.

Original: 2 deﬁnes the loss (cid:96) neg.sampl.
Revised: 2 deﬁnes the loss (cid:96) neg.sampl.

Original: as a stochastic function.
Revised: as a stochastic function.

Original: SGD minimizes its expectation,
Revised: SGD minimizes its expectation,

Original: [Equation]
Revised: [Equation]

Original: where the sum over x runs over all feature vectors in the training set.
Revised: where the sum over x runs over all feature vectors in the training set.

Original: We obtain the gradient
Revised: We obtain the gradient

Original: [Equation]
Revised: [Equation]

Original: where we used the relation ∇ z  σ ( z )
Revised: where we used the relation ∇ z log σ ( z )

Original: =
Revised: =

Original: σ ( − z ) .
Revised: σ ( − z ) .

Original: The gradient is a vector whose components span all combinations of x and y .
Revised: The gradient is a vector whose components span all combinations of x and y .

Original: The Hessian matrix H (cid:96) contains the derivatives of each gradient component g x,y by each coordinate ξ ˜ x, ˜ y .
Revised: The Hessian matrix H (cid:96) contains the derivatives of each gradient component g x,y by each coordinate ξ ˜ x, ˜ y .

Original: Since g x,y in Eq.
Revised: Since g x,y in Eq.

Original: A2 depends only on the single coordinate ξ x,y , only the diagonal parts of the Hessian, i.e., the pars with x = ˜ x and y =
Revised: A2 depends only on the single coordinate ξ x,y , only the diagonal parts of the Hessian are nonzero, i.e., the components with x = ˜ x and y =

Original: ˜
Revised: ˜

Original: y are nonzero.
Revised: y .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Using the identity ∇ z σ ( z )
Revised: Using the identity ∇ z σ ( z )

Original: = σ ( z ) σ ( − z )
Revised: = σ ( z ) σ ( − z )

Original: , we ﬁnd
Revised: , we ﬁnd

Original: [Equation]
Revised: [Equation]

Original: Since we evaluate the Hessian in the nonparametric limit at the minimum of the loss, the scores ξ x,y satisfy Eq.
Revised: Since we evaluate the Hessian in the nonparametric limit at the minimum of the loss, the scores ξ x,y satisfy Eq.

Original: 11, i.e.,
Revised: 11, i.e.,

Original: [Equation]
Revised: [Equation]

Original: This allows us to simplify Eq.
Revised: This allows us to simplify Eq.

Original: A4,
Revised: A4 by eliminating p D ,

Original: [Equation]
Revised: [Equation]

Original: Eqs.
Revised: Eqs.

Original: A3 and A6 together prove Eq.
Revised: A3 and A6 together prove Eq.

Original: 13 of the main text.
Revised: 13 of the main text.

Original: Noise Covariance Matrix.
Revised: Noise Covariance Matrix.

Original: SGD uses estimates ˆ (cid:96) of the loss function in Eq.
Revised: SGD uses estimates ˆ (cid:96) of the loss function in Eq.

Original: A1, obtained by drawing a positive sample ( x, y ) ∼ D and a label for the negative sample y (cid:48) ∼ p n ( y [Equation]
Revised: A1, obtained by drawing a positive sample ( x, y ) ∼ D and a label for the negative sample y (cid:48) ∼ p n ( y (cid:48)

Original: | x ) , thus
Revised: | x ) , thus

Original: [Equation]
Revised: [Equation]

Original: where the factor of N ≡ |D| is because the sum over x in Eq.
Revised: where the factor of N ≡ |D| is because the sum over x in Eq.

Original: A1 scales proportionally to the size of the data set D (in practice one typically normalizes the loss function by N without affecting the signal to noise ratio).
Revised: A1 scales proportionally to the size of the data set D (in practice one typically normalizes the loss function by N without affecting the signal to noise ratio).

Original: One uses ˆ (cid:96) to obtain unbiased gradient estimates ˆ g .
Revised: One uses ˆ (cid:96) to obtain unbiased gradient estimates ˆ g .

Original: We introduce new symbols ˜ x and ˜ y for the components
Revised: We introduce new symbols ˜ x and ˜ y for the components

Original: ˆ g ˜ x, ˜
Revised: ˆ g ˜ x, ˜

Original: y of the gradient estimate to avoid confusion with the x and y drawn from the data set and the y (cid:48) drawn from the noise distribution in Eq.
Revised: y of the gradient estimate to avoid confusion with the x and y drawn from the data set and the y (cid:48) drawn from the noise distribution in Eq.

Original: A7 above.
Revised: A7 above.

Original: Since the scores are independent variables in the nonparametric limit, the derivative ∇ ξ ˜ x, ˜ y ξ x,y is one if ˜ x = x and ˜ y
Revised: Since the scores are independent variables in the nonparametric limit, the derivative ∇ ξ ˜ x, ˜ y ξ x,y is one if ˜ x = x and ˜ y

Original: = y , and zero otherwise.
Revised: = y , and zero otherwise.

Original: We denote this by indicator functions 1 ˜ x = x and 1 ˜ y
Revised: We denote this by indicator functions 1 ˜ x = x and 1 ˜ y

Original: =
Revised: =

Original: y .
Revised: y .

Original: Thus, we obtain
Revised: Thus, we obtain

Original: [Equation]
Revised: [Equation]

Original: We evaluate the covariance matrix of ˆ g at the minimum of the loss function.
Revised: We evaluate the covariance matrix of ˆ g at the minimum of the loss function.

Original: Here, E [ˆ g ] ≡ g = 0 , and thus Cov [ˆ g, ˆ g ] ≡ E
Revised: Here, E [ˆ g ] ≡ g = 0 , and thus Cov [ˆ g, ˆ g ] ≡ E

Original: [ˆ g ˆ g (cid:62) ] − E [ˆ g ] E
Revised: [ˆ g ˆ g (cid:62) ] − E [ˆ g ] E

Original: [ˆ g (cid:62) ] simpliﬁes to E [ˆ g ˆ g (cid:62) ] .
Revised: [ˆ g (cid:62) ] simpliﬁes to E [ˆ g ˆ g (cid:62) ] .

Original: Introducing yet another pair of indices ˜˜ x and ˜˜ y to distinguish the two factors of ˆ g , we denote the components of the covariance matrix as
Revised: Introducing yet another pair of indices ˜˜ x and ˜˜ y to distinguish the two factors of ˆ g , we denote the components of the covariance matrix as

Original: [Equation]
Revised: [Equation]

Original: Here, the expectation is over p D ( x, y ) p n ( y (cid:48) | x ) =
Revised: Here, the expectation is over p D ( x, y ) p n ( y (cid:48) | x ) =

Original: p D ( x ) p D ( y | x ) p n ( y (cid:48)
Revised: p D ( x ) p D ( y | x ) p n ( y (cid:48)

Original: | x ) .
Revised: | x ) .

Original: We start with the evaluation of the expectation over x , using E x ∼ p D [ · ] = 1 N (cid:80) x [ · ] where the sum runs over all x in the
Revised: We start with the evaluation of the expectation over x , using E x ∼ p D [ · ] = 1 N (cid:80) x [ · ] where the sum runs over all x in the

Original: A data set.
Revised: A data set.

Original: If x (cid:54) =
Revised: If x (cid:54) =

Original: ˜ x or x (cid:54) =
Revised: ˜ x or x (cid:54) =

Original: ˜˜ x , then either one of the two gradient estimates ˆ g in the expectation on the right-hand side of Eq.
Revised: ˜˜ x , then either one of the two gradient estimates ˆ g in the expectation on the right-hand side of Eq.

Original: A9 vanishes.
Revised: A9 vanishes.

Original: Therefore, only terms with x = ˜ x
Revised: Therefore, only terms with x = ˜ x

Original: = ˜˜ x contribute, and the covariance matrix is block diagonal in x as claimed in Eq.
Revised: = ˜˜ x contribute, and the covariance matrix is block diagonal in x as claimed in Eq.

Original: 14 of the main text.
Revised: 14 of the main text.

Original: The blocks C x of the block diagonal matrix have entries
Revised: The blocks C x of the block diagonal matrix have entries

Original: [Equation]
Revised: [Equation]

Original: where we ﬁnd for the product ˆ g x, ˜
Revised: where we ﬁnd for the product ˆ g x, ˜

Original: y
Revised: y

Original: ˆ g x, ˜˜ y by  multiplying out the terms,
Revised: ˆ g x, ˜˜ y by inserting Eq. A8 and multiplying out the terms,

Original: [Equation]
Revised: [Equation]

Original: Taking the expectation in Eq.
Revised: Taking the expectation in Eq.

Original: A10 leads to the following substitutions:
Revised: A10 leads to the following substitutions:

Original: [Equation]
Revised: [Equation]

Original: Thus, we ﬁnd,
Revised: Thus, we ﬁnd,

Original: [Equation]
Revised: [Equation]

Original: Using Eq.
Revised: Using Eq.

Revised: A

Revised: [Equation]

Original: A5, we can  eliminate p D ,
Revised: we can again eliminate p D ,

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: A14 is the component-wise explicit form of Eq.
Revised: A14 is the component-wise explicit form of Eq.

Original: 14 of the main text.
Revised: 14 of the main text.

Original: A
Revised: A