Original: A
Revised: A

Original: T RULY
Revised: T RULY

Original: C ONSTANT - TIME
Revised: C ONSTANT - TIME

Original: D ISTRIBUTION - AWARE
Revised: D ISTRIBUTION - AWARE

Original: N EGATIVE S
Revised: N EGATIVE S

Original: AMPLING
Revised: AMPLING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Softmax classiﬁers with a very large number of classes naturally occur in many applications such as natural language processing and information retrieval.
Revised: Softmax classiﬁers with a very large number of classes naturally occur in many applications such as natural language processing and information retrieval.

Original: The calculation of full-softmax is very expensive from the computational and energy perspective.
Revised: The calculation of full-softmax is very expensive from the computational and energy perspective.

Original: There have been a variety of sampling approaches to overcome this challenge, popularly known as negative sampling (NS).
Revised: There have been a variety of sampling approaches to overcome this challenge, popularly known as negative sampling (NS).

Original: Ideally, NS should sample negative classes from a distribution that is dependent on the input data, the current parameters, and the correct positive class.
Revised: Ideally, NS should sample negative classes from a distribution that is dependent on the input data, the current parameters, and the correct positive class.

Original: Unfortunately, due to the dynamically updated parameters and data samples, there does not exist any sampling scheme that is truly adaptive and also samples the negative classes in constant time every iteration.
Revised: Unfortunately, due to the dynamically updated parameters and data samples, there does not exist any sampling scheme that is truly adaptive and also samples the negative classes in constant time every iteration.

Original: Therefore, alternative heuristics like random sampling, static frequencybased sampling, or learning-based biased sampling, which primarily trade either the sampling cost or the adaptivity of samples per iteration, are adopted.
Revised: Therefore, alternative heuristics like random sampling, static frequencybased sampling, or learning-based biased sampling, which primarily trade either the sampling cost or the adaptivity of samples per iteration, are adopted.

Original: In this paper, we show a class of distribution where the sampling scheme is truly adaptive and provably generates negative samples in constant time.
Revised: In this paper, we show a class of distribution where the sampling scheme is truly adaptive and provably generates negative samples in constant time.

Original: Our implementation in C++ on commodity CPU is signiﬁcantly faster, in terms of wall clock time, compared to the most optimized TensorFlow implementations of standard softmax or other sampling approaches on modern GPUs (V100s).
Revised: Our implementation in C++ on commodity CPU is signiﬁcantly faster, in terms of wall clock time, compared to the most optimized TensorFlow implementations of standard softmax or other sampling approaches on modern GPUs (V100s).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Networks (NN) have successfully pushed the boundaries of many application tasks, such as image or text classiﬁcation (Wang et al., 2017; Yao et al., 2019), speech recognition (Dong et al., 2018) and recommendation systems (Zhang et al., 2015; Medini et al., 2019).
Revised: Neural Networks (NN) have successfully pushed the boundaries of many application tasks, such as image or text classiﬁcation (Wang et al., 2017; Yao et al., 2019), speech recognition (Dong et al., 2018) and recommendation systems (Zhang et al., 2015; Medini et al., 2019).

Original: Many hard AI problems are currently modeled as massive multiclass or multilabel problems leading to a drastic improvement over prior work.
Revised: Many hard AI problems are currently modeled as massive multiclass or multilabel problems leading to a drastic improvement over prior work.

Original: For example, popular NLP models predicts the best word, given the full context observed so far.
Revised: For example, popular NLP models predicts the best word, given the full context observed so far.

Original: Such models are becoming the state-of-the-art.
Revised: Such models are becoming the state-of-the-art.

Original: Recommendation systems and related
Revised: Recommendation systems and related

Original: Information Retrieval (IR) problems are classical examples of machine learning with outrageously large outputs (Medini et al., 2019; Jain et al., 2019).
Revised: Information Retrieval (IR) problems are classical examples of machine learning with outrageously large outputs (Medini et al., 2019; Jain et al., 2019).

Original: In IR, given the user query, the task is to predict few relevant documents (or products) from among hundreds of millions possible documents, a typical machine learning problem with massive output space.
Revised: In IR, given the user query, the task is to predict few relevant documents (or products) from among hundreds of millions possible documents, a typical machine learning problem with massive output space.

Original: Owing to the signiﬁcance of the problem, machine learning with large output space or alternatively also known as extreme classiﬁcation is a ﬁeld in itself (Bengio et al., 2019).
Revised: Owing to the signiﬁcance of the problem, machine learning with large output space or alternatively also known as extreme classiﬁcation is a ﬁeld in itself (Bengio et al., 2019).

Original: A large number of classes naturally brings a new set of computational and memory challenge.
Revised: A large number of classes naturally brings a new set of computational and memory challenge.

Original: Fortunately, with access to our powerful Graphic Processing Unit (GPU) (Owens et al., 2008), training processes of large models have been accelerated heavily.
Revised: Fortunately, with access to our powerful Graphic Processing Unit (GPU) (Owens et al., 2008), training processes of large models have been accelerated heavily.

Original: That is because GPUs have a unique advantage for matrix multiplication, which usually requires a cubic time algebraic operation ( O ( N 3 ) ) and is the major and costly building block of NN computations.
Revised: That is because GPUs have a unique advantage for matrix multiplication, which usually requires a cubic time algebraic operation ( O ( N 3 ) ) and is the major and costly building block of NN computations.

Original: However, the number of concurrent operations required in large matrix multiplications for classiﬁcation with extensive number of classes has reached a limit for further speedups even using GPUs.
Revised: However, the number of concurrent operations required in large matrix multiplications for classiﬁcation with extensive number of classes has reached a limit for further speedups even using GPUs.

Original: 1. N EGATIVE S AMPLING
Revised: 1. N EGATIVE S AMPLING

Original: The common approach to address this challenge is known as negative sampling (Pennington et al., 2014; Jean et al., 2014; Rawat et al., 2019; Mikolov et al., 2013b).
Revised: The common approach to address this challenge is known as negative sampling (Pennington et al., 2014; Jean et al., 2014; Rawat et al., 2019; Mikolov et al., 2013b).

Original: In Negative Sampling, we only sample a small subset of classes for each input and compute the softmax and cross-entropy functions.
Revised: In Negative Sampling, we only sample a small subset of classes for each input and compute the softmax and cross-entropy function.

Original: This subset usually includes the positive (true) and a small set of negative (false) classes.
Revised: This subset usually includes the positive (true) and a small set of negative (false) classes.

Original: Negative sampling scales down the computations in the most cumbersome last layer, thereby making training efﬁcient.
Revised: Negative sampling scales down the computations in the most cumbersome last layer, thereby making training efﬁcient.

Original: However, approximating full-softmax with small sub-sample results in poor convergence if the negative samples are not chosen appropriately.
Revised: However, approximating full-softmax with small sub-sample results in poor convergence if the negative samples are not chosen appropriately.

Original: For instance, let us take the example of a recommendation system (predicting products relevant to a query) with a large number of products.
Revised: For instance, let us take the example of a recommendation system (predicting products relevant to a query) with a large number of products.

Original: If the input query is ‘Nike Running Shoes’, the true loss concentrates on the speciﬁc small number of confusing (’hard’)
Revised: If the input query is ‘Nike Running Shoes’, the true loss concentrates on the speciﬁc small number of confusing (’hard’)

Original: negative classes like ‘Adidas Running Shoes’.
Revised: negative classes like ‘Adidas Running Shoes’.

Original: Since the number of classes is huge, random sampling is unlikely to identify this hard negative class.
Revised: Since the number of classes is huge, random sampling is unlikely to identify this hard negative class.

Original: Other heuristics like frequent class sampling as negative samples are also unlikely to ﬁnd these hard negatives most of the time.
Revised: Other heuristics like frequent class sampling as negative samples are also unlikely to ﬁnd these hard negatives most of the time.

Original: Clearly, without discriminating between closely related negative samples, the classiﬁer cannot achieve good accuracy.
Revised: Clearly, without discriminating between closely related negative samples, the classiﬁer cannot achieve good accuracy.

Original: Our experiments on recommendations datasets clearly indicate this sub-optimality of current negative sampling heuristics.
Revised: Our experiments on recommendations datasets clearly indicate this sub-optimality of current negative sampling heuristics.

Original: If there exists a way to sample the subset of confusing classes from the skewed distribution, the training progress would be largely accelerated.
Revised: If there exists a way to sample the subset of confusing classes from the skewed distribution, the training progress would be largely accelerated.

Original: However, as evident from the example, such ground-truth distribution depends on the input sample and current model parameters.
Revised: However, as evident from the example, such ground-truth distribution depends on the input sample and current model parameters.

Original: Moreover, this distribution varies signiﬁcantly as training progresses.
Revised: Moreover, this distribution varies signiﬁcantly as training progresses.

Original: Consider the same query ’Nike Running Shoes’, initially, when the network has not learned anything and has random weights, all classes are equally confusing.
Revised: Consider the same query ’Nike Running Shoes’, initially, when the network has not learned anything and has random weights, all classes are equally confusing.

Original: Thus, uniform sampling is optimal initially as the network has just started to learn.
Revised: Thus, uniform sampling is optimal initially as the network has just started to learn.

Original: As the training progresses, the network’s belief starts getting more concentrated on a few classes; at this time, a negative sample of say ’baby toys’ is not at all useful because the network has already learned to tell them apart.
Revised: As the training progresses, the network’s belief starts getting more concentrated on a few classes; at this time, a negative sample of say ’baby toys’ is not at all useful because the network has already learned to tell them apart.

Original: The sampling distribution keeps changing, often drastically, as the training progresses.
Revised: The sampling distribution keeps changing, often drastically, as the training progresses.

Original: To the best of our knowledge, there does not exist any statistical sampling scheme and implementation for adaptive Negative Sampling, where the cost of maintaining and updating the distribution, per iteration, is O (1) (independent of the number of classes).
Revised: To the best of our knowledge, there does not exist any statistical sampling scheme and implementation for adaptive Negative Sampling, where the cost of maintaining and updating the distribution, per iteration, is O (1) (independent of the number of classes).

Original: This is because the input, current true class, and parameters update all the sampling weights in every iteration.
Revised: This is because the input, current true class, and parameters update all the sampling weights in every iteration.

Original: It is widely assumed that there is no such sampling scheme, and hence several heuristic alternatives are proposed.
Revised: It is widely assumed that there is no such sampling scheme, and hence several heuristic alternatives are proposed.

Original: The ﬁrst set of alternatives use a static distribution.
Revised: The ﬁrst set of alternatives use a static distribution.

Original: The most popular ones, implemented in TensorFlow, assume a static distribution such as static distribution based on the classes’ frequency.
Revised: The most popular ones, implemented in TensorFlow, assume a static distribution such as the distribution based on the frequency of classes.

Original: Uniform sampling is another popular choice.
Revised: Uniform sampling is another popular choice.

Original: Learning-based alternatives are also proposed (Bamler & Mandt, 2020), where a machine learning generator predicts (or generates) the negative samples.
Revised: Learning-based alternatives are also proposed (Bamler & Mandt, 2020), where a machine learning generator predicts (or generates) the negative samples.

Original: The sampler is solving the same hard problem, prediction over a large number of classes, as a sub-routine.
Revised: The sampler is solving the same hard problem, prediction over a large number of classes, as a sub-routine.

Original: Most importantly, since the sampling distribution for the same data point shifts drastically throughout training, ML models are likely to suffer.
Revised: Most importantly, since the sampling distribution for the same data point shifts drastically throughout training, ML models are likely to suffer.

Original: Negative sampling alternatives try to balance the sampling cost with quality.
Revised: Negative sampling alternatives try to balance the sampling cost with quality.

Original: So far, negative sampling methods, other than the ones based on static sampling, have failed to demonstrate any training time improvements over the optimized full softmax implementation, such as TensorFlow packages, over GPUs.
Revised: So far, negative sampling methods, other than the ones based on static sampling, have failed to demonstrate any training time improvements over the optimized full softmax implementation over GPUs.

Original: Static sampling strategies are known to be fast but lead to poor accuracy.
Revised: Static sampling strategies are known to be fast but lead to poor accuracy.

Original: With current strategies, the cost of improving the quality with current alternatives does not seem worth it over the GPU acceleration of softmax.
Revised: With current strategies, the cost of improving the quality with current alternatives does not seem worth it over the GPU acceleration of softmax.

Original: In this paper, we change this.
Revised: In this paper, we change this.

Original: Our work provides a truly constant time adaptive sampling scheme utilizing the recent advances in Locality Sensitive Sampling (Charikar & Siminelakis, 2017; Spring & Shrivastava, 2017a).
Revised: Our work provides a truly constant time adaptive sampling scheme utilizing the recent advances in Locality Sensitive Sampling (Charikar & Siminelakis, 2017; Spring & Shrivastava, 2017a).

Original: More impressively, we provide an efﬁcient implementation of our proposal on CPU, which outperforms TensorFlow’s implementation of softmax and other negative sampling strategies on some of the best available GPUs (V100) in terms of wall-clock training time.
Revised: More impressively, we provide an efﬁcient implementation of our proposal on CPU, which outperforms TensorFlow’s implementation of softmax and other negative sampling strategies on some of the best available GPUs (V100) in terms of wall-clock training time.

Original: Summary of Contributions:
Revised: Summary of Contributions:

Original: 1) We propose two efﬁcient schemes for sampling ‘hard’ negatives where the negative sampling distribution provably adapts to changing parameters and the data instance.
Revised: 1) We propose two efﬁcient schemes for sampling ‘hard’ negatives where the negative sampling distribution provably adapts to changing parameters and the data instance.

Original: Furthermore, the sampling cost is provably constant (independent of the number of classes).
Revised: Furthermore, the sampling cost is provably constant (independent of the number of classes).

Original: We show that our technique is not only provable but also practical.
Revised: We show that our technique is not only provably adaptive but also practical.

Original: We provide an efﬁcient CPU implementation, in C++, of our negative sampling approach.
Revised: We provide an efﬁcient CPU implementation, in C++, of our negative sampling approach.

Original: We demonstrate the effectiveness of a truly constant time negative sampler by showing that our implementation signiﬁcantly outperforms standard TensorFlow on V100 GPU in-wall clock speed, while retaining the accuracy.
Revised: We demonstrate the effectiveness of a truly constant time negative sampler by showing that our implementation signiﬁcantly outperforms standard TensorFlow on V100 GPU in-wall clock speed, while retaining the accuracy.

Original: 3) We provide a rigorous evaluation of our proposal with its efﬁcient implementation against full softmax and popular approximations like sampled softmax, frequency-based sampled softmax, top-K activation softmax, differentiation softmax (D-Softmax), and Noise Contrastive Estimation(NCE).
Revised: 3) We provide a rigorous evaluation of our proposal with its efﬁcient implementation against full softmax and popular approximations like sampled softmax, frequency-based sampled softmax, top-K activation softmax, differentiation softmax (D-Softmax), and Noise Contrastive Estimation(NCE).

Original: We report the time-wise and iteration-wise precision on large recommendation datasets like Amazon670K (Kaggle), WikiLSH-325K and popular natural language processing dataset Text8 corpus.
Revised: We report the time-wise and iteration-wise precision on large recommendation datasets like Amazon670K (Kaggle), WikiLSH-325K and popular natural language processing dataset Text8 corpus.

Original: 1. LSH B ASED H ASH T ABLES
Revised: 1. LSH B ASED H ASH T ABLES

Original: In this section, we brieﬂy describe the recent development of using locality sensitive hashing for sampling and estimation (Chen et al., 2019b; Spring & Shrivastava, 2017a; Charikar & Siminelakis, 2017; Spring & Shrivastava, 2017b).
Revised: In this section, we brieﬂy describe the recent development of using locality sensitive hashing for sampling and estimation (Chen et al., 2019b; Spring & Shrivastava, 2017a; Charikar & Siminelakis, 2017; Spring & Shrivastava, 2017b).

Original: Locality Sensitive Hashing (Indyk & Motwani, 1998; Indyk & Woodruff, 2006) is a widely used paradigm for large scale similarity search and nearest neighbor search.
Revised: Locality Sensitive Hashing (Indyk & Motwani, 1998; Indyk & Woodruff, 2006) is a widely used paradigm for large scale similarity search and nearest neighbor search.

Original: LSH is a family of hash functions with a unique property that vectors ‘close’ wrt some distance metric are more likely to have the same hash code as opposed to vectors that are ‘far’ from each other.
Revised: LSH is a family of hash functions with a unique property that vectors ‘close’ wrt some distance metric are more likely to have the same hash code as opposed to vectors that are ‘far’ from each other.

Original: Formally, one sufﬁcient condition for a hash family H to be a LSH family is that the collision probability Pr H ( h ( x ) =
Revised: Formally, one sufﬁcient condition for a hash family H to be a LSH family is that the collision probability Pr H ( h ( x ) =

Original: h ( y )) is a monotonically increasing function of the similarity:
Revised: h ( y )) is a monotonically increasing function of the similarity:

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: where f is a monotonically increasing function.
Revised: where f is a monotonically increasing function.

Original: The idea is to use the hash value of x , i.e., h ( x ) , to generate key of x in the hash table.
Revised: The idea is to use the hash value of x , i.e., h ( x ) , to generate key of x in the hash table.

Original: We ﬁrst initialize L hash tables by constructing a meta-LSH hash function using K independent hash functions for each of them.
Revised: We ﬁrst initialize L hash tables by constructing a meta-LSH hash function using K independent hash functions for each of them.

Original: For details, see (Andoni & Indyk, 2004).
Revised: For details, see (Andoni & Indyk, 2004).

Original: There are three major steps:
Revised: There are three major steps:

Original: Pre-processing Phase:
Revised: Pre-processing Phase:

Original: Given a dataset of size n , we ﬁrst insert all the data points into the hash tables using the meta-LSH formed by concatenating K independent LSH hash functions.
Revised: Given a dataset of size n , we ﬁrst insert all the data points into the hash tables using the meta-LSH formed by concatenating K independent LSH hash functions.

Original: We only store the index/pointer of the data point in the hash tables instead of the entire vector.
Revised: We only store the index/pointer of the data point in the hash tables instead of the entire vector.

Original: The cost of addition is K × L hash computations followed by L insertions in the buckets.
Revised: The cost of addition is K × L hash computations followed by L insertions in the buckets.

Original: Query Phase: During the query phase, we use the same meta-LSH hash to compute the hash codes for the query.
Revised: Query Phase: During the query phase, we use the same meta-LSH hash to compute the hash codes for the query.

Original: Then we probe the corresponding bucket of each table and retrieve samples from it.
Revised: Then we probe the corresponding bucket of each table and retrieve samples from it.

Original: The union of candidates from all hash tables constitute the samples for the particular query.
Revised: The union of candidates from all hash tables constitute the samples for the particular query.

Original: Update Phase: If an existing element in the database is updated, we can delete it from the hash table and re-add it.
Revised: Update Phase: If an existing element in the database is updated, we can delete it from the hash table and re-add it.

Original: The cost is equivalent to twice the insertion cost of an element which is 2 × K × L .
Revised: The cost is equivalent to twice the insertion cost of an element which is 2 × K × L .

Original: 1. A DAPTIVE S AMPLING VIEW OF LSH
Revised: 1. A DAPTIVE S AMPLING VIEW OF LSH

Original: [Figure]
Revised: [Figure]

Original: Denote p qx be the probability of retrieving x from the datasets, when queried with a given query q .
Revised: Denote p qx be the probability of retrieving x from the datasets, when queried with a given query q .

Original: In (Indyk & Motwani, 1998), it was shown that for ( K, L ) parametrized LSH algorithm the precise form of p qx = 1 − (1 − α K ) L , where α is the collision probability of query q and x under the given LSH function, i.e. α =
Revised: In (Indyk & Motwani, 1998), it was shown that for ( K, L ) parametrized LSH algorithm the precise form of p qx = 1 − (1 − α K ) L , where α is the collision probability of query q and x under the given LSH function, i.e. α =

Original: Pr H ( h ( x ) = h ( q )) .
Revised: Pr H ( h ( x ) = h ( q )) .

Original: p qx is monotonic in α which is further monotonic in the similarity between query q and the data element x .
Revised: p qx is monotonic in α which is further monotonic in the similarity between query q and the data element x .

Original: Note the similarity measure is dependent on the
Revised: Note the similarity measure is dependent on the

Original: LSH function in use.
Revised: LSH function in use.

Original: See (Spring & Shrivastava, 2017a) for details.
Revised: See (Spring & Shrivastava, 2017a) for details.

Original: Constant Time Sampling:
Revised: Constant Time Sampling:

Original: It should be noted that the cost of sampling is the cost of querying, which is only K × L .
Revised: It should be noted that the cost of sampling is the cost of querying, which is only K × L .

Original: This sampling cost is independent of the number of elements in the data.
Revised: This sampling cost is independent of the number of elements in the data.

Original: Clearly, the probability p qx is dependent on the query, and every element x in the data has a different sampling probability.
Revised: Clearly, the probability p qx is dependent on the query, and every element x in the data has a different sampling probability.

Original: Thus, even though our sampling scheme induces n different sampling probabilities every time the query q is changed, the sampling cost is independent of n , and in-fact is constant if K and L are small constants.
Revised: Thus, even though our sampling scheme induces n different sampling probabilities every time the query q is changed, the sampling cost is independent of n , and in-fact is constant if K and L are small constants.

Original: All this is assuming one O ( n ) time preprocessing.
Revised: All this is assuming one O ( n ) time preprocessing.

Original: This efﬁcient sampling view of LSH has been used in a wide range of applications, such as deep neural networks (Spring & Shrivastava, 2017b;
Revised: This efﬁcient sampling view of LSH has been used in a wide range of applications, such as deep neural networks (Spring & Shrivastava, 2017b;

Original: Chen et al., 2019a), kernel density estimation (Charikar &
Revised: Chen et al., 2019a), kernel density estimation (Charikar &

Original: Siminelakis, 2017), record linkage (Chen et al., 2018), and optimization (Chen et al., 2019c).
Revised: Siminelakis, 2017), record linkage (Chen et al., 2018), and optimization (Chen et al., 2019c).

Original: Recent advances in fast inner product search using asymmetric LSH has made it possible to sample large inner products (Shrivastava & Li, 2014).
Revised: Recent advances in fast inner product search using asymmetric LSH has made it possible to sample large inner products (Shrivastava & Li, 2014).

Original: Effectively, given a query q , it is possible to sample an element x from the database with probability proportional to a monotonic function of inner product f ( q T x ) .
Revised: Effectively, given a query q , it is possible to sample an element x from the database with probability proportional to a monotonic function of inner product f ( q T x ) .

Original: Here f is a monotonically increasing function.
Revised: Here f is a monotonically increasing function.

Original:  O UR P ROPOSAL : L OCALITY SENSITIVE N EGATIVE S AMPLING (LNS)
Revised:  O UR P ROPOSAL : L OCALITY SENSITIVE N EGATIVE S AMPLING (LNS)

Original: Notations: We will start by deﬁning a few vectors in the neural network setting illustrated in
Revised: Notations: We will start by deﬁning a few vectors in the neural network setting illustrated in

Original: Figure 2.
Revised: Figure 2.

Original: We are in large softmax settings.
Revised: We are in large softmax settings.

Original: Here, we will use N to denote the total number of classes.
Revised: Here, we will use N to denote the total number of classes.

Original: Deﬁne a vector w i ∈ R d ( class vectors ) to be the vector associated with class i in the last layer of the neural network.
Revised: Deﬁne a vector w i ∈ R d ( class vectors ) to be the vector associated with class i in the last layer of the neural network.

Original: We will use ( x, y ) to denote the current input sample to the neural network for which we want to generate negative samples.
Revised: We will use ( x, y ) to denote the current input sample to the neural network for which we want to generate negative samples.

Original: We will use E x ∈ R d ( ﬁnal input embedding ) to denote the vector of activation in the penultimate layer of the neural network when fed with input x .
Revised: We will use E x ∈ R d ( ﬁnal input embedding ) to denote the vector of activation in the penultimate layer of the neural network when fed with input x .

Original: We ﬁrst describe our sampling procedure, and later we argue why it is distribution aware and constant time.
Revised: We ﬁrst describe our sampling procedure, and later we argue why it is distribution aware and constant time.

Original: Our approach, just like the LSH algorithm, has three phases.
Revised: Our approach, just like the LSH algorithm, has three phases.

Original: The ﬁrst phase is a one time costly ( O ( N ) ) prepossessing stage.
Revised: The ﬁrst phase is a one time costly ( O ( N ) ) prepossessing stage.

Original: The other two phases, sampling and update phase, are performed in each iteration, and both of them are constant-time operation independent of N .
Revised: The other two phases, sampling and update phase, are performed in each iteration, and both of them are constant-time operation independent of N .

Original: One time Preprocessing Phase during Initialization: We start with randomly initializing the neural network parameters.
Revised: One time Preprocessing Phase during Initialization: We start with randomly initializing the neural network parameters.

Original: This automatically initializes all the class vectors w i .
Revised: This automatically initializes all the class vectors w i .

Original: We now preprocess all these randomly initialized class vectors in ( K, L ) parameterized LSH hash tables, as described in
Revised: We now preprocess all these randomly initialized class vectors in ( K, L ) parameterized LSH hash tables, as described in

Original: Section 1.2.
Revised: Section 1.2.

Original: This is a one-time operation during initialization.
Revised: This is a one-time operation during initialization.

Original: Sampling Phase for every input ( x, y ) :
Revised: Sampling Phase for every input ( x, y ) :

Original: In this phase, we process the input x to the penultimate layer and get the ﬁnal input embedding E x .
Revised: In this phase, we process the input x to the penultimate layer and get the ﬁnal input embedding E x .

Original: Now instead of processing all the N nodes in the last layer, we query the hash tables with either vector E x ( LSH Embedding ) or with the vector corresponding to the true label y , i.e., w y ( LSH Label ).
Revised: Now instead of processing all the N nodes in the last layer, we query the hash tables with either vector E x ( LSH Embedding ) or with the vector corresponding to the true label y , i.e., w y ( LSH Label ).

Original: This preciously describes our two sampling schemes.
Revised: This preciously describes our two sampling schemes.

Original: We can obviously mix and match, but we consider these two choices as two different methods for the simplicity of analysis and evaluations.
Revised: We can obviously mix and match, but we consider these two choices as two different methods for the simplicity of analysis and evaluations.

Original: When we query, we generate a small set of the sampled candidates, call them C , forming our negative samples.
Revised: When we query, we generate a small set of the sampled candidates, call them C , forming our negative samples.

Original: Thus, we only compute the activation of nodes belonging to C ∪ y in the last layer and treat others as zero activation.
Revised: Thus, we only compute the activation of nodes belonging to C ∪ y in the last layer and treat others as zero activation.

Original: Update Hash Tables with Update in Weights: During backpropagation for input ( x, y ) , we only update C ∪ y weights in the last layer.
Revised: Update Hash Tables with Update in Weights: During backpropagation for input ( x, y ) , we only update C ∪ y weights in the last layer.

Original: We update these changed weights in the LSH hash tables.
Revised: We update these changed weights in the LSH hash tables.

Original: Next, we ﬁrst argue why this sampling is distribution aware and adaptive with every parameter and input change.
Revised: Next, we ﬁrst argue why this sampling is distribution aware and adaptive with every parameter and input change.

Original: We will then argue that the sampling and update process is signiﬁcantly efﬁcient.
Revised: We will then argue that the sampling and update process is signiﬁcantly efﬁcient.

Original: It is a constant-time operation that is easily parallelizable.
Revised: It is a constant-time operation that is easily parallelizable.

Original: 2. W HAT IS THE S AMPLING D ISTRIBUTION ? I S IT A DAPTIVE ?
Revised: 2. W HAT IS THE S AMPLING D ISTRIBUTION ? I S IT A DAPTIVE ?

Original: We start with two theorems that give the precise probability distribution of sampling a class as  negative sampling provided the input ( x, y ) and current parameter with LSH Label and LSH Embedding
Revised: We start with two theorems that give the precise probability distribution of sampling a class as a negative sample with LSH Label and LSH Embedding methods provided the input ( x, y ) and current parameters.

Original: Methods.

Original: We will use p xy as the collision probability of the LSH hash value of x and y .
Revised: We will use p xy as the collision probability of the LSH hash value of x and y .

Original: Theorem 1 LSH Label Distribution For an input ( x, y ) and LSH parameters ( K, L ) , the probability of sampling a class
Revised: Theorem 1 LSH Label Distribution For an input ( x, y ) and LSH parameters ( K, L ) , the probability of sampling a class

Original: i (cid:54) =
Revised: i (cid:54) =

Original: y as negative sampling with LSH Label method is given by
Revised: y as negative sampling with LSH Label method is given by

Original: [Equation]
Revised: [Equation]

Original: where w y and w i are the weights associated with true class
Revised: where w y and w i are the weights associated with true class

Original: y
Revised: y

Original: and i respectively.
Revised: and i respectively.

Original: Furthermore, the probability of sampling class i is more than any other class j , if and only if sim ( w y , w i ) >
Revised: Furthermore, the probability of sampling class i is more than any other class j , if and only if sim ( w y , w i ) >

Original: sim ( w y , w j ) .
Revised: sim ( w y , w j ) .

Original: Here sim is the underlying similarity function of the LSH.
Revised: Here sim is the underlying similarity function of the LSH.

Original: Theorem 2 LSH Embedding Distribution For an input ( x, y ) and LSH parameters ( K, L ) , the probability of sampling a class i (cid:54)
Revised: Theorem 2 LSH Embedding Distribution For an input ( x, y ) and LSH parameters ( K, L ) , the probability of sampling a class i (cid:54)

Original: =
Revised: =

Original: y as negative sampling with LSH Embedding method is given by
Revised: y as negative sampling with LSH Embedding method is given by

Original: [Equation]
Revised: [Equation]

Original: where E x is the embedding vector of input x and w i is the weights associated with class i respectively.
Revised: where E x is the embedding vector of input x and w i is the weights associated with class i respectively.

Original: Furthermore, the probability of sampling class i is more than any other class j , if and only if sim ( E x , w i ) >
Revised: Furthermore, the probability of sampling class i is more than any other class j , if and only if sim ( E x , w i ) >

Original: sim ( E x , w j ) .
Revised: sim ( E x , w j ) .

Original: Here sim is the underlying similarity function of the LSH.
Revised: Here sim is the underlying similarity function of the LSH.

Original: Comments: The expressions of probability are immediate from the sampling view of LSH.
Revised: Comments: The expressions of probability are immediate from the sampling view of LSH.

Original: The expressions 1 − (1 − p K
Revised: The expressions 1 − (1 − p K

Original: L is monotonically increasing in p , the collision probability, which in turn is monotonically increasing in the underlying similarity function sim .
Revised: L is monotonically increasing in p , the collision probability, which in turn is monotonically increasing in the underlying similarity function sim .

Original: Clearly, the distribution is adaptive as they change with the input x, y as well as the parameters.
Revised: Clearly, the distribution is adaptive as they change with the input x, y as well as the parameters.

Original: So any update in the parameter or any change in the input change the sampling distribution completely.
Revised: So any update in the parameter or any change in the input change the sampling distribution completely.

Original: However, the sampling cost is constant and independent of the number of classes we are sampling from!
Revised: However, the sampling cost is constant and independent of the number of classes we are sampling from!

Original: Intuition of LSH Label: Coming back to our example of class ’Nike Running Shoes’.
Revised: Intuition of LSH Label: Coming back to our example of class ’Nike Running Shoes’.

Original: Let us focus on LSH Label distribution.
Revised: Let us focus on LSH Label distribution.

Original: Initially, when all other labels have random weights, the similarity between the label ’Nike Running Shoes’ and any other label will be random.
Revised: Initially, when all other labels have random weights, the similarity between the label ’Nike Running Shoes’ and any other label will be random.

Original: So initial negative sampling should be like uniform sampling.
Revised: So initial negative sampling should be like uniform sampling.

Original: However, as the learning progress, it is likely that ’Nike
Revised: However, as the learning progress, it is likely that ’Nike

Original: Running Shoes’ and ‘Adidas Running Shoes’ will likely get close enough.
Revised: Running Shoes’ and ‘Adidas Running Shoes’ will likely get close enough.

Original: Their weights will have high similarity (high sim ), at that time, the LSH Label sampling will select ‘Adidas Running Shoes’ as likely negative sample for ‘Nike Running Shoes’ class.
Revised: Their weights will have high similarity (high sim ), at that time, the LSH Label sampling will select ‘Adidas Running Shoes’ as likely negative sample for ‘Nike Running Shoes’ class.

Original: Intuition of LSH Embedding: The LSH Embedding method is also adaptive.
Revised: Intuition of LSH Embedding: The LSH Embedding method is also adaptive.

Original: Consider the similarity function as an inner product.
Revised: Consider the similarity function as an inner product.

Original: LSH embedding inner product with class vector is directly proportional to its activation.
Revised: LSH embedding inner product with class vector is directly proportional to its activation.

Original: Thus, it naturally selects classes in which the classiﬁer is confused (high activation but incorrect) as negative samples.
Revised: Thus, it naturally selects classes in which the classiﬁer is confused (high activation but incorrect) as negative samples.

Original: Again, the distribution is adaptive.
Revised: Again, the distribution is adaptive.

Original: 2. C OMPUTATIONAL C OST FOR P ROCESSING E ACH I NPUT
Revised: 2. C OMPUTATIONAL C OST FOR P ROCESSING E ACH I NPUT

Original: Given an input ( x, y ) , the cost of processing it without any negative sampling is O ( N ) .
Revised: Given an input ( x, y ) , the cost of processing it without any negative sampling is O ( N ) .

Original: With our proposed negative sampling the cost of sampling is the cost of query which is K × L , a negligible number compared to N in practice.
Revised: With our proposed negative sampling the cost of sampling is the cost of query which is K × L , a negligible number compared to N in practice.

Original: [Figure]
Revised: [Figure]

Original: The cost of update is slightly more ( | C | + 1) × K × L because we have to update | C | + 1 weights.
Revised: The cost of update is slightly more ( | C | + 1) × K × L because we have to update | C | + 1 weights.

Original: In negative sampling, C is a very small constant.
Revised: In negative sampling, C is a very small constant.

Original: Also, in practice K and L are small constants.
Revised: Also, in practice K and L are small constants.

Original: Furthermore, we have a choice to delay the hash table updates, as it only changes the sampling probability of few elements.
Revised: Furthermore, we have a choice to delay the hash table updates, as it only changes the sampling probability of few elements.

Original: 2. A LGORITHM AND I MPLEMENTATION D ETAILS
Revised: 2. A LGORITHM AND I MPLEMENTATION D ETAILS

Original: First we construct K × L hash functions and initialize the weights of the network and L hash tables.
Revised: First we construct K × L hash functions and initialize the weights of the network and L hash tables.

Original: The LSH hash code of weight vectors of the last layer are computed and the id of the corresponding neuron is saved into the hash buckets.
Revised: The LSH hash code of weight vectors of the last layer are computed and the id of the corresponding neuron is saved into the hash buckets.

Original: During the feed-forward path in the last layer, we query whether the embedding vector (LSH-embedding scheme) or the label vector of true class (LSH-label scheme) and retrieve the classes from hash table which are considered as negative classes.
Revised: During the feed-forward path in the last layer, we query whether the embedding vector (LSH-embedding scheme) or the label vector of true class (LSH-label scheme) and retrieve the classes from hash table which are considered as negative classes.

Original: Instead of computing the activation of all the output nodes (full softmax), we compute the activations of the true classes and the corresponding retrieved negative classes.
Revised: Instead of computing the activation of all the output nodes (full softmax), we compute the activations of the true classes and the corresponding retrieved negative classes.

Original: For the backpropagation, we backpropagate the errors to calculate the gradient and update the weights for the active nodes.
Revised: For the backpropagation, we backpropagate the errors to calculate the gradient and update the weights for the active nodes.

Original: Please refer to
Revised: Please refer to

Original: Algorithm 1 in the Appendix.
Revised: Algorithm 1 in the Appendix.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we will empirically evaluate the performance of our LSH Negative Sampling (LNS) approach against other sampling schemes that are conducive to GPUs.
Revised: In this section, we will empirically evaluate the performance of our LSH Negative Sampling (LNS) approach against other sampling schemes that are conducive to GPUs.

Original: The real advantage of LNS is noticeable with huge neural networks.
Revised: The real advantage of LNS is noticeable with huge neural networks.

Original: The popular extreme classiﬁcation challenges have models with more than 100 million parameters, which are ideal for our purpose.
Revised: The popular extreme classiﬁcation challenges have models with more than 100 million parameters, which are ideal for our purpose.

Original: For these challenges, most of the heavy computations happen in the last layer.
Revised: For these challenges, most of the heavy computations happen in the last layer.

Original: 3. D ATASETS
Revised: 3. D ATASETS

Original: We evaluate our framework and other baselines on three datasets.
Revised: We evaluate our framework and other baselines on three datasets.

Original: Amazon670K and WikiLSH-325K are two datasets from extreme classiﬁcation repository (Bhatia et al., 2016) and Text8 is a popular
Revised: Amazon670K and WikiLSH-325K are two datasets from extreme classiﬁcation repository (Bhatia et al., 2016) and Text8 is a popular

Original: NLP dataset.
Revised: NLP dataset.

Original: The detailed statistics about the dimensions and samples sizes are included in Tablein appendix A.1.
Revised: The detailed statistics about the dimensions and samples sizes are included in Tablein appendix A.1.

Original: 3. B ASELINES
Revised: 3. B ASELINES

Original: We benchmark our proposed framework against full-softmax, sampled-softmax, topK-softmax, frequency-based-softmax, noise contrastive estimation and differentiated softmax (all explained below).
Revised: We benchmark our proposed framework against full-softmax, sampled-softmax, topK-softmax, frequency-based-softmax, noise contrastive estimation and differentiated softmax (all explained below).

Original: All the baselines are implemented on TensorFlow-GPU.
Revised: All the baselines are implemented on TensorFlow-GPU.

Original: To have a fair comparison, the architecture, optimizer and size of hidden layer are exactly the same for all the methods on each dataset.
Revised: To have a fair comparison, the architecture, optimizer and size of hidden layer are exactly the same for all the methods on each dataset.

Original: Please note that our proposal only uses CPU and yet outperforms the other methods.
Revised: Please note that our proposal only uses CPU and yet outperforms the other methods.

Original: Full-Softmax : Full-softmax updates the weights of all the output neurons, which makes it computationally expensive and intractable for extreme classiﬁcation framework.
Revised: Full-Softmax : Full-softmax updates the weights of all the output neurons, which makes it computationally expensive and intractable for extreme classiﬁcation framework.

Original: Sampled-Softmax :
Revised: Sampled-Softmax :

Original: Sampled-softmax draws negative samples based on log-uniform distribution and updates their corresponding weights plus the weights for the true classes.
Revised: Sampled-softmax draws negative samples based on log-uniform distribution and updates their corresponding weights plus the weights for the true classes.

Original: This approach alleviates the computational
Revised: This approach alleviates the computational

Original: [Figure]
Revised: [Figure]

Original: The methods closely mimicsin of distributionaware bottleneck but degrades the performance in terms of accuracy.
Revised: The methods closely mimics distribution bottleneck but degrades the performance in terms of accuracy.

Original: TopK-Softmax : TopK-softmax updates the weights of the output neurons with k highest activations (including the true classes).
Revised: TopK-Softmax : TopK-softmax updates the weights of the output neurons with k highest activations (including the true classes).

Original: This framework maintains better accuracy than sampled-softmax but with a slower convergence rate due to the scoring and sorting of all classes.
Revised: This framework maintains better accuracy than sampled-softmax but with a slower convergence rate due to the scoring and sorting of all classes.

Original: Frequency-based-Softmax: Frequency-based-softmax samples the classes in proportion to the frequency of their occurrence in the training data.
Revised: Frequency-based-Softmax: Frequency-based-softmax samples the classes in proportion to the frequency of their occurrence in the training data.

Original: Computationally, this is the same as Sampled-softmax, however the convergence is better because we sample ‘hard’ negatives from more frequent classes with higher probability.
Revised: Computationally, this is the same as Sampled-softmax, however the convergence is better because we sample ‘hard’ negatives from more frequent classes with higher probability.

Original: Noise-Contrastive Estimation (NCE): NCE loss (Gutmann & Hyv¨arinen, 2010) tackles multi-calss classiﬁcation problem as multiple binary classiﬁers instead.
Revised: Noise-Contrastive Estimation (NCE): NCE loss (Gutmann & Hyv¨arinen, 2010) tackles multi-class classiﬁcation problem as multiple binary classiﬁers instead.

Original: Each binary classiﬁer is trained by logistic loss to distinguish between true classes and negative classes.
Revised: Each binary classiﬁer is trained by logistic loss to distinguish between true classes and negative classes.

Original: Negative classes are sampled from a noise distribution which is typically log-uniform distribution or based on class frequencies.
Revised: Negative classes are sampled from a noise distribution which is typically log-uniform distribution or based on class frequencies.

Original: Differentiated Softmax
Revised: Differentiated Softmax

Original: (D-Softmax): D-softmax (Chen et al., 2015) deﬁnes softmax layer weight matrix as a sparse block diagonal matrix where the blocks are constructed based on class frequencies, e.g. more frequent classes attains more parameters.
Revised: (D-Softmax): D-softmax (Chen et al., 2015) deﬁnes softmax layer weight matrix as a sparse block diagonal matrix where the blocks are constructed based on class frequencies, e.g. more frequent classes attains more parameters.

Original: LNS (our proposal) :
Revised: LNS (our proposal) :

Original: Our proposed negative sampling scheme is a proxy to topK-softmax.
Revised: Our proposed negative sampling scheme is a proxy to topK-softmax.

Original: It selects the top-k classes via LSH (instead of computing the activations and sorting), where the nearest neighbor classes of the true class are chosen as the negative samples.
Revised: It selects the top-k classes via LSH (instead of computing the activations and sorting), where the nearest neighbor classes of the true class are chosen as the negative samples.

Original: 3. A RCHITECTURE AND H YPERPARAMETERS
Revised: 3. A RCHITECTURE AND H YPERPARAMETERS

Original: For Amazon-670K and WikiLSH-325K we use a standard fully connected neural network with hidden layer size of 128, where both the input and output are multi-hot encoded vectors.
Revised: For Amazon-670K and WikiLSH-325K we use a standard fully connected neural network with hidden layer size of 128, where both the input and output are multi-hot encoded vectors.

Original: For Text8, We utilize standard word2vec language model with hidden layer size of 200, where input and output are one-hot and multi-hot encoded vectors, respectively.
Revised: For Text8, We utilize standard word2vec language model with hidden layer size of 200, where input and output are one-hot and multi-hot encoded vectors, respectively.

Original: In word2vec architecture we utilize skip-gram model introduced in Mikolov et al.
Revised: In word2vec architecture we utilize skip-gram model introduced in Mikolov et al.

Original: (2013a).
Revised: (2013a).

Original: Skip-gram model aims to predict nearby words in a document by learning continuous representation of words.
Revised: Skip-gram model aims to predict nearby words in a document by learning continuous representation of words.

Original: Particularly, given a word, skip-gram model targets to predict the m left and m right neighbor words, where m is window size and is considered as a hyperparameter.
Revised: Particularly, given a word, skip-gram model targets to predict the m left and m right neighbor words, where m is window size and is considered as a hyperparameter.

Original: We pick m = 2 for our experiments.
Revised: We pick m = 2 for our experiments.

Original: We performed hyperparameter tuning for all the baselines to maintain their best trade-off between convergence time and accuracy.
Revised: We performed hyperparameter tuning for all the baselines to maintain their best trade-off between convergence time and accuracy.

Original: The optimizer is Adam with learning rate 0.0001 for all the experiments.
Revised: The optimizer is Adam with learning rate 0.0001 for all the experiments.

Original: The batch size for Amazon670, WikiLSH-325 and Text8 is 1024, 256 and 512 respectively for all the experiments.
Revised: The batch size for Amazon670, WikiLSH-325 and Text8 is 1024, 256 and 512 respectively for all the experiments.

Original: We apply hash functions for the last layer where we have the computational bottleneck.
Revised: We apply hash functions for the last layer where we have the computational bottleneck.

Original: In LSH literature, L denotes the number of hash tables and K denotes the number of bits in the hash code for each hash table (thereby having 2 K buckets per hash table).
Revised: In LSH literature, L denotes the number of hash tables and K denotes the number of bits in the hash code for each hash table (thereby having 2 K buckets per hash table).

Original: We use DWTA hash functions for Amazon670 and WikiLSH-325K with K=6, L=400 and K=5, L=350 respectively.
Revised: We use DWTA hash function [A.3] for Amazon670 and WikiLSH-325K with K=6, L=400 and K=5, L=350 respectively.

Original: For Text8 we use Simhash hash function  with K=9 and L=50.
Revised: For Text8 we use Simhash hash function [A.2] with K=9 and L=50.

Original: We update the hash tables with an initial update period of 50 iterations and then exponentially decaying the updating frequency (as we need less updates near convergence).
Revised: We update the hash tables with an initial update period of 50 iterations and then exponentially decaying the updating frequency (as we need less updates near convergence).

Original: Our experiments are performed on a single machine with 28-core and 224-thread processors.
Revised: Our experiments are performed on a single machine with 28-core and 224-thread processors.

Original: All the baselines are run on the state of the art NVIDIA V100 GPUs with 32 GB memory.
Revised: All the baselines are run on the state of the art NVIDIA V GPUs with 32 GB memory.

Original: 3. R ESULTS
Revised: 3. R ESULTS

Original: Figure 4 shows the plots comparing Precision @1 (denoted here-on by P@1) versus both wall-clock time and the number of iterations for our method and all the baselines.
Revised: Figure 4 shows the plots comparing Precision @1 (denoted here-on by P@1) versus both wallclock time and the number of iterations for our method and all the baselines.

Original: For WikiLSH-325K dataset, LSH-label and LSH-embedding are respectively 4x and 4.3x faster than Tensorﬂow full softmax on GPU in terms of wall-clock training time.
Revised: For WikiLSH-325K dataset, LSH-label and LSH-embedding are respectively 4x and 4.3x faster than TensorFlow full softmax on GPU in terms of wall-clock training time.

Original: Moreover, both of them outperform all other Tesorﬂow baselines on GPU with signiﬁcant margin.
Revised: Moreover, both of them outperform all other TensorFlow baselines on GPU with signiﬁcant margin.

Original: The same is true for Amazon-670K whereLSH-label and
Revised: The same is true for Amazon-670K where

Original:  LSH-embedding are 10x and 9.6x faster than Tensorﬂow full softmax on GPU correspondingly.
Revised: LSH-label and LSH-embedding are 10x and 9.6x faster than TensorFlow full softmax on GPU correspondingly.

Original: On

Original:  Text8 dataset, all methods perform more or less equally in terms of P@1 and our LSH-Embedding is the second fastest negative sampling method after sampled softmax in terms of training time.
Revised: On Text8 dataset, all methods perform more or less equally in terms of P@1 and our LSH-Embedding is the second fastest negative sampling method after sampled softmax in terms of training time.

Original: According to iteration-wise plots, both LSH-label and LSH-Embedding schemes attain the highest P@1 among all other sampling methods for Amazon-670K and WikiLSH-325K datasets, and achieves pretty similar performance with other sampling methods on Text8 dataset.
Revised: According to iteration-wise plots, both LSH-label and LSH-Embedding schemes attain the highest P@1 among all other sampling methods for Amazon-670K and WikiLSH-325K datasets, and achieves pretty similar performance with other sampling methods on Text8 dataset.

Original: Therefore, both variations of our LNS method outperform almost all other Tensorﬂow baselines on all datasets while being very similar to full softmax iteration-wise.
Revised: Therefore, both variations of our LNS method outperform almost all other all other TensorFlow baselines on all datasets while being very similar to full softmax iteration-wise.

Original: This establishes the earlier statement that LNS does not compromise performance for speed-up.
Revised: This establishes the earlier statement that LNS does not compromise performance for speed-up.

Original: This is particularly noteworthy because our implementation of LSN uses only CPU while all other baselines run Tensorﬂow-GPU on a V100.
Revised: This is particularly noteworthy because our implementation of LNS uses only CPU while all other baselines run TensorFlow-GPU on a V100.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed two provable, efﬁcient and adaptive negative sampling schemes for neural networks with extreme number of classes.
Revised: We proposed two provable, efﬁcient and adaptive negative sampling schemes for neural networks with extreme number of classes.

Original: Our method samples negative classes in constant time, while adapts to the continuous change of the input, true class and network parameters.
Revised: Our method samples negative classes in constant time, while adapts to the continuous change of the input, true class and network parameters.

Original: We efﬁciently implemented our algorithm on CPU in C++ and benchmarked it against standard Tensorﬂow implementation of six baselines on GPU.
Revised: We efﬁciently implemented our algorithm on CPU in C++ and benchmarked it against standard TensorFlow implementation of six baselines on GPU.

Original: Our method on CPU outperforms almost all the Tensorﬂow baselines on GPU with signiﬁcant margin on three datasets.
Revised: Our method on CPU outperforms almost all the TensorFlow baselines on GPU with signiﬁcant margin on three datasets.

Original: A A PPENDIX A. D ATASET
Revised: A A PPENDIX A. D ATASET

Original: Amazon-670K dataset is a product recommendation dataset with 670K labels.
Revised: Amazon-670K dataset is a product recommendation dataset with 670K labels.

Original: Here, each input is a vector representation of a product, and the corresponding labels are other products (among 670K choices) that a user might be interested in purchase.
Revised: Here, each input is a vector representation of a product, and the corresponding labels are other products (among 670K choices) that a user might be interested in purchase.

Original: This is an anonymized and aggregated behavior data from Amazon and poses a signiﬁcant challenge owing to a large number of classes.
Revised: This is an anonymized and aggregated behavior data from Amazon and poses a signiﬁcant challenge owing to a large number of classes.

Original: WikiLSHTC-325K is based on two main sources: Wikipedia and ODP web directory.
Revised: WikiLSHTC-325K is based on two main sources: Wikipedia and ODP web directory.

Original: Each instance is in sparse vector format where the value of a feature corresponds to the frequency and the label corresponds to the category of the instance.
Revised: Each instance is in sparse vector format where the value of a feature corresponds to the frequency and the label corresponds to the category of the instance.

Original: Text8 is a popular NLP dataset and a preprocessed version of the ﬁrst 100million tokens of English
Revised: Text8 is a popular NLP dataset and a preprocessed version of the ﬁrst 100million tokens of English

Original: Wikipedia which contains 253K vocabulary.
Revised: Wikipedia which contains 253K vocabulary.

Original: We utilized standard word2vec language model for text dataset, where input and output are one-hot and multi-hot
Revised: We utilized standard word2vec language model for text dataset, where input and output are one-hot and multi-hot

Original: encoded vectors, respectively.
Revised: encoded vectors, respectively.

Original: A. A LGORITHM
Revised: A. S IM H ASH : S IGNED R ANDOM P ROJECTIONS

Revised: SimHash is a popular LSH that originates from Signed Random Projections (SRP) (Charikar, 2002; Rajaraman & Ullman, 2010; Henzinger, 2006) that preserves the cosine similarity measure. Given a vector x , SRP generates a random w vector with each component generated from i.i.d.

Revised: normal, , w i ∼ N (0 , 1) , and only stores the sign of the projection.

Revised: Formally SimHash is given by

Original: [Table]
Revised: [Equation]

Original: input E x ﬁnal input embedding, w i class vectors output W o weights to update 1: Initialize weights w l for each layer l 2: Create K × L hash functions and initialize L hash tables for the last layer 3: Compute h l ( w l ) for all output neurons 4: for i =  1 : Iterations do 5: for Batch B do 6: if LSH
Revised: It was shown in (Goemans & Williamson, 1995) that the collision probability under SRP satisﬁes the following formula: [Equation] where θ = cos − 1 (cid:16) x T y ||

Original: Embedding then 7:
Revised: x || 2 ·|| y || 2 (cid:17) .

Original: W o ={+There is a variant of SimHash where, instead of w i ∼ N (0 , 1) , we choose each w i independently as either +
 or -1 with probability 12 . It is known that this variant performs similarly to the one with w ∼ N (0 , 1) (Rajaraman & Ullman, 2011). Since 1 − θπ is monotonic to cosine similarity S , it is a valid LSH.+}
Revised: {+There is a variant of SimHash where, instead of w i ∼ N (0 , 1) , we choose each w i independently as either +
 or -1 with probability 12 . It is known that this variant performs similarly to the one with w ∼ N (0 , 1) (Rajaraman & Ullman, 2011). Since 1 − θπ is monotonic to cosine similarity S , it is a valid LSH.+}

Original: Query( h l ( E x ) , HT l ) 8: end
Revised: DWTA (Chen & Shrivastava, 2018) hash transforms the data into the transformed space such that their hamming distance correlates with their similarity measure in their original space. Densiﬁed

Original: if 9: if LSH Label then 10:
Revised: WTA (DWTA) hash combines traditional WTA hashing and densiﬁcation to improve discriminative power over sparse datasets. Compared to vanilla WTA hash which assigns value of 1 to all empty bins , DWTA assigns empty bins the hash value from some non-empty bins plus some constant offset. It has been shown that the collision probability of DWTA is precisely the collision probability of WTA hash for nonempty bins, irrespective of the sparsity.

Original: W o =
Revised: [Equation]

Original: Query( h l ( w i ) , HT l ) Query i ∈ C B ( h ( w i ) , T t ) 11: end if 12: end for 13: end for
Revised: where Θ( x ) is the set of random features of x with permutation Θ .