Original: MACH: E MBARRASSINGLY PARALLEL K - CLASS CLASSIFICATION IN O ( d log K ) MEMORY AND O ( K log K + d log K ) TIME , INSTEAD OF O ( Kd )
Revised: MACH: E MBARRASSINGLY PARALLEL K - CLASS CLASSIFICATION IN O ( d log K ) MEMORY AND O ( K log K + d log K ) TIME , INSTEAD OF O ( Kd )

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present Merged-Averaged Classiﬁers via Hashing (MACH) forKclassiﬁcation with large K.
Revised: We present Merged-Averaged Classiﬁers via Hashing (MACH) for K

Revised: - classiﬁcation with large

Original:  Compared to traditional one-vs-all classiﬁers that require O( Kd) memory and inference cost, MACH only needO ( d log K ) memory while only requiring O ( K log K + d log K ) operation for inference.
Revised: K . Compared to traditional one-vs-all classiﬁers that require O Kd memory and inference cost, MACH only need

Revised: O

Revised: ( d log K ) memory while only requiring

Revised: O

Revised: ( K log

Revised: K + d log K ) operation for inference.

Original: MACH is the ﬁrst generic K -classiﬁcation algorithm, with provably theoretical guarantees which requires O (log K ) memory without any assumption on the relationship between classes.
Revised: MACH is the ﬁrst generic K -classiﬁcation algorithm, with provably theoretical guarantees which requires O log K ) memory without any assumption on the relationship between classes.

Original: MACH uses universal hashing to reduce classiﬁcation with a large number of classes to few independent classiﬁcation task with small (constant) number of classes.
Revised: MACH uses universal hashing to reduce classiﬁcation with a large number of classes to few independent classiﬁcation task with small (constant) number of classes.

Original: We provide theoretical quantiﬁcation of accuracy-memory tradeoff by showing the ﬁrst connection between extreme classiﬁcation and heavy hitters.
Revised: We provide theoretical quantiﬁcation of accuracy-memory tradeoff by showing the ﬁrst connection between extreme classiﬁcation and heavy hitters

Original: With MACH we can train ODP dataset with 100,000 classes and400, features on a single Titan X GPU (12GB), with the classiﬁcation accuracy of18%, which is the best-reported accuracy on this dataset.
Revised: With MACH we can train ODP dataset with 100, classes and features on a single Titan X GPU 12GB ), with the classiﬁcation accuracy of which is the best-reported accuracy on this dataset.

Original: Before this work, the best performing baseline is a one-vs-all classiﬁer that requires 40 billion parameters ( GB model size) and achieves 9% accuracy.
Revised: Before this work, the best performing baseline is a one-vs-all classiﬁer that requires 40 billion parameters (160 GB model size) and achieves 9% accuracy.

Original: In contrast, MACH can achieve 9% accuracy with 480x reduction in the model size (of mere 0.3GB).
Revised: In contrast, MACH can achieve 9% accuracy with 480x reduction in the model size (of mere 0.3GB).

Original: With MACH, we also demonstrate complete training of ﬁne-grained imagenet dataset (compressed size 104GB), with 21,000 classes, on a single GPU.
Revised: With MACH, we also demonstrate complete training of ﬁne-grained imagenet dataset (compressed size 104GB ), with 21,000 classes, on a single GPU.

Original:  M OTIVATION
Revised:  M OTIVATION

Original: The area of extreme multi-class classiﬁcation has gained signiﬁcant interest recently (Choromanska & Langford, 2015).
Revised: The area of extreme multi-class classiﬁcation has gained signiﬁcant interest recently (Choromanska & Langford, 2015).

Original: Extreme multi-class refers to the vanilla multi-class classiﬁcation problem where the number of classes K is signiﬁcantly large.
Revised: Extreme multi-class refers to the vanilla multi-class classiﬁcation problem where the number of classes K is signiﬁcantly large.

Original: A large number of classes K brings a new set of computational and memory challenges in training and deploying classiﬁers.
Revised: A large number of classes K brings a new set of computational and memory challenges in training and deploying classiﬁers.

Original: The growth in the volume and dimensionality of data is a well-known phenomenon.
Revised: The growth in the volume and dimensionality of data is a well-known phenomenon.

Original: Concurrently, there is also a signiﬁcant growth in the number of classes or labels of interest.
Revised: Concurrently, there is also a signiﬁcant growth in the number of classes or labels of interest.

Original: There are several reasons behind this growth.
Revised: There are several reasons behind this growth.

Original: In the last decade, it has been shown that many hard AI problems can be naturally modeled as a massive multi-class problem, leading to a drastic improvement over prior art.
Revised: In the last decade, it has been shown that many hard AI problems can be naturally modeled as a massive multi-class problem, leading to a drastic improvement over prior art.

Original: The success in NLP is a classic example of this phenomena.
Revised: The success in NLP is a classic example of this phenomena.

Original: For example, popular models predict the best word, given the full context observed so far.
Revised: For example, popular models predict the best word, given the full context observed so far.

Original: Such models are becoming the state-of-the-art in machine translation (Sutskever et al., 2014), word embeddings (Mikolov et al., 2013), etc.
Revised: Such models are becoming the state-of-the-art in machine translation (Sutskever et al., 2014), word embeddings (Mikolov et al., 2013), etc.

Original: For large-dataset, the vocabulary size, can quickly run into billions (Mikolov et al., 2013).
Revised: For large-dataset, the vocabulary size, can quickly run into billions (Mikolov et al., 2013).

Original: 1. T HE H ARDNESS A SSOCIATED WITH THE L ARGE N UMBER OF C LASSES
Revised: 1. T HE H ARDNESS A SSOCIATED WITH THE L ARGE N UMBER OF C LASSES

Original: Due to the popularity of word classiﬁcation, it is not difﬁcult to ﬁnd public datasets with a large number of classes.
Revised: Due to the popularity of word classiﬁcation, it is not difﬁcult to ﬁnd public datasets with a large number of classes.

Original: Microsoft released ODP data (Choromanska & Langford, 2015), where the task is to predict the category for each document (see Section 4.1) which has more than 100,000 classes.
Revised: Microsoft released ODP data (Choromanska & Langford, 2015), where the task is to predict the category for each document (see Section 4.1) which has more than 100,000 classes.

Original: The favorite Imagenet dataset, with ﬁne-grained categories, has over 21,000 classes.
Revised: The favorite Imagenet dataset, with ﬁne-grained categories, has over 21,000 classes.

Original: Deployment Cost: The computational as well as the memory requirements of the classiﬁers scale linearly with the number of classes K .
Revised: Deployment Cost: The computational as well as the memory requirements of the classiﬁers scale linearly with the number of classes K .

Original: For d dimensional dataset the memory required by simple linear models, such as logistic regression, is O ( Kd ) , which is a signiﬁcant cost for deployment.
Revised: For d dimensional dataset the memory required by simple linear models, such as logistic regression, is O ( Kd ) , which is a signiﬁcant cost for deployment.

Original: For example, for ODP dataset, with 400,000 features and 100,000 classes, the model size of simple logistic regression is 4 × 10 10 × 8 bytes, assuming 32 bits double, or 160 gigabytes just to store the model (or parameters).
Revised: For example, for ODP dataset, with 400,000 features and 100,000 classes, the model size of simple logistic regression is 4 × 10 10 × 8 bytes, assuming 32 bits double, or 160 gigabytes just to store the model (or parameters).

Original: Such a large model size will run out of any GPU memory, and even deploying the model size is out of the scope of the main memory of any ordinary machine.
Revised: Such a large model size will run out of any GPU memory, and even deploying the model size is out of the scope of the main memory of any ordinary machine.

Original: The situation naturally applied to deep networks with softmax output layer.
Revised: The situation naturally applied to deep networks with softmax output layer.

Original: Not surprisingly, the ﬁnal layer in deep networks is a known memory bottleneck.
Revised: Not surprisingly, the ﬁnal layer in deep networks is a known memory bottleneck.

Original: Furthermore, inference with linear classiﬁer requires K inner products of size d each, totaling
Revised: Furthermore, inference with linear classiﬁer requires K inner products of size d each, totaling

Original: O ( dK ) multiplications just to predict one instance.
Revised: O ( dK ) multiplications just to predict one instance.

Original: Such high inference cost is near-infeasible in many online and latency-critical applications.
Revised: Such high inference cost is near-infeasible in many online and latency-critical applications.

Original: Training Cost: Training with large parameter space, such as 40 billion in case of ODP dataset, is always an issue.
Revised: Training Cost: Training with large parameter space, such as 40 billion in case of ODP dataset, is always an issue.

Original: The iterations are not parallelizable.
Revised: The iterations are not parallelizable.

Original: With large number of parameters and sequential updates, the convergence is time consuming.
Revised: With large number of parameters and sequential updates, the convergence is time consuming.

Original: As argued, due to signiﬁcant model sizes (160GB or higher) we cannot train ODP dataset on a single GPU.
Revised: As argued, due to signiﬁcant model sizes (160GB or higher) we cannot train ODP dataset on a single GPU.

Original: All existing methods use either distributed cluster or large server machines with a large amount of main memory (RAM).
Revised: All existing methods use either distributed cluster or large server machines with a large amount of main memory (RAM).

Original: 1. M EMORY -C OMPUTATIONS T RADEOFF : P RIOR A RT
Revised: 1. M EMORY -C OMPUTATIONS T RADEOFF : P RIOR A RT

Original: In the context of multi-class classiﬁcation, reducing the prediction time at the cost of increase model size is studied recently.
Revised: In the context of multi-class classiﬁcation, reducing the prediction time at the cost of increase model size is studied recently.

Original: There are two main lines of work in this regard.
Revised: There are two main lines of work in this regard.

Original: It was ﬁrst observed in (Shrivastava & Li, 2014) that the prediction cost, of multi-class classiﬁcation problem, can be reduced to a maximum inner product search instance which admits faster solution using locality sensitive hashing.
Revised: It was ﬁrst observed in (Shrivastava & Li, 2014) that the prediction cost, of multi-class classiﬁcation problem, can be reduced to a maximum inner product search instance which admits faster solution using locality sensitive hashing.

Original: (Vijayanarasimhan et al., 2014) demonstrated this empirically in the context of deep networks.
Revised: (Vijayanarasimhan et al., 2014) demonstrated this empirically in the context of deep networks.

Original: However, this reduction comes at the cost of increased memory required for storing hash tables, along with O ( Kd ) for the parameters.
Revised: However, this reduction comes at the cost of increased memory required for storing hash tables, along with O ( Kd ) for the parameters.

Original: (Daume III et al., 2016), instead of hash tables, uses smartly learned trees to prune down the search space quickly.
Revised: (Daume III et al., 2016), instead of hash tables, uses smartly learned trees to prune down the search space quickly.

Original: However, they still requiring storing all the parameters and the additional memory needed to save trees doubles the overall space complexity.
Revised: However, they still requiring storing all the parameters and the additional memory needed to save trees doubles the overall space complexity.

Original: Furthermore, the parallelism is hurt in the inference time due to the tree traversals.
Revised: Furthermore, the parallelism is hurt in the inference time due to the tree traversals.

Original: Memory is the primary barrier in extreme classiﬁcation both during training and testing.
Revised: Memory is the primary barrier in extreme classiﬁcation both during training and testing.

Original: The computation required for prediction, in one-vs-all classiﬁcation, is trivially parallelizable.
Revised: The computation required for prediction, in one-vs-all classiﬁcation, is trivially parallelizable.

Original: For example, in logistic regression, all the K probabilities (inner products) can be computed in parallel followed by computing the maximum via reduction.
Revised: For example, in logistic regression, all the K probabilities (inner products) can be computed in parallel followed by computing the maximum via reduction.

Original: Thus, latency can be reduced with more parallelization.
Revised: Thus, latency can be reduced with more parallelization.

Original: However, memory creates the major hurdle in deployment and communication cost of distributed computing is prohibitive.
Revised: However, memory creates the major hurdle in deployment and communication cost of distributed computing is prohibitive.

Original: Unfortunately, reducing memory in extreme scale classiﬁcation has not received enough formal treatment.
Revised: Unfortunately, reducing memory in extreme scale classiﬁcation has not received enough formal treatment.

Original: There are, however, several tricks to reduce memory in multi-class classiﬁcation.
Revised: There are, however, several tricks to reduce memory in multi-class classiﬁcation.

Original: The high-level idea is to compress the parameter space, and whenever the parameters are needed, they are reconstructed back.
Revised: The high-level idea is to compress the parameter space, and whenever the parameters are needed, they are reconstructed back.

Original: Notable approaches include hashed nets (Chen et al., 2015), Huffman compression (Han et al., 2015), etc.
Revised: Notable approaches include hashed nets (Chen et al., 2015), Huffman compression (Han et al., 2015), etc.

Original: As expected, the compression and decompression for accessing any parameter lead to costly overheads.
Revised: As expected, the compression and decompression for accessing any parameter lead to costly overheads.

Original: Furthermore, most of the compression methods only lead to around a constant factor improvement without any formal information theoretic guarantees.
Revised: Furthermore, most of the compression methods only lead to around a constant factor improvement without any formal information theoretic guarantees.

Original: A Note on Dimensionality Reduction: Dimensionality reduction techniques such as random projections and feature hashing (Weinberger et al., 2009) is an orthogonal idea to reduce computations and memory by decreasing the factor d .
Revised: A Note on Dimensionality Reduction: Dimensionality reduction techniques such as random projections and feature hashing (Weinberger et al., 2009) is an orthogonal idea to reduce computations and memory by decreasing the factor d .

Original: On the contrary, this paper is focused on reducing the factor
Revised: On the contrary, this paper is focused on reducing the factor

Original: K .
Revised: K .

Original: All the analysis and results of the paper assuming d is the optimum dimensionality which is sufﬁcient for learning.
Revised: All the analysis and results of the paper assuming d is the optimum dimensionality which is sufﬁcient for learning.

Original: Difference from Multi-Label and Structured Predictions: There are other tasks with a large number of classes such as multi-label learning (Yu et al., 2014) and structured prediction (BakIr, 2007).
Revised: Difference from Multi-Label and Structured Predictions: There are other tasks with a large number of classes such as multi-label learning (Yu et al., 2014) and structured prediction (BakIr, 2007).

Original: If we allow multiple outputs for each example, then every combination can easily give us a potentially large number of classes.
Revised: If we allow multiple outputs for each example, then every combination can easily give us a potentially large number of classes.

Original: However, the classes are not independent, and the structure can be exploited to reduce both computations and memory (Jasinska & Karampatziakis, 2016).
Revised: However, the classes are not independent, and the structure can be exploited to reduce both computations and memory (Jasinska & Karampatziakis, 2016).

Original: Our Focus: We do not make any assumptions on the classes.
Revised: Our Focus: We do not make any assumptions on the classes.

Original: Our results are for any generic K -class classiﬁcation without any relations, whatsoever, between the classes.
Revised: Our results are for any generic K -class classiﬁcation without any relations, whatsoever, between the classes.

Original: For the sake of readability, we use standard logistic regression as a running example for our technique and experimentations.
Revised: For the sake of readability, we use standard logistic regression as a running example for our technique and experimentations.

Original: However, the results naturally extend to any K -class classiﬁer, including deep networks.
Revised: However, the results naturally extend to any K -class classiﬁer, including deep networks.

Original: Existing methods, reduce inference time at the cost of increased memory and vice versa.
Revised: Existing methods, reduce inference time at the cost of increased memory and vice versa.

Original: It is long known that there is a tradeoff between computations, memory, and accuracy.
Revised: It is long known that there is a tradeoff between computations, memory, and accuracy.

Original: However, in the context of multi-class classiﬁcation, even with approximations, there are no principled methods that can reduce both computation and memory simultaneously, compared to O ( Kd ) , while providing theoretical guarantees of the tradeoffs.
Revised: However, in the context of multi-class classiﬁcation, even with approximations, there are no principled methods that can reduce both computation and memory simultaneously, compared to O ( Kd ) , while providing theoretical guarantees of the tradeoffs.

Original: This work presents the ﬁrst such result for K -class classiﬁcation.
Revised: This work presents the ﬁrst such result for K -class classiﬁcation.

Original: 1. O UR C ONTRIBUTIONS :
Revised: 1. O UR C ONTRIBUTIONS :

Original: We propose a simple hashing based divide and conquer algorithm MACH (Merged-Average Classiﬁcation via Hashing) for K -class classiﬁcation, which only requires O ( d log K ) model size (memory) instead of O
Revised: We propose a simple hashing based divide and conquer algorithm MACH (Merged-Average Classiﬁcation via Hashing) for K -class classiﬁcation, which only requires O ( d log K ) model size (memory) instead of O

Original: ( Kd ) required by traditional linear classiﬁers.
Revised: ( Kd ) required by traditional linear classiﬁers.

Original: MACH also provides computational savings by requiring only O ( d log K + K log K ) multiplications instead of O ( Kd ) .
Revised: MACH also provides computational savings by requiring only O ( d log K + K log K ) multiplications instead of O ( Kd ) .

Original: Furthermore, the training process of MACH is embarrassingly parallelizable.
Revised: Furthermore, the training process of MACH is embarrassingly parallelizable.

Original: Overall, MACH uses a 2-universal random hash function to assign classes to a small number of buckets.
Revised: Overall, MACH uses a 2-universal random hash function to assign classes to a small number of buckets.

Original: The large output classiﬁcation problem is reduced to a small output space problem.
Revised: The large output classiﬁcation problem is reduced to a small output space problem.

Original: A small classiﬁer is built on these merged buckets (or meta-classes).
Revised: A small classiﬁer is built on these merged buckets (or meta-classes).

Original: Using only logarithmic (in K ) such independent classiﬁers, MACH can discriminate any pair of classes with high probability.
Revised: Using only logarithmic (in K ) such independent classiﬁers, MACH can discriminate any pair of classes with high probability.

Original: We provide strong theoretical guarantees quantifying the tradeoffs between computations accuracy and memory.
Revised: We provide strong theoretical guarantees quantifying the tradeoffs between computations accuracy and memory.

Original: In particular, we show that in log K √ δ d memory MACH can discriminate between any two pair of classes with probability 1 − δ .
Revised: In particular, we show that in log K √ δ d memory MACH can discriminate between any two pair of classes with probability 1 − δ .

Original: Our analysis provides a novel treatment for approximation in classiﬁcation via a property call distinguishability between any pair of classes.
Revised: Our analysis provides a novel treatment for approximation in classiﬁcation via a property call distinguishability between any pair of classes.

Original: Our novel formalism of approximate multi-class classiﬁcation and its strong connections with compressed sensing and heavy hitters problem could be of independent interest in itself.
Revised: Our novel formalism of approximate multi-class classiﬁcation and its strong connections with compressed sensing and heavy hitters problem could be of independent interest in itself.

Original: MACH achieves 19.28% accuracy on the ODP dataset which is the best-reported accuracy seen so far on this dataset.
Revised: MACH achieves 19.28% accuracy on the ODP dataset which is the best-reported accuracy seen so far on this dataset.

Original: The best previous accuracy on this partitions was only 9% (Daume III et al., 2016).
Revised: The best previous accuracy on this partitions was only 9% (Daume III et al., 2016).

Original: Furthermore, MACH only requires a single GPU (Nvidia Pascal Titan X).
Revised: Furthermore, MACH only requires a single GPU (Nvidia Pascal Titan X).

Original: The model size with
Revised: The model size with

Original: MACH is mere around 1.2GB, to achieve around 15% accuracy compared to around 160GB with the one-vs-all classiﬁer that gets 9% and requires high-memory servers to process heavy models.
Revised: MACH is mere around 1.2GB, to achieve around 15% accuracy compared to around 160GB with the one-vs-all classiﬁer that gets 9% and requires high-memory servers to process heavy models.

Original: A sequential training of MACH on a single Titan X GPU takes only 7 hours on the ODP dataset.
Revised: A sequential training of MACH on a single Titan X GPU takes only 7 hours on the ODP dataset.

Original: As MACH is trivially parallelizable, the same training time becomes 17 minutes with 25 GPUs.
Revised: As MACH is trivially parallelizable, the same training time becomes 17 minutes with 25 GPUs.

Original: Similarly, we can train linear classiﬁers on ﬁne-grained imagenet features, with 21000 categories, in around 23 hours on a single titan X and around an hour using 20 GPUs.
Revised: Similarly, we can train linear classiﬁers on ﬁne-grained imagenet features, with 21000 categories, in around 23 hours on a single titan X and around an hour using 20 GPUs.

Original: To the best of our knowledge, this is the ﬁrst work to demonstrate training of linear classiﬁers over 100,000 classes on a single GPU, and the ﬁrst demonstration of training ﬁne-grained Imagenet, with 21000 classes, on a single GPU.
Revised: To the best of our knowledge, this is the ﬁrst work to demonstrate training of linear classiﬁers over 100,000 classes on a single GPU, and the ﬁrst demonstration of training ﬁne-grained Imagenet, with 21000 classes, on a single GPU.

Original: These new results illustrate the power of MACH for extremeclassiﬁcation scenarios.
Revised: These new results illustrate the power of MACH for extremeclassiﬁcation scenarios.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We will use the standard [] for integer range, i.e., [ l ] denotes the integer set from 1 to l : [ l ] = { 1 , 2 , · · · , l } .
Revised: We will use the standard [] for integer range, i.e., [ l ] denotes the integer set from 1 to l : [ l ] = { 1 , 2 , · · · , l } .

Original: We will use the standard logistic regressions settings.
Revised: We will use the standard logistic regressions settings.

Original: The data D is given by
Revised: The data D is given by

Original: D =
Revised: D =

Original: ( x i , y i )
Revised: ( x i , y i )

Original: Ni =1 .
Revised: Ni =1 .

Original: x i ∈ R
Revised: x i ∈ R

Original: d will be d dimensional features and y i ∈ { 1 , 2 , · · · , K } , where K denote the number of categories in the multi-class classiﬁcation problem.
Revised: d will be d dimensional features and y i ∈ { 1 , 2 , · · · , K } , where K denote the number of categories in the multi-class classiﬁcation problem.

Original: We will drop the subscript i for simplicity whenever we are talking about a generic data and only use ( x, y ) .
Revised: We will drop the subscript i for simplicity whenever we are talking about a generic data and only use ( x, y ) .

Original: Given an x , we will denote the probability of y (label) taking the value i , under the given classiﬁer model, as p i =
Revised: Given an x , we will denote the probability of y (label) taking the value i , under the given classiﬁer model, as p i =

Original: Pr ( y = i | x ) .
Revised: Pr ( y = i | x ) .

Original: 2. 2-U NIVERSAL H ASHING
Revised: 2. 2-U NIVERSAL H ASHING

Original: Deﬁnition: A randomized function h : [ l ] → [ B ] is 2-universal if for all, i, j ∈ [ l ] with i (cid:54) =
Revised: Deﬁnition: A randomized function h : [ l ] → [ B ] is 2-universal if for all, i, j ∈ [ l ] with i (cid:54) =

Original: j , we have the following property for any z 1 , z 2 ∈
Revised: j , we have the following property for any z 1 , z 2 ∈

Original: [ k ]
Revised: [ k ]

Original: Pr ( h ( i ) =
Revised: Pr ( h ( i ) =

Original: z 1 and h ( j ) =
Revised: z 1 and h ( j ) =

Original: z 2 )
Revised: z 2 )

Original: = 1 B (1)
Revised: = 1 B (1)

Original: (Carter & Wegman, 1977) showed that the simplest way to create a 2-universal hashing scheme is to pick a prime number p ≥ B , sample two random numbers a, b uniformly in the range [0 , p ] and compute h ( x )
Revised: (Carter & Wegman, 1977) showed that the simplest way to create a 2-universal hashing scheme is to pick a prime number p ≥ B , sample two random numbers a, b uniformly in the range [0 , p ] and compute h ( x )

Original: =
Revised: =

Original: (( ax + b ) mod p ) mod B
Revised: (( ax + b ) mod p ) mod B

Original:  M ERGED -A VERAGED C LASSIFIERS VIA H ASHING (MACH)
Revised:  M ERGED -A VERAGED C LASSIFIERS VIA H ASHING (MACH)

Original: We present Merged-Averaged Classiﬁers via Hashing (MACH), which is the ﬁrst algorithm in the classiﬁcation literature that shows the power of universal hashing in reducing both computations as well as memory in extreme multi-class classiﬁcation problems.
Revised: We present Merged-Averaged Classiﬁers via Hashing (MACH), which is the ﬁrst algorithm in the classiﬁcation literature that shows the power of universal hashing in reducing both computations as well as memory in extreme multi-class classiﬁcation problems.

Original: Our idea is simple and comes with sound theoretical guarantees.
Revised: Our idea is simple and comes with sound theoretical guarantees.

Original: Our procedure randomly merges classes into a small, manageable number of random-meta-classes (or buckets).
Revised: Our procedure randomly merges classes into a small, manageable number of random-meta-classes (or buckets).

Original: We then run any off-the-shelf classiﬁer, such as logistic regression or deep networks, on this meta-class classiﬁcation problem.
Revised: We then run any off-the-shelf classiﬁer, such as logistic regression or deep networks, on this meta-class classiﬁcation problem.

Original: We then repeat the process independently R
Revised: We then repeat the process independently R

Original: =
Revised: =

Original: O (log K ) times each time using an independent 2-universal hashing scheme.
Revised: O (log K ) times each time using an independent 2-universal hashing scheme.

Original: During prediction, we aggregate the output from each of the R classiﬁers models to obtain the predicted class.
Revised: During prediction, we aggregate the output from each of the R classiﬁers models to obtain the predicted class.

Original: We later show that this simple scheme is theoretically sound and only needs log K memory.
Revised: We later show that this simple scheme is theoretically sound and only needs log K memory.

Original: We present information theoretic connections of our scheme with compressed sensing and heavy hitters literature.
Revised: We present information theoretic connections of our scheme with compressed sensing and heavy hitters literature.

Original: Formally, we use R , independently chosen, 2-universal hash functions h i :
Revised: Formally, we use R , independently chosen, 2-universal hash functions h i :

Original: [ K ] → [ B ] , i = { 1 , 2 , · · · , R } .
Revised: [ K ] → [ B ] , i = { 1 , 2 , · · · , R } .

Original: Each h i uniformly maps the classes (total K classes) into one of the B buckets.
Revised: Each h i uniformly maps the classes (total K classes) into one of the B buckets.

Original: B and R are our parameters that we can tune to trade accuracy with both computations and memory.
Revised: B and R are our parameters that we can tune to trade accuracy with both computations and memory.

Original: B is usually a small constant like 10 or 50 .
Revised: B is usually a small constant like 10 or 50 .

Original: Given the data { x i , y i } Ni =1 , it is convenient to visualize, that each hash function h j , transforms the data D into D j = { x i , h j ( y i ) } Ni =1 .
Revised: Given the data { x i , y i } Ni =1 , it is convenient to visualize, that each hash function h j , transforms the data D into D j = { x i , h j ( y i ) } Ni =1 .

Original: We do not have to materialize the hashed class values for all small datasets, we can simply access the class values through h j .
Revised: We do not have to materialize the hashed class values for all small datasets, we can simply access the class values through h j .

Original: We then train R classiﬁers (models) on each of these D j ’s to get R (small) models M j s.
Revised: We then train R classiﬁers (models) on each of these D j ’s to get R (small) models M j s.

Original: This concludes our training process.
Revised: This concludes our training process.

Original: Note each h j is independent.
Revised: Note each h j is independent.

Original: Training R classiﬁers is trivially parallelizable across R machines or GPUs.
Revised: Training R classiﬁers is trivially parallelizable across R machines or GPUs.

Original: We need a few more notations.
Revised: We need a few more notations.

Original: Each meta-classiﬁer can only classify among the merged metaclasses.
Revised: Each meta-classiﬁer can only classify among the merged metaclasses.

Original: Let us denote the probability of the meta-class b ∈ [ B ] , with the j th classiﬁer with capitalized P jb .
Revised: Let us denote the probability of the meta-class b ∈ [ B ] , with the j th classiﬁer with capitalized P jb .

Original: If the meta-class contains the class i , i.e. h j ( i ) =
Revised: If the meta-class contains the class i , i.e. h j ( i ) =

Original: b , then we can also write it as P jh j ( i ) .
Revised: b , then we can also write it as P jh j ( i ) .

Original: We are now ready to describe the prediction phase.
Revised: We are now ready to describe the prediction phase.

Original: During prediction, we show (in Section 3.1) that the following expression is an good estimator of the probability of class i , given any feature vector x .
Revised: During prediction, we show (in Section 3.1) that the following expression is a good estimator of the probability of class i , given any feature vector x .

Original: [Equation]
Revised: [Equation]

Original: here P jh j ( i ) ( x ) is the predicted probability of meta-class h j ( i ) under the j th model ( M j ), for given x .
Revised: here P jh j ( i ) ( x ) is the predicted probability of meta-class h j ( i ) under the j th model ( M j ), for given x .

Original: Thus, our classiﬁcation rule is given by arg max i Pr ( y = i | x ) =
Revised: Thus, our classiﬁcation rule is given by arg max i Pr ( y = i | x ) =

Original: arg max i (cid:80) R j =1 P j h j ( i ) ( x ) .
Revised: arg max i (cid:80) R j =1 P j h j ( i ) ( x ) .

Original: In other words, during prediction, for every class i , we get the probability assigned to associated metaclass P jh j ( i ) ( x ) , by the j th classiﬁer on x .
Revised: In other words, during prediction, for every class i , we get the probability assigned to associated metaclass P jh j ( i ) ( x ) , by the j th classiﬁer on x .

Original: We compute the sum of these assigned probabilities, where the summation is over j .
Revised: We compute the sum of these assigned probabilities, where the summation is over j .

Original: Our predicted class is the class with the highest value of this sum.
Revised: Our predicted class is the class with the highest value of this sum.

Original: The overall details are summarized in algorithm 1 and 2.
Revised: The overall details are summarized in algorithm 1 and 2.

Original: Clearly, the total model size of MACH is BRd to store R models of size Bd each.
Revised: Clearly, the total model size of MACH is BRd to store R models of size Bd each.

Original: The prediction cost requires RBd multiplications to get meta probabilities, followed by KR to compute equationfor each of the classes, the maximum can be computed on the ﬂy.
Revised: The prediction cost requires RBd multiplications to get meta probabilities, followed by KR to compute equationfor each of the classes, the maximum can be computed on the ﬂy.

Original: Thus, the total cost of prediction is RBd + KR .
Revised: Thus, the total cost of prediction is RBd + KR .

Original: 3. A NALYSIS
Revised: 3. A NALYSIS

Original: No Assumptions: We would like to emphasize that we do not make any assumption on the classes, and we do not assume any dependencies between them.
Revised: No Assumptions: We would like to emphasize that we do not make any assumption on the classes, and we do not assume any dependencies between them.

Original: It is therefore not even clear how we can go from K to log K .
Revised: It is therefore not even clear how we can go from K to log K .

Original: As noted before, we use R independent B -class classiﬁers each.
Revised: As noted before, we use R independent B -class classiﬁers each.

Original: Classiﬁcation algorithms such as logistic regression and deep networks models the probability Pr
Revised: Classiﬁcation algorithms such as logistic regression and deep networks models the probability Pr

Original: ( y = i | x )
Revised: ( y = i | x )

Original: = p i .
Revised: = p i .

Original: For example, the famous softmax or logistic modelling uses Pr ( y = i | x ) =
Revised: For example, the famous softmax or logistic modelling uses Pr ( y = i | x ) =

Original: e θi · x Z , where Z is the partition function.
Revised: e θi · x Z , where Z is the partition function.

Original: With MACH, we use R 2-universal hash functions.
Revised: With MACH, we use R 2-universal hash functions.

Original: For every hash function j , we instead model
Revised: For every hash function j , we instead model

Original: Pr ( y = b | x ) =
Revised: Pr ( y = b | x ) =

Original: P jb , where b ∈ [ B ] .
Revised: P jb , where b ∈ [ B ] .

Original: Since b is a meta-class, we can also write P jb as
Revised: Since b is a meta-class, we can also write P jb as

Original: P bj = (cid:88) i : h j ( i )=
Revised: P bj = (cid:88) i : h j ( i )=

Original: b p i ; 1 =
Revised: b p i ; 1 =

Original: K (cid:88) i = p i = (cid:88)
Revised: K (cid:88) i = p i = (cid:88)

Original: b ∈ [ B ]
Revised: b ∈ [ B ]

Original: P bj ∀ j (3)
Revised: P bj [Equation] j (3)

Original: Data: D =
Revised: Data: D =

Original: ( X, Y )
Revised: ( X, Y )

Original: =
Revised: =

Original: ( x i , y i )
Revised: ( x i , y i )

Original: Ni =1 .
Revised: Ni =1 .

Original: x i ∈ R d y i ∈ { 1 , 2 , · · · , K }
Revised: x i ∈ R d y i ∈ { 1 , 2 , · · · , K }

Original: Input : B, R
Revised: Input : B, R

Original: Output: R trained multi-class classiﬁers initialize R 2-universal hash functions h 1 , h 2 , ...h R initialize result as an empty list for i ← 1 to R do
Revised: Output: R trained multi-class classiﬁers initialize R 2-universal hash functions h 1 , h 2 , ...h R initialize result as an empty list for i ← 1 to R do

Original: Y h i ← h
Revised: Y h i ← h

Original: i ( Y ) M i =
Revised: i ( Y ) M i =

Original: trainLogistic
Revised: trainLogistic

Original: ( X, Y h i )
Revised: ( X, Y h i )

Original: Append M i to result endreturn result
Revised: Append M i to result endreturn result

Original: Algorithm 2: Predict
Revised: Algorithm 2: Predict

Original: Data: D = (  x i , y i )
Revised: Data: D = ( X, Y ) = ( x i , y i )

Original: Ni =1 .
Revised: Ni =1 .

Original: x i ∈ R d y i ∈ { 1 , 2 , · · · , K }
Revised: x i ∈ R d y i ∈ { 1 , 2 , · · · , K }

Original: Input : M = M 1 , M 2 , ..., M R
Revised: Input : M = M 1 , M 2 , ..., M R

Original: Output: N predicted labels load R 2-universal hash functions
Revised: Output: N predicted labels load R 2-universal hash functions

Original: h 1 , h 2 , ...h R used in training initialize P as a an empty list initialize G as a ( | N | ∗ K ) matrix for i ← 1 to R do
Revised: h 1 , h 2 , ...h R used in training initialize P as a an empty list initialize G as a ( | N | ∗ K ) matrix for i ← 1 to R do

Original: P i =
Revised: P i =

Original: getProbability ( X, M i )
Revised: getProbability ( X, M i )

Original: Append P i to P endfor j ← 1 to K do /* G [: , j ] indicates the j th column in matrix G */
Revised: Append P i to P endfor j ← 1 to K do /* G [: , j ] indicates the j th column in matrix G */

Original: G [: , j ] =
Revised: G [: , j ] =

Original: ( (cid:80)
Revised: ( (cid:80)

Original: Rr =1 P r
Revised: Rr =1 P r

Original: [: , h r ( j )]) /R endreturn argmax(G, axis [Equation]
Revised: [: , h r ( j )]) /R endreturn argmax

Original: With the above equation, given the R classiﬁer models, an unbiased estimator of p i is:
Revised: With the above equation, given the R classiﬁer models, an unbiased estimator of p i is:

Original: Theorem
Revised: Theorem

Original: [Equation]
Revised: [Equation]

Original: Proof: Since the hash function of universal, we can always write
Revised: Proof: Since the hash function of universal, we can always write

Original: [Equation]
Revised: [Equation]

Original: where 1 h ( j )=
Revised: where 1 h ( j )=

Original: h ( i ) is an indicator random variable with expected value of 1 B .
Revised: h ( i ) is an indicator random variable with expected value of 1 B .

Original: For a d dimensional dataset (or d non-zeros for sparse data), the memory required by a vanilla logistic regression model (or any linear classiﬁer) is O
Revised: For a d dimensional dataset (or d non-zeros for sparse data), the memory required by a vanilla logistic regression model (or any linear classiﬁer) is O

Original: ( Kd ) .
Revised: ( Kd ) .

Original: O ( Kd ) is also the computational complexity of prediction.
Revised: O ( Kd ) is also the computational complexity of prediction.

Original: With MACH, the computational, as well as the memory complexity, is dependent on B and R roughly as O ( BR + KR ) .
Revised: With MACH, the computational, as well as the memory complexity, is dependent on B and R roughly as O ( BR + KR ) .

Original: To obtain signiﬁcant savings, we want BR to be signiﬁcantly smaller than Kd .
Revised: To obtain signiﬁcant savings, we want BR to be signiﬁcantly smaller than Kd .

Original: We next show that BR ≈ O (log K ) is sufﬁcient for identifying the ﬁnal class with high probability.
Revised: We next show that BR ≈ O (log K ) is sufﬁcient for identifying the ﬁnal class with high probability.

Original: Deﬁnition 1 Indistinguishable Class Pairs:
Revised: Deﬁnition 1 Indistinguishable Class Pairs:

Original: Given any two classes c 1 and c 2 ∈ [ K ] , they are indistinguishable under MACH if they fall in the same meta-class for all the R hash functions, i.e., h j ( c 1 ) =
Revised: Given any two classes c 1 and c 2 ∈ [ K ] , they are indistinguishable under MACH if they fall in the same meta-class for all the R hash functions, i.e., h j ( c 1 ) =

Original: h j ( c 2 ) for all j ∈ [ R ] .
Revised: h j ( c 2 ) for all j ∈ [ R ] .

Original: Otherwise, there is at least one classiﬁer which provides discriminating information between them.
Revised: Otherwise, there is at least one classiﬁer which provides discriminating information between them.

Original: Given that the only sources of randomness are the independent 2-universal hash functions, we can have the following lemma
Revised: Given that the only sources of randomness are the independent 2-universal hash functions, we can have the following lemma

Original: Lemma 1 MACH with R independent B -class classiﬁer models, any two original classes c 1 and c∈ [ K ] will be indistinguishable with probability at most
Revised: Lemma 1 MACH with R independent B -class classiﬁer models, any two original classes c 1 and c∈ [ K ] will be indistinguishable with probability at most

Original: [Equation]
Revised: [Equation]

Original: There are total K ( K − 1)≤ K 2 possible pairs, and therefore, the probability that there exist at least one pair of classes, which is indistinguishable under MACH is given by the union bound as
Revised: There are total K ( K − 1)≤ K 2 possible pairs, and therefore, the probability that there exist at least one pair of classes, which is indistinguishable under MACH is given by the union bound as

Original: [Equation]
Revised: [Equation]

Original: Thus, all we need is K 2 (cid:18) 1 B (cid:19) R ≤ δ to ensure that there is no indistinguishable pair with probability ≥ 1 − δ .
Revised: Thus, all we need is K 2 (cid:18) 1 B (cid:19) R ≤ δ to ensure that there is no indistinguishable pair with probability ≥ 1 − δ .

Original: Overall, we get the following theorem:
Revised: Overall, we get the following theorem:

Original: Theorem 2 For any B , R = 2log K √ δ log B , guarantees that all pairs of classes
Revised: Theorem 2 For any B , R = 2log K √ δ log B , guarantees that all pairs of classes

Original: c i and c j are distinguishable (not indistinguishable) from each other with probability greater than 1 − δ .
Revised: c i and c j are distinguishable (not indistinguishable) from each other with probability greater than 1 − δ .

Original: From section 3, our memory cost is BRd to guarantee all pair distinguishably with probability 1 − δ , which is equal to 2log K √ δ log B Bd .
Revised: From section 3, our memory cost is BRd to guarantee all pair distinguishably with probability 1 − δ , which is equal to 2log K √ δ log B Bd .

Original: This holds for any constant value of B ≥ 2 .
Revised: This holds for any constant value of B ≥ 2 .

Original: Thus, we bring the dependency on memory from O ( Kd ) to O (log Kd ) in general with approximations.
Revised: Thus, we bring the dependency on memory from O ( Kd ) to O (log Kd ) in general with approximations.

Original: Our inference cost is 2log K √ δ log B Bd + 2log K √ δ log B K which is O
Revised: Our inference cost is 2log K √ δ log B Bd + 2log K √ δ log B K which is O

Original: ( K log K + d log K ) , which for high dimensional dataset can be signiﬁcantly smaller than Kd .
Revised: ( K log K + d log K ) , which for high dimensional dataset can be signiﬁcantly smaller than Kd .

Original: 3. C ONNECTIONS WITH C OMPRESSED S ENSING AND H EAVY H ITTERS
Revised: 3. C ONNECTIONS WITH C OMPRESSED S ENSING AND H EAVY H ITTERS

Original: Given a data instance x , a generic classiﬁer outputs the probabilities p i  i ∈ { 1 , 2 , ..., K } .
Revised: Given a data instance x , a generic classiﬁer outputs the probabilities p i , i ∈ { 1 , 2 , ..., K } .

Original: We can then use these K numbers (probabilities) for inference and training.
Revised: We can then use these K numbers (probabilities) for inference and training.

Original: What we want is essentially to compress the information of these K numbers to log K , i.e., we can only keep track of log K = BR numbers (or measurements).
Revised: What we want is essentially to compress the information of these K numbers to log K , i.e., we can only keep track of log K = BR numbers (or measurements).

Original: Ideally, without any assumption, we cannot compress the information in K numbers to anything less than Ω( K ) , if we want to retain all information.
Revised: Ideally, without any assumption, we cannot compress the information in K numbers to anything less than Ω( K ) , if we want to retain all information.

Original: However, in classiﬁcation, the most informative quantity is the identity of arg max p i .
Revised: However, in classiﬁcation, the most informative quantity is the identity of arg max p i .

Original: If we assume that max p i ≥ 1 m , for sufﬁciently small m ≤ K , which should be true for any good classiﬁer.
Revised: If we assume that max p i ≥ 1 m , for sufﬁciently small m ≤ K , which should be true for any good classiﬁer.

Original: Then, identifying arg max p i with (cid:80) p i = 1 and max p i ≥ 1 m (cid:80) p i is a classical heavy hitter problem.
Revised: Then, identifying arg max p i with (cid:80) p i = 1 and max p i ≥ 1 m (cid:80) p i is a classical heavy hitter problem.

Original: Finding heavy hitters in the data stream is more than 30 years old problem with a rich literature.
Revised: Finding heavy hitters in the data stream is more than 30 years old problem with a rich literature.

Original: Count-Min sketch (Cormode & Muthukrishnan, 2005) is the most popular algorithm for solving heavy hitters over positive streams.
Revised: Count-Min sketch (Cormode & Muthukrishnan, 2005) is the most popular algorithm for solving heavy hitters over positive streams.

Original: Our method of using R universal hashes with B range is precisely the count-min sketch measurements, which we know preserve sufﬁcient information to identify heavy hitters (or sparsity) under good signal-to-noise ratio.
Revised: Our method of using R universal hashes with B range is precisely the count-min sketch measurements, which we know preserve sufﬁcient information to identify heavy hitters (or sparsity) under good signal-to-noise ratio.

Original: Thus, if max p i (signal) is larger than p j , j (cid:54) =
Revised: Thus, if max p i (signal) is larger than p j , j (cid:54) =

Original: i (noise), then we should be able to identify the heavy co-ordinates (sparsity structure) in sparsity × log K measurements (or memory) (Candes & Tao, 2006).
Revised: i (noise), then we should be able to identify the heavy co-ordinates (sparsity structure) in sparsity × log K measurements (or memory) (Candes & Tao, 2006).

Original: Our work provides the ﬁrst bridge that connects compressed sensing and sketching algorithms from the data streams literature with classical extreme K -class classiﬁcation.
Revised: Our work provides the ﬁrst bridge that connects compressed sensing and sketching algorithms from the data streams literature with classical extreme K -class classiﬁcation.

Original: This connection also emphasizes that fundamental advancements in solving computational problems go way farther.
Revised: This connection also emphasizes that fundamental advancements in solving computational problems go way farther.

Original: Name
Revised: Name

Original: Type #Train / #Test #Classes
Revised: Type #Train / #Test #Classes

Original: #Features size
Revised: #Features size

Original: ODP
Revised: ODP

Original: Text 1084404 /  3GB(sparse format)
Revised: Text 1084404 /  3GB(sparse format)

Original: Imagenet Images 12777062 /  104GB(Compressed)
Revised: Imagenet Images 12777062 /  104GB(Compressed)

Original:  E VALUATIONS 4. D ATASETS
Revised:  E VALUATIONS 4. D ATASETS

Original: We use the two large public benchmarks from (Daume III et al., 2016): 1) ODP dataset and 2)
Revised: We use the two large public benchmarks from (Daume III et al., 2016): 1) ODP dataset and 2)

Original: Fine-grained Imagenet.
Revised: Fine-grained Imagenet.

Original: ODP is a multi-class dataset extracted from Open Directory Project, the largest, most comprehensive human-edited directory of the Web.
Revised: ODP is a multi-class dataset extracted from Open Directory Project, the largest, most comprehensive human-edited directory of the Web.

Original: Each sample in the dataset is a document, and the feature representation is bag-of-words.
Revised: Each sample in the dataset is a document, and the feature representation is bag-of-words.

Original: The class label is the category associated with the document.
Revised: The class label is the category associated with the document.

Original: The dataset is obtained from (Choromanska & Langford, 2015).
Revised: The dataset is obtained from (Choromanska & Langford, 2015).

Original: The number of categories in this dataset is 105,033.
Revised: The number of categories in this dataset is 105,033.

Original: Imagenet is a dataset consist of features extracted from an intermediate layer of a convolutional neural network trained on the ILVSRC2012 challenge dataset.
Revised: Imagenet is a dataset consist of features extracted from an intermediate layer of a convolutional neural network trained on the ILVSRC2012 challenge dataset.

Original: This dataset was originally developed to study transfer learning in visual tasks (Oquab et al., 2014).
Revised: This dataset was originally developed to study transfer learning in visual tasks (Oquab et al., 2014).

Original: Please see (Choromanska & Langford, 2015) for more details.
Revised: Please see (Choromanska & Langford, 2015) for more details.

Original: The class label is the ﬁne-grained object category present in the image.
Revised: The class label is the ﬁne-grained object category present in the image.

Original: The number of categories in this dataset is 21,841.
Revised: The number of categories in this dataset is 21,841.

Original: The statistics of the two datasets are summarized in Table 1.These datasets cover different domains – images and text.
Revised: The statistics of the two datasets are summarized in Table 1.These datasets cover different domains – images and text.

Original: The datasets are sufﬁciently hard and demonstrate the challenges of having large classes.
Revised: The datasets are sufﬁciently hard and demonstrate the challenges of having large classes.

Original: There is only one published work (Daume III et al., 2016) which could successfully train a standard one-vs-all linear classiﬁer.
Revised: There is only one published work (Daume III et al., 2016) which could successfully train a standard one-vs-all linear classiﬁer.

Original: The training was done over a sophisticated distributed machine and the training time spanned several days.
Revised: The training was done over a sophisticated distributed machine and the training time spanned several days.

Original: All the experiments are performed on the same server with GeForce GTX TITAN X, Intel(R)
Revised: All the experiments are performed on the same server with GeForce GTX TITAN X, Intel(R)

Original: Core(TM) i7-5960X 8-core CPU @
Revised: Core(TM) i7-5960X 8-core CPU @

Original: 3.00GHz and 64GB memory.
Revised: 3.00GHz and 64GB memory.

Original: We used Tensorﬂow to train each individual model M i and obtain the probability matrix P i from model M i .
Revised: We used Tensorﬂow to train each individual model M i and obtain the probability matrix P i from model M i .

Original: We use OpenCL to compute the global score matrix that encodes the score for all classes [1 , K ] in testing datapoint and perform argmax to ﬁnd the predicted class.
Revised: We use OpenCL to compute the global score matrix that encodes the score for all classes [1 , K ] in testing data and perform argmax to ﬁnd the predicted class.

Original: We have our codes and scripts ready for release on Github for reproducibility.
Revised: We have our codes and scripts ready for release on Github for reproducibility.

Original: 4. A CCURACY B ASELINES
Revised: 4. A CCURACY B ASELINES

Original: On these large benchmarks there  three published methods that have reported successful evaluations – 1) OAA, traditional one-vs-all classiﬁers, 2) LOMTree and 3) Recall Tree.
Revised: On these large benchmarks there are three published methods that have reported successful evaluations – 1) OAA, traditional one-vs-all classiﬁers, 2) LOMTree and 3) Recall Tree.

Original: The result of allthese
Revised: The result of all

Original: [Table]
Revised: [Table]

Original:  methods are taken from (Daume III et al., 2016).
Revised: these methods are taken from (Daume III et al., 2016).

Original: OAA is the standard one-vs-all classiﬁers whereas LOMTree and Recall Tree are tree-based methods to reduce the computational cost of prediction at the cost of increased model size.
Revised: OAA is the standard one-vs-all classiﬁers whereas LOMTree and Recall Tree are tree-based methods to reduce the computational cost of prediction at the cost of increased model size.

Original: Recall Tree uses twice as much model size compared to OAA.
Revised: Recall Tree uses twice as much model size compared to OAA.

Original: Even LOMtree has signiﬁcantly more parameters than OAA.
Revised: Even LOMtree has signiﬁcantly more parameters than OAA.

Original: Thus, our proposal MACH is the only method that reduces the model size compared to OAA.
Revised: Thus, our proposal MACH is the only method that reduces the model size compared to OAA.

Original: We will primarily use these baselines to contrast the accuracy-memory tradeoff provided by MACH.
Revised: We will primarily use these baselines to contrast the accuracy-memory tradeoff provided by MACH.

Original: Our system is currently not capable of running any of the above methods.
Revised: Our system is currently not capable of running any of the above methods.

Original: For example on ODP datasets, all these methods will require more than 160GB model ﬁle, while the memory capacity of our system is 64GB only.
Revised: For example on ODP datasets, all these methods will require more than 160GB model ﬁle, while the memory capacity of our system is 64GB only.

Original: However, since we are using the same partitions, we can directly use the reported accuracy as benchmarks to compare with.
Revised: However, since we are using the same partitions, we can directly use the reported accuracy as benchmarks to compare with.

Original: 4. R ESULTS AND D ISCUSSIONS
Revised: 4. R ESULTS AND D ISCUSSIONS

Original: We run MACH on these two datasets varying B and R .
Revised: We run MACH on these two datasets varying B and R .

Original: B and R are two knobs in MACH to balance resource and accuracy.
Revised: B and R are two knobs in MACH to balance resource and accuracy.

Original: We used plain logistic regression classiﬁer, i.e., cross entropy loss without any regularization, in the Tensorﬂow environment (Abadi et al., 2016).
Revised: We used plain logistic regression classiﬁer, i.e., cross entropy loss without any regularization, in the Tensorﬂow environment (Abadi et al., 2016).

Original: We plot the accuracy as a function of different values of B and R in Figure 1.
Revised: We plot the accuracy as a function of different values of B and R in Figure 1.

Original: The plots show that for ODP dataset MACH can even surpass OAA achieving 18% accuracy while the best-known accuracy on this partition is only 9%.
Revised: The plots show that for ODP dataset MACH can even surpass OAA achieving 18% accuracy while the best-known accuracy on this partition is only 9%.

Original: LOMtree and Recall Tree can only achieve 6-6.5% accuracy.
Revised: LOMtree and Recall Tree can only achieve 6-6.5% accuracy.

Original: It should be noted that with 100,000 classes, a random accuracy is 10 − 5 .
Revised: It should be noted that with 100,000 classes, a random accuracy is 10 − 5 .

Original: Thus, the improvements are staggering with MACH.
Revised: Thus, the improvements are staggering with MACH.

Original: Even with B = 32 and R = 25 , we can obtain more than 15% accuracy with 105 , 000 32 × 25 = 120 times reduction in the model size.
Revised: Even with B = 32 and R = 25 , we can obtain more than 15% accuracy with 105 , 000 32 × 25 = 120 times reduction in the model size.

Original: Thus, OAA needs 160GB model size, while we only need around 1.2GB.
Revised: Thus, OAA needs 160GB model size, while we only need around 1.2GB.

Original: To get the same accuracy as OAA, we only need
Revised: To get the same accuracy as OAA, we only need

Original: R = 50 and B = 4 , which is a 480x reduction in model size requiring mere 0.3GB model ﬁle.
Revised: R = 50 and B = 4 , which is a 480x reduction in model size requiring mere 0.3GB model ﬁle.

Original: We believe that randomization and averaging in MACH cancel the noise and lead to better generalization.
Revised: We believe that randomization and averaging in MACH cancel the noise and lead to better generalization.

Original: Another reason for the poor accuracy of other baselines could be due to the use of
Revised: Another reason for the poor accuracy of other baselines could be due to the use of

Original: VW (Langford et al., 2007) platforms.
Revised: VW (Langford et al., 2007) platforms.

Original: VW platform inherently uses feature hashing that may lose some information in features, which is critical for high dimensional datasets such as ODP.
Revised: VW platform inherently uses feature hashing that may lose some information in features, which is critical for high dimensional datasets such as ODP.

Original: On Imagenet dataset, MACH can achieve around 11% which is roughly the same accuracy of
Revised: On Imagenet dataset, MACH can achieve around 11% which is roughly the same accuracy of

Original: LOMTree and Recall Tree while using R = 20 and B = 512 .
Revised: LOMTree and Recall Tree while using R = 20 and B = 512 .

Original: With R = 20 and B = 512 , the memory requirement is 21841512 × 20 = 2 times less than that of OAA.
Revised: With R = 20 and B = 512 , the memory requirement is 21841512 × 20 = 2 times less than that of OAA.

Original: On the contrary, Recall Tree and LOMTree use 2x more memory than OAA.
Revised: On the contrary, Recall Tree and LOMTree use 2x more memory than OAA.

Original: OAA achieves the best result of 17% .
Revised: OAA achieves the best result of 17% .

Original: With MACH, we can run at any memory budget.
Revised: With MACH, we can run at any memory budget.

Original: 4. R UNNING T IMES AND P ARALLELISM
Revised: 4. R UNNING T IMES AND P ARALLELISM

Original: All our experiments used a single Titan X to run the combinations shown in Figure 1.
Revised: All our experiments used a single Titan X to run the combinations shown in Figure 1.

Original: There has been no prior work that has reported the run of these two datasets on a single Titan X GPU with 12GB memory.
Revised: There has been no prior work that has reported the run of these two datasets on a single Titan X GPU with 12GB memory.

Original: This is not surprising due to memory limit.
Revised: This is not surprising due to memory limit.

Original: With MACH, training is embarrassingly easy, all we need is to run R small logistic regression which is trivially parallelization over R GPUs.
Revised: With MACH, training is embarrassingly easy, all we need is to run R small logistic regression which is trivially parallelization over R GPUs.

Original: In Table 2, we have compiled the running time of some of the reasonable combination and have shown the training and prediction time.
Revised: In Table 2, we have compiled the running time of some of the reasonable combination and have shown the training and prediction time.

Original: The prediction time includes the work of computing probabilities of meta-classes followed by sequential aggregation of probabilities and ﬁnding the class with the max probability.
Revised: The prediction time includes the work of computing probabilities of meta-classes followed by sequential aggregation of probabilities and ﬁnding the class with the max probability.

Original: It should be noted that if we have R machines then this time would go down by factor of R , due to trivial parallelism of MACH because all classiﬁers are completely independent.
Revised: It should be noted that if we have R machines then this time would go down by factor of R , due to trivial parallelism of MACH because all classiﬁers are completely independent.

Original: Although, we cannot compare the running time with any previous methods because they use different systems.
Revised: Although, we cannot compare the running time with any previous methods because they use different systems.

Original: Nonetheless, the wall clock times are signiﬁcantly faster than the one reported by RecallTree, which is optimized for inference.
Revised: Nonetheless, the wall clock times are signiﬁcantly faster than the one reported by RecallTree, which is optimized for inference.

Original: The most exciting message is that with MACH we can train this intimidating datasets on a single GPU.
Revised: The most exciting message is that with MACH we can train this intimidating datasets on a single GPU.

Original: We hope MACH gets adopted in practice.
Revised: We hope MACH gets adopted in practice.