Paper:
Computations for the softmax function in neural network models are expensive when the number of output classes is large. This can become a significant issue in both training and inference for such models. In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse Experts, to improve the efficiency for softmax inference. During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts. Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts. During inference, our method quickly locates the most probable expert to compute small-scale softmax. Our method is learning-based and requires no knowledge of the output class partition space a priori. We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of performance.

Review:
The present paper proposes a fast approximation to the softmax computation when the number of classes is very large. This is typically a bottleneck in deep learning architectures. The approximation is a sparse two-layer mixture of experts.
The paper lacks rigor and the writing is of low quality, both in its clarity and its grammar. See a list of typos below.
An example of lack of mathematical rigor is equation 4 in which the same variable name is used to describe the weights before and after pruning, as if it was computer code instead of an equation. Also pervasive is the use of the asterisk to denote multiplication, again as if it was code and not math.
Algorithm 1 does not include mitosis, which may have an effect on the resulting approximation.
How are the lambda and threshold parameters tuned? The authors mention a validation set, are they just exhaustively explored on a 3D grid on the validation set?
The results only compare with Shim et al. Why only this method? Why would it be expected to be faster than all the other alternatives? Wouldn't similar alternatives like the sparsely gated MoE, D-softmax and adaptive-softmax have chances of being faster?
The column "FLOPS" in the result seems to measure the speedup, whereas the actual FLOPS should be less when the speed increases. Also, a "1x" label seems to be missing in for the full softmax, so that the reference is clearly specified.
All in all, the results show that the proposed method provides a significant speedup with respect to Shim et al., but it lacks comparison with other methods in the literature.
A brief list of typos:
"Sparse Mixture of Sparse of Sparse Experts"
"if we only search right answer"
"it might also like appear"
"which is to design to choose the right"
sparsly
"will only consists partial"
"with γ is a lasso threshold"
"an arbitrarily distance function"
"each 10 sub classes are belonged to one"
"is also needed to tune to achieve"