Paper:
Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets
An important class of problems involves training deep neural networks with sparse prediction targets of very high dimension D. These occur naturally in e.g. neural language models or the learning of word-embeddings, often posed as predicting the probability of next words among a vocabulary of size D (e.g. 200 000). Computing the equally large, but typically non-sparse D-dimensional output vector from a last hidden layer of reasonable dimension d (e.g. 500) incurs a prohibitive O(Dd) computational cost for each example, as does updating the D × d output weight matrix and computing the gradient needed for backpropagation to previous layers. While efficient handling of large sparse network inputs is trivial, the case of large sparse targets is not, and has thus so far been sidestepped with approximate alternatives such as hierarchical softmax or sampling-based approximations during training. In this work we develop an original algorithmic approach which, for a family of loss functions that includes squared error and spherical softmax, can compute the exact loss, gradient update for the output weights, and gradient for backpropagation, all in O(d 2 ) per example instead of O(Dd), remarkably without ever computing the D-dimensional output. The proposed algorithm yields a speedup of D 4d , i.e. two orders of magnitude for typical sizes, for that critical part of the computations that often dominates the training time in this kind of network architecture.

Review:
Summary:This paper gives a novel technique for gradient-based learning in neural networks with a large but sparse target vector. The technique calculates the exact, correct gradient direction, while never computing the full large-dimensional output, albeit for a restricted class of loss functions. The paper shows, theoretically and empirically, that this drastically improves the computational complexity of gradient descent learning. Experiments document the speedup, but show that the loss function restrictions may negatively impact performance.Major comments:This paper presents a promising technique for speeding up neural language models with large, sparse output vectors. Remarkably, the paper shows how to perform exact gradient updates (for a restricted set of loss functions) in an efficient way that scales in the sparsity of the targets, rather than in the size of the output vector. This paper is not a tweak of previous approaches to dealing with large sparse targets, but a wholly new direction which may inspire a variety of related algorithms and improvements. It will be no surprise to the authors that the paper would be strengthened greatly if it used the proposed method to attain state of the art performance on some task. The experiments in the paper show that the method indeed attains in practice the speedup promised, but this speedup is not leveraged to improve performance. It may be that the loss function restrictions are severe enough to stop this method from being practically useful. Most concerning is the comparison between hierarchical softmax and the proposed method. While H-softmax is only a heuristic, it is about equally fast and performs better than the proposed LST models on the example dataset in the paper. However the theoretical contributions of the paper are already substantial, and even if it takes some time to find a compelling application, the insights given here seem likely to be useful in some future application.The experimental results are also valuable in showing that there are no numerical stability issues (other than the ill-conditioning of U which is identified and addressed).The paper is very clearly written, with the algorithm spelled out step-by-step in useful tables. Additionally, code will be released upon publication, which should also aid clarity and adoption of the method in practice. The supplementary material adds the practically important detail of how to perform mini batch updates. The clarity of the algorithm could be further improved by explicitly including pseudocode for the rank-1 V update used to control numerical stability.Minor comments:pg 2 ln 097 "workFinally" ->"work. Finally" Fig 1 caption: "inO(Kd)"->"in O(Kd)" pg 4 ln 172: "non-lienarity"->"non-linearity" pg 5 ln 258: "theK"->"the K" pg 5 ln 259: "sited" -> "instead" pg 5 ln 261: "theW^Ty"->"the W^Ty" pg 7 ln 329: "appropriate do a rank-1" -> "appropriate rank-1" pg 7 ln 363: I don't believe the LST acronym has been introduced yet at this point in the text. pg 8 fig 2: the right hand plot does not appear to contain all points in the lefthand plot (the red curve never goes above the blue curve, for instance). Citation formats are inconsistent (some have whole name; some abbreviate first names; some say NIPS, some say Advances in Neural...) Citation 4 has a typo