Paper:
SEBOOST -Boosting Stochastic Learning Using Subspace Optimization Techniques
We present SEBOOST, a technique for boosting the performance of existing stochastic optimization methods. SEBOOST applies a secondary optimization process in the subspace spanned by the last steps and descent directions. The method was inspired by the SESOP optimization method, and has been adapted for the stochastic learning. It can be applied on top of any existing optimization method with no need to tweak the internal algorithm. We show that the method is able to boost the performance of different algorithms, and make them more robust to changes in their hyper-parameters. As the boosting steps of SEBOOST are applied between large sets of descent steps, the additional subspace optimization hardly increases the overall computational burden. We introduce hyper-parameters that control the balance between the baseline method and the secondary optimization process. The method was evaluated on several deep learning tasks, demonstrating significant improvement in performance. Video presentation is given in [15]

Review:
The paper proposes to improve stochastic gradient algorithms by periodically applying an additional optimization in a subspace spanned by a set of previously stored then-locally-optimal directions. It is an adaptation of the Sequential Subspace Optimization (SESOP) framework of Nakiss & Zibulevsky to the minibatch setting. Experiments show that it can improve the performance compared to the baselie stochastic gradient descent algorithms that it is used in conjunction with.
The paper is overall clearly written, but one important aspect of the algorithm remains not sufficiently expounded: how precisely the subspace optimization is carried over. The paper only mentions in passing that it uses conjugate gradient (CG), but a number of points would deserve further clarification: a) is CG done over a *single* larger minibatch? And how precisely is this minibatch chosen. b) specify how you compute/provide the gradients needed for that CG subspace optimization, and what is the associated computational cost c) that optimization is still not convex, so you must be meaning nonlinear-CG? Which version/implementation do you use?   The computational cost *and* additional memory requirement (as this can constitute a practical limitation for large nets) for the subspace optimization would need to be disclosed and made precise.   Since in effect multiple CG optimizations are performed on different minibatches, it would have been welcome to contrast this approach, at least in discussion, with attempts to adapt CG to the stochastic/minibatch case.  Experimental results convincingly show that the approach can give a boost to minibatch-SGD-with-momentum, Nesterov accelerated gradient, and adagrad. But for vanilla SGD at least I would have appreciated to see the use of a learning rate decay rather than a fixed learning rate, and the assurance that all optimization hyperparameters (starting learning rate, momentum, learning rate decay) had been properly hyper-optimized to yield the lowest training error after the number of considered and displayed epochs. Captions in the figure stating that "baseline parameters was set .... which provided good convergence" is not precise enough (what was your criterion for "good convergence" and how extensively were these parameters tuned).  Also why, for an optimization method, do you report only test error in your graphs (all graphs are labeled with "test error")? Wouldn't the average error over the whole *training* set have been more appropriate, as it wouldn't risk conflating optimization and generalization ability? The best would be to show both.  In line 156 you comment Figure 3 saying "SEBOOST is able to significantly improve SGD and NAG and shows improvement over AdaGrad although not as noticeable", but I see exactly the contrary happening in Figure 3 (unnoticeable difference for SGD and NAG)!?! Your statement regarding SGD qualitatively supported by Figure 2 but then this is contradicted by Figure 3...   Further detailed remarks: l 82:"as few parameters as required" plase clarify l 103: "current gradient": specify what is meant precisely by current gradient l 107 "useful to normalize": how precisely, and why is it useful to normalize a *direction*?