Paper:
Large Margin Deep Networks for Classification
We present a formulation of deep learning that aims at producing a large margin classifier. The notion of margin, minimum distance to a decision boundary, has served as the foundation of several theoretically profound and empirically successful results for both classification and regression tasks. However, most large margin algorithms are applicable only to shallow models with a preset feature representation; and conventional margin methods for neural networks only enforce margin at the output layer. Such methods are therefore not well suited for deep networks. In this work, we propose a novel loss function to impose a margin on any chosen set of layers of a deep network (including input and hidden layers). Our formulation allows choosing any l p norm (p ≥ 1) on the metric measuring the margin. We demonstrate that the decision boundary obtained by our loss has nice properties compared to standard classification loss functions. Specifically, we show improved empirical results on the MNIST, CIFAR-10 and ImageNet datasets on multiple tasks: generalization from small training sets, corrupted labels, and robustness against adversarial perturbations. The resulting loss is general and complementary to existing data augmentation (such as random/adversarial input transform) and regularization techniques such as weight decay, dropout, and batch norm.

Review:
This paper proposes a new method for encouraging large margin in the classification problems. In particular, this work is a generalization of SVM to deep models. The idea is interesting, and the method has been tested on many datasets and tasks.   This paper has a few problems.  1. The paper should elaborate the optimization of Eqn 9 and 10, at least in the appendix. The optimization of such problems appears to be nontrivial. The short description in Sec 4.1 is not very easy to follow. Also, what is the time complexity of a forward and backward propagation? Is it much more expensive than optimization the cross-entropy loss?  2. Although I am not sure, I think the proposed large-margin method is very similar to adversarial training. The discussion of the similarity and difference may be interesting.  3. In the experiments, is the large-margin formulation (Eqn 10) applied to every layer or just a few selected layers? Also, Eqn 10 has a parameter $\gamma_l$. If the Eqn 10 is applied to many layers, will the hyper-parameter tuning be expensive?  4. As mentioned by Sec 4.1, the proposed method is expensive if there are many classes. The proposed method does not actually apply to the full ImageNet dataset. I am not sure whether the heuristic of using only one class $i \neq y_k$ is a reasonable approach.  5. Line 160 "... computation of second-order derivatives...": Do the 2nd-order derivatives even exist?