Paper:
AN EMPIRICAL STUDY OF BINARY NEURAL NETWORKS' OPTIMISATION
Neural networks with deterministic binary weights using the Straight-ThroughEstimator (STE) have been shown to achieve state-of-the-art results, but their training process is not well-founded. This is due to the discrepancy between the evaluated function in the forward path, and the weight updates in the backpropagation, updates which do not correspond to gradients of the forward path. Efficient convergence and accuracy of binary models often rely on careful finetuning and various ad-hoc techniques. In this work, we empirically identify and study the effectiveness of the various ad-hoc techniques commonly used in the literature, providing best-practices for efficient training of binary models. We show that adapting learning rates using second-moment methods is crucial for the successful use of the STE, and that other optimisers can easily get stuck in local minima. We also find that many of the commonly employed tricks are only effective towards the end of the training, with these methods making early stages of the training considerably slower. Our analysis disambiguates necessary from unnecessary ad-hoc techniques for the training of binary neural networks, paving the way for future development of solid theoretical foundations for these. Our newly-found insights further lead to new procedures which make training of existing binary neural networks notably faster.

Review:
The paper systematically studies the training of binary neural networks, where binary in this case refers to single bit weight elements in the network. In particular, different existing training methods are tested and compared for training both MLPs and CNNs.
The main findings of the paper are:
- Using methods such as AdaGrad, AdaDelta, RMSProp and ADAM yields better performance than simpler momentum-based methods such as vanilla momentum and Nesterov momentum, which in turn are much better than vanilla SGD
- When training binary models, it is common to clip weights and/or gradients for the proxy weights in the network. In the paper it is however shown that these methods hinder using a fast learning rate in the beginning of training, while the methods are required in later stages of training in order to achieve good results
- Pre-training the model with full-precision training works well in speeding up training
For a practitioner, the paper presents a very useful reference for what methods work well when training binary networks. Although there are some proposals and hypotheses for reasons behind the results, I see the paper as a review paper of existing methods for training binary networks, showing experiments where the methods are tested using the same benchmark and training procedure in order to give a fair comparison.
As a practical guide, the paper therefore has clear value. What is lacking compared to typical ICLR papers is rigorously presenting new findings. The authors present a hypothesis for why different batch sizes are needed in the beginning compared to the end of training, but I found neither the justification nor the results very convincing with respect to the hypothesis. The way I see it, the actual novel proposals that are made in the paper are two two-stage training methods: one in which the tricks of weight and gradient clipping are only used towards the end of training, and one where the first stage of training is done using a full precision model. It is however quite well known that some training schemes with different stages can lead to improved performance: for instance with ADAM, even if it is an adaptive method, lowering the learning rate towards the end of training is often beneficial. It might therefore be fair to compare the methods to other multi-stage training methods. In addition, I could not find the training curves or final performance figures of the method where clipping is only activated towards the end of training.
To put it all together, the paper is clearly useful for the community as it provides a useful summary of the performance of different methods for training binary neural networks. In addition, it presents two two-stage training schemes that seem to make training even faster. What the paper lacks is rigorous theoretical justifications and clearly novel ideas.
Small comments:
- How are the training lengths decided for the different methods? If I am not mistaken, in Figure 2, it seems like the SGD and momentum methods have not yet converged when training is halted. Is there a budget for wall clock time or is early stopping used or something similar? Considering the nature of the paper, I would see this kind of decisions as important to report.
- In the abstract, you might want to refer to binary weights somewhere. Based on the abstract it is easy to mix the binary networks in this paper with stochastic binary networks that can also be trained using the STE estimator
- Are the differences in the performances in Table 3 statistically significant?