Paper:
Deep neural networks (DNN) are widely used in many applications. However, their deployment on edge devices has been difficult because they are resource hungry. Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1-bit. We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values. In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions. Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation. These additions are based on linear operations that are easily implementable into the binary training framework. We show experimental results on CIFAR-10 obtaining an accuracy of 86.5%, on AlexNet and 91.3% with VGG network. On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using AlexNet by a margin of 4% and 2% top-1 accuracy respectively.

Review:
Summary: 
This paper presents three small improvements for training binarized neural networks: (1) a modified straight-through estimator, (2) a novel regularizer to push weights to +/- 1, and (3) the use of scaling factors for the binarized weights. Using the methods presented, the validation accuracies on ImageNet and CIFAR-10 are improved by just under 2 percentage points.
  Pros:
    - Decent improvement in the performance of the binarized network in the end
    - The presented regularizers make sense and seem effective. The modified straight-through estimator seems reasonable as well, although the authors do not compare to recent work with a similar adjustment. 
  Cons:
    - The paper is poorly written and confusing. It reads as if it was written in one pass with no editing or re-writing to clarify contributions or key points, or ensure consistency.
    - While the final numbers are acceptable, the experiments themselves could be stronger and could be presented more effectively.
   - The novelty of the scale factors is questionable.
Questions and comments:
1. How exactly is the SS_\beta activation used? It is entirely unclear from the paper, which contradicts itself in multiple ways. Is SS_\beta used in the forward pass at all for either the weight or activation binarization? Or is only its derivative used in the backward pass? If the latter, then you are not replacing the activation anywhere but are simply using a different straight-through estimator in place of the saturated straight-through estimator (e.g., see [1]).
   (a) At the beginning of Section 3.3, you say that you modify the training procedure by replacing the sign binarization with the SS_\beta activation. This sounds like it is referring to the activation function at each layer; however, the pseudocode says that you are using sign() as the per-layer activation. 
   (b) Further, Figure 4 shows that you are using the SS_\beta function to do weight binarization. However, again, the pseudocode shows that you are using sign() to do the weight binarization. 
2. In [1], the authors used a similar type of straight-through estimator (essentially, the gradient of tanh instead of hard_tanh) and found that to be quite effective. You should compare to their method. Also, it's possible that SS_\beta reduces to tanh for some choice of \beta -- is this true?
3. The use of scale factors seems to greatly increase the number of parameters in the network and thus greatly decrease the compression benefits gained by using binarization, i.e., you require essentially #scale_factors =  a constant factor times the number of actual parameters in the network (since you have a scale factor for each convolutional filter and for each column of each fully-connected layer). As a result of this, what is the actual compression multiplier that your network achieves relative to the original network?
4. For the scale factor, how does yours differ from that used in Rastegari et al. (2016)? It seems the same but you claim that it is a novel contribution of your work. Please clarify.
5. Why did learning \beta not work? What was the behavior? What values of \beta did learning settle on? 
6. I assume that for each layer output y_i = f(W_i x_i), the regularizer is applied as R(y_i) while at the same time y_i is passed to the next layer -- is this correct? The figures do not clearly show this and should be changed to more clearly show how the regularizer is computed and used, particularly in relation to the activation.
7. In the pseudocode:
   (a) What does "mostly bitwise operations" mean? Are some floating point?
   (b) Is this the shift-based batchnorm of Hubara et al. (2016)?
8. For Table 1:
   (a) I assume these are accuracies? The caption should say.
   (b) Why are there no comparisons to the performance of other methods on this dataset?
   (c) Any thoughts as to why your method performs better than the full-precision method on this dataset for VGG?
8. For Table 2:
   (a) Does Table 2 show accuracies on ImageNet? You need to make this clear in the caption.
   (b) What type of behavior do the runs that do not converge show? This seems like a learning rate problem that is easily fixable. Are there no hyperparameter values that allow it to converge?
   (c) What behavior do you see when you use SS_1 or SS_2, i.e., \beta = 1 or \beta = 2? Since lower \beta values seem better.
   (d) The regularization seems to be the most useful contribution -- do you agree?
   (e) Why did you not do any ablations for the scale factor? Please include these as well.
9. For Table 3, did you compute the numbers for the other approaches or did you use the numbers from their papers? Each approach has its own pros and cons. Please be clear.
10. Are there any plots of validation accuracy versus epoch/time for the different algorithms in order to ensure that the reported numbers were not simply cherry-picked from the run? I assume that you simply used the weights from the end of the 50th epoch -- correct? 
11. Is there evidence for your introductory claims that 'quantizing weights ... make neural networks harder to train due to a large number of sign fluctuations' and 'maintaining a global structure to minimize a common cost function is important' ? If so, you should cite this evidence. If not, you should make it clear that these are hypotheses. 
12. Why are there not more details about the particular architectures used? These should be included in the appendices to aid those who would like to rerun your experiments. In general, please include more experiment details in the body or appendices.
Detailed comments:
- R(l) is not defined in Figure 1 and thus is confusing. Also, its replacement of 'Error' from the original figure source makes the figure much more confusing and less clear.
- Typos:
   - 'accustomed' (p.1)
   - 'the speed by quantizing the activation layers' doesn't make sense (p.1)
   - 'obtaining' (p.4)
   - 'asymmetric' doesn't make sense because these are actually symmetric functions across the y-axis (p.4)
   - 'primary difference is that this regularization ...' --> 'primary difference is that their regularization ...' (p.4)
   - 'the scales with 75th percentile of the absolute value ... ' is very confusing and unclear (p.7)
   - 'the loss metric used was the cross-entropy loss, the order of R_1.' I do not know what you're trying to say here (p.8)
- Citations: Fix the capitalization issues, typos, and formatting inconsistencies.
[1]  Friesen and Domingos. Deep Learning as a Mixed Convex-Combinatorial Optimization Problem. ICLR 2018.
-------------------
After reading the author response, I do not think the paper does a sufficient job of evaluating the contributions or comparing to existing work. The authors should run ablation experiments, compare to existing work such as [1], and evaluate on additional datasets. These were easy tasks that could have been done during the review period but were not.
If I wanted to build on top of this paper to train higher accuracy binary networks, I would have to perform all of these tasks myself to determine which contributions to employ and which are unnecessary. As such, the paper is currently not ready for publication.