From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper53/AnonReviewer1
comment: Thank you for the clarification. Now, I believe that the idea and terminology are clearly explained, and I revised my score.

From: ~Mehrdad_Yazdani1
comment: Apologies, it appears that I replied to Reviewer 1 comments here. The above are meant for Reviewer 3. We paste our response to Reviewer 3 here (which was accidentally posted for Reviewer 1):


1). We have included Backprop (adopted from the Deep Learning book by Goodfellow and colleagues) and Linear Backprop in Algorithms 1 and 2. We have also highlighted in red the key distinction between these two algorithms. We hope this clears up the confusions in our terminology. 

2) We have included the computational cost savings on line 45. 

3) We have replaced Figure 2 with a more clear figure comparing the two algorithms for 100 epochs and placed the old Figure 2 in the appendix. We feel that the old Figure 2 is still valuable for demonstrating how learning occurs, although unfortunately we have still not managed to find a clear way of demonstrating multiple learning curves and parameters in a clear manner. 

4) Our primary motivation for the Binarized ResNet is to understand how results change when the architecture is changed. Since all of our parameter sweeps for the VGG19 experiments were using 100 epochs we wanted comparison to remain fair. 

Although due to space limitations we have not provided a figure demonstrating Linear Backprop, we hope that placing Linear Backprop and Backprop
title: Wrong comment order!

From: ~Mehrdad_Yazdani1
comment: Apologies, it appears that I replied to Reviewer 3 comments here. The above are meant for Reviewer 3. We paste our response to Reviewer 1 here (which was accidentally posted for Reviewer 3):

We apologize for using sloppy language by interchanging gradients and derivative. and have updated the manuscript to better terminology. What we mean by “gradients are never computed” is more accurately described as activation function derivatives are never computed. 

We thank the reviewer for pointing out the connection to the straight-through estimator. We believe our method may have many points in common with this method and have cited the paper in our work. The distinction of our work is to apply Linear Backprop to any activation function. 

We had not properly included the references from the colleagues from the Courbariaux et al, 2016. Paper and have updated our manuscript to include this implementation. Our primary motivation for the Binarized ResNet is to understand how results change when the architecture is changed. Since all of our parameter sweeps for the VGG19 experiments were using 100 epochs we wanted comparison to remain fair. We also believe this may be of an interest for workshop participants. 
title: Wrong comment order!

From: ~Mehrdad_Yazdani1
comment: Greatly appreciate the immensely constructive feedback and questions. Below we address each question



Since the training set is so small and the number of parameters so large in the example posed by Wu et al. (2017), we believe concluding that Linear Backprop is more generalizable based on our single set of experiments to not provide strong enough evidence. For this reason we did not specifically discuss whether Linear Backprop leads to better solutions than Backprop. We have re-run and shown the distribution in a boxplot in the middle panel of Figure 1. The results have not changed significantly. 


We have decided to replace Figure 2 with re-running the CIFAR 10 experiments with 1000 epochs based on the best learning rate and weight decays for Backprop and Linear Backprop. Our learning curves in this case have more clearly converged so we can more decidedly conclude that Linear Backprop is learning better in this case. 


We apologize by our loose language as we have interchangeably used the terms “derivatives” and “gradients” loosely and have made our language in the paper more tight. What we mean is that the derivatives of the activation functions are never computed. We hope this distinction is also made more clear by explicitly writing down the Backprop and Linear Backprop algorithms on page 2. We have highlighted the distinction between the two algorithm in red in Linear Backprop. 

In terms of computations saved, since we are not computing the derivatives of the activation functions, the number of computes saved is on the order of O(ml) where we have assumed there are m units in each layer and there are l layers total.


The ReLU activation function is one of the remedies of dealing with saturation. For this reason we have re-ran the same experiments with the same hyperparameter search using the ReLU activation function. We have included these results in the appendix as there is no space in the paper. The ReLU activation function has comparable test accuracy however as you can see from the figures the variance of the learning curves using Backprop is significantly higher than using Linear Backprop. A possible explanation may be the regularization interpretation of Linear Backprop that we posited in the paper, but further research is needed.



We have made an error in our loose language and have corrected this issue. We also are thankful for the reference that provides better historical background for our paper. 



In our updated Figure 1, we have placed a histogram of the unit activations for layer 4 in the 5 layer network (very similar results for the other hidden layers) using sigmoidal activation functions. As can be seen from this figure the activation levels learned from Linear Backprop are allowed to venture to much higher regions than using Backprop. We believe this is because learning in Linear Backprop is not slowed when units saturated and hence activations can grow large.  


We have updated Figure 1 to include regions that go outside of the training points. Neither network generalizes gracefully but it appears that the function learned using Linear Backprop generalizes a bit better. 

title: Updates

From: ~Mehrdad_Yazdani1
comment: Many thanks for pointing out our lacking areas and details. 

1). We have included Backprop (adopted from the Deep Learning book by Goodfellow and colleagues) and Linear Backprop in Algorithms 1 and 2. We have also highlighted in red the key distinction between these two algorithms. We hope this clears up the confusions in our terminology. 

2) We have included the computational cost savings on line 45. 

3) We have replaced Figure 2 with a more clear figure comparing the two algorithms for 100 epochs and placed the old Figure 2 in the appendix. We feel that the old Figure 2 is still valuable for demonstrating how learning occurs, although unfortunately we have still not managed to find a clear way of demonstrating multiple learning curves and parameters in a clear manner. 

4) Our primary motivation for the Binarized ResNet is to understand how results change when the architecture is changed. Since all of our parameter sweeps for the VGG19 experiments were using 100 epochs we wanted comparison to remain fair. 

Although due to space limitations we have not provided a figure demonstrating Linear Backprop, we hope that placing Linear Backprop and Backprop

title: Updated with algorithms and new figures and computation costs

From: ~Mehrdad_Yazdani1
comment: Many thanks for the insightful feedback and comments. 

We apologize for using sloppy language by interchanging gradients and derivative. and have updated the manuscript to better terminology. What we mean by “gradients are never computed” is more accurately described as activation function derivatives are never computed. 

We thank the reviewer for pointing out the connection to the straight-through estimator. We believe our method may have many points in common with this method and have cited the paper in our work. The distinction of our work is to apply Linear Backprop to any activation function. 

We had not properly included the references from the colleagues from the Courbariaux et al, 2016. Paper and have updated our manuscript to include this implementation. Our primary motivation for the Binarized ResNet is to understand how results change when the architecture is changed. Since all of our parameter sweeps for the VGG19 experiments were using 100 epochs we wanted comparison to remain fair. We also believe this may be of an interest for workshop participants. 
title: Heavily updated manuscript

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper53/AnonReviewer5
title: well-written paper, critical details are missing
review: The author proposed linear backprop algorithms to ensure gradients flow for all parts during backpropagation. The network still produce nonlinear activations during inference but acts as linear network during back propagation.

The paper is generally well-written and the result seems very promising. However, some critical details are missing. 

The authors claimed that the gradients are “never computed”, which are not supported by any detailed explanation and is contradicted to the claim of “when gradients are computed, all the non-linear components are cancelled and only linear gradients persist”. How does this algorithm actually saves the computation time? 

Also it seems very similar with the “straight-through estimator” which treats all nonlinearity of quantization as linear functions. I hope the authors could explain the connection with it.

There is no Resnet architecture mentioned in the the reference: Courbariaux et al, 2016. How was the binary ResNet trained with backpropagation? Is algorithm like BinaryConnect or STE adopted in training?

rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper53/AnonReviewer2
title: The authors propose Linear Backprop, a modified backpropagation that ignores nonlinearities of activation functions on the backward pass of training. This seems to accelerate training without reducing performance, possibly by ameliorating saturation. More thorough data is necessary, especially to investigate whether this affects the maximum attainable accuracy of trained networks.
review: The authors propose Linear Backprop, a modification of the backpropagation algorithm in which nonlinear activation functions are treated as though they were linear for the purpose of computing network parameters updates.The motivation is to prevent unit saturation, which does not occur with Linear Backprop. As an added bonus, training is more efficient, as the gradients of the activation functions are not evaluated.

The paper contains three experiments comparing Linear Backprop to standard backpropagation (Backprop). First, the authors consider a problem on which Wu et al. (2017) showed that networks trained with Backprop can achieve good generalization even when they are extremely overfitted. The authors reproduce the experiment, and conclude that Linear Backprop does not seem to reduce the tendency for networks to generalize well; I agree with the authors on this. In fact, it appears to me that the data in Fig 1 may actually imply that Linear Backprop is leading to improved generalization. In Fig 1(right), although the Backprop and Linear Backprop results overlap, the max MSE is much higher with Backprop, and the min MSE is much lower with Linear Backprop. Is this a real result? Can the authors add more experiments to Fig 1?

In the second experiment, the authors train a 19 layer VGG architecture on the CIFAR-10 dataset for a variety of learning rates and L2 regularization weights. Networks trained with Linear Backprop consistently achieved better testing accuracy. I agree that Linear Backprop performs better at the beginning of training. However, training was only conducted for 100 epochs, which does not appear to have been long enough for all of the networks to converge. In the left plot of Fig 2, the two best instances are still improving significantly; also, testing accuracy has already peaked in most of the networks trained with Linear Backprop. I am curious whether Linear Backprop reduces the maximum achievable test accuracy. Regardless, Linear Backprop does produce reasonably good solutions much more quickly than Backprop.

The third experiment is similar to the second, but now for ResNet Binarized Neural Networks. In this case, Linear Backprop performs slightly less well than Backprop. Again, training was only conducted for 100 epochs; Courbariaux et al. noted that ResNet Binarized Neural Networks are slow to train. Can the authors comment on the loss vs epoch curves for these experiments? For instance, after 100 epochs, were any of the networks starting to overfit in the manner seen in Fig 2(left)?

In summary, the authors have investigated Linear Backprop as a remedy for unit saturation. This promises to facilitate training without modifying the activation functions, which preserves the representational power of the network. The authors have provided promising results suggesting that Linear Backprop may accelerate training without significantly reducing performance. My primary concern is whether Linear Backprop reduces the maximum attainable accuracy when used to train complicated networks on difficult problems. More data is necessary, but these results are very enticing. I find the work to be novel and impactful for CDNNRIA: Linear Backprop might provide accelerated training at reduced computational cost. 

Some more questions:
- The authors state that in Linear Backprop "gradients are never computed." Are they referring to only the gradients of the activation functions? Aren't many gradients are still being evaluated? Can the authors comment on how many operations (e.g. FLOPS) are avoided by using Linear Backprop? Have the authors conducted any experiments measuring this speed-up?
- Saturation is known to be an impediment to training using Backprop. However, as the authors have discussed, many other techniques have been proposed in the literature for dealing with it. How does Linear Backprop compare to these other techniques?
- On line 17, the authors state that "It is has been long known that deep neural networks with non-linear units are universal function approximators." To my knowledge, the universal approximation theorems apply when the activation functions are non-polynomial; nonlinearity is not sufficient. See for instance Leshno et al. (1993). Can the authors please clarify?
- On line 90, the authors discuss that tanh activation functions are particularly prone to saturation. The authors also implied in the caption of Fig 2 that Linear Backprop requires stronger regularization to keep units in the linear regime. Can the authors quantify the number of saturated units in their experiments?
- The first experiment only considers the generalization of the network between training points, i.e. interpolation. Can the authors comments on extrapolation?

Leshno, M., Ya. Lin, V., Pinkus, A., & Schocken, S. (1993). Multilayer
feedforward networks with a nonpolynomial activation function
can approximate any function. Neural Networks, 6, 861-867.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper53/AnonReviewer1
title: Summary: This paper proposes a modified version of the backpropagation algorithm where the gradient is never computed. The new algorithm is validated on a synthetic dataset and the VGG-16 and ResNet Binarized neural networks were used on others datasets.  
review: 
Key strengths: 
1-    The idea is interesting and novel and can be used as a technique for training deep neural networks on specific hardware 
Key weakness: 
1-    The equation for both Backprop and linear Backprop is not written in the paper. As a result, it’s difficult to understand which part of the equation is removed or modified. It’s not obvious that either the gradients are not computed in the linear Backprop or non-linear components are removed from the computation.
2-    Number of computations or order of algorithm (Big-O notation) for both Backprop and linear Backprop were not mention in the paper.
3-    Figure 2 is messy and should be improved. It’s not clear which of these algorithms obtain a better accuracy 
4-    It is not clear why the author use a binarized ResNet neural network and why the author selected 100 epochs for training?

Recommendation: 
1-    The concept of linear reign is interesting. If you could show this concept with a figure that would be very valuable.

rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature