From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper27/Authors
comment: We also note however that our best performing pruning technique results in compression ratios (that is, the ratio of number of parameters after pruning divided by number before pruning) that reach close to state of the art for the LeNet-300-100 architecture trained for MNIST (and we mention this in the paper). This suggests that our pruning techniques may be useful as well if the network is to be deployed on a general purpose processor. Note that the alternative technique is training without wiring length in mind, and then implementing a specialized circuit for that particular graph (which may have unnecessarily long wires because it was trained and pruned without this in mind). 

As we propose to say in our conclusion in our responses to the previous reviewer: " The results suggest that these techniques may be worth the computational effort if the neural network is to be widely deployed, if significantly lower energy is worth the slight decrease in accuracy, or if the application is to be deployed on either a specialized circuit \emph{or} general purpose processor."

Inspired by this comment, we additionally propose to place the citation (and a very brief discussion) of the Ganesan paper in our introduction so that the reader knows there is experimental as well as theoretical evidence justifying out model.


In addition to all of the experimental justification for our energy model, there is also a far deeper justification, and this is discussed at length in the cited Blake thesis, chapter 8, and the Grover information friction paper: it is that energy is proportional to communication distance not just for the circuit technologies of the day, but also for any conceivable and practical technology that could ever be created in our Universe. Underlying this claim is a basic observation: physical systems have friction. Once there is any friction, communicating information is at least proportional to distance. We propose the wiring length model to capture this fundamental cost.

"L33: "lowest-weight nodes" : Should not this be lowest-weight edges."

We thank the reviewers for noticing this and will update the text for the final submission.

"Table 1 : @98% accuracy. One would imagine that as p increases, the distance-based regularization would be stronger and thus the final energy should be lower (as long-distance connections are then more likely to be pruned). The trend observed does not show a monotonic reduction of energy as p increases, however. Can you please explain?"

We appreciate the reviewers' perceptive question. The answer is, we would not expect that necessarily as p increases final energy at 98% would decrease. It is hard to prove a concrete reason for this (as interpreting the "why" in machine learning is in general difficult), but here are some possibilities. First, a very large p can force the weights on the longer (and shorter) wires to be too low, and thus reduce the accuracy too quickly, because it may be that the most accurate network includes some longer wires. In the extreme this is easy to see. If p was very large, this could force the weights of all the wires to be close to 0, and we could simply lose the expressive power of the network. Moreover, if p is too large, the size of the regularization term can dominate, and then the re-training after each pruning step will modify weights almost entirely to minimize this weight-distance cost term, and not the cross-entropy term which is related to accuracy. Another reason for this phenomenon is possibly due to the randomness of initial conditions. As we see in our table, the energy at 98%: has high variance. We suspect this is mostly due to variance in initial accuracy (and 98% is very close to the initial accuracy). A higher initial accuracy may allow many more pruning iterations before accuracy goes below 98%, since each iteration empirically seems to change the accuracy only slightly.  The appropriate takeaway from this is not to "increase p to decrease energy for the same accuracy" but rather to "use p as a hyperparameter to vary in searching for the lowest energy network."  As we propose to say in our conclusion: "The results suggests that these techniques may be worth the computational effort if the neural network is to be widely deployed."

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper27/Authors
comment:  The fact that different graphs result in different energy specialized circuits also has experimental justification. For example, consider the result of the reference below:

K. Ganesan, P. Grover, and J. Rabaey, The power cost of over-designing codes, in Proc. 2011 IEEE Workshop Signal Proc. Sys., Oct. 2011, pp. 128133

In this reference the authors consider designing a specialized circuit for another widely used algorithm that is characterized by a computation on a graph: an error control coding circuit. In this paper the authors consider two graphs that have similar numbers of parameters but have different girth (which the authors suggest relates to higher layout area). After applying contemporary circuit design techniques to each of the algorithms, the authors showed that  the higher girth graph has a circuit implementation that consumes 36% more energy. The key point here is that the different energy consumptions are not a result of different techniques applied to  laying out the graph (the same design process was used) but rather there was a different technique to design the graph to be laid out in the first place. Analogous to error control coding, finding sparse graphs for image classification will also require algorithmic level techniques to ensure that the produced graph has low area once optimally laid out. Our work herein is the first example of a paper to demonstrate that different algorithms result in different area graphs. The previous literature allows us to make the reasonable extrapolation that these different energy graphs will result in different energy circuit implementations.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper27/Authors
comment: "The paper presents an interesting take on energy-aware network pruning... accuracy level"

-We thank the reviewers for this accurate description of our paper. We should also note that we present experiments pruning the fully connected layers of a convnet for the street view house numbers problem.

"To my knowledge, the work is novel and opens a new frontier when thinking about network pruning. My main criticism is that the logical network graph is not really physically embodied in today's hardware. You will typically have weights fetched from memory to carry out the network computation. Distance between nodes, etc. thus does not make sense in today's hardware. While we know the brain uses point-to-point axonal connections, current machine learning accelerators most often do not. Even compute-in-memory architecture based on novel devices such as memristors (where you have physical persistent connections) typically use crossbars and other rigid arrangements to connect neurons. These arrangements are dictated by physical considerations and will typically not conform to the arbitrary physical arrangement used in the paper. Maybe the authors can comment more as to how relevant their work is to current technology. "

We appreciate the reviewer's apt criticism of our work. At the heart of our motivation for studying the wiring length of the neural network is not because our energy model corresponds exactly to the energy consumption of a specialized circuit; in fact, in many cases it may not as there are many different factors that change circuit consumption. For example, things like leakage current, multiple switches in a given clock cycle, inconsistent logic gate sizes, the strategic use of memory elements with different energy scaling, etc, can all alter the energy consumption of a circuit (see Section 2.2 of the cited Blake thesis for a more thorough discussion of limitation of models like the one we are considering).There is also the problem of circuit technologies changing over time. Each different generation can result in different factors dominating energy consumption. Nonetheless, it is fairly robust (and informed by the laws of physics), that different computational graphs, once implemented in specific technologies, can have different final energies even if the number of parameters are the same. With modern circuit technologies, that energy is dominated by total wiring length is a result of the energy to charge and discharge a capacitor being proportional to its capacitance, which is proportional to its area, which is proportional to total wiring length. This is also why, as a rule of thumb, larger circuits consume proportionally more energy: this is because they take more area and thus have higher capacitance, a fraction of which is charged and discharged during each clock cycle.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Score 3/5
The paper presents an interesting take on energy-aware network pruning. The authors consider the logical graph of a neural network as a physical graph instantiated on a computing substrate. They argue that communication energy between the nodes of this physical graph are a fundamental part of, and can come to dominate, the total energy consumption. They propose a regularizer that penalizes the network weights corresponding to long edges in the physical network graph, thereby making it more likely that these weights will be pruned by a magnitude-based pruner. They also introduce a novel pruning method that can prune based on weight magnitude as well as edge distance. They re-organize the physical network post-pruning to put connected nodes on consecutive layers closer together. They present results on LeNet-300-100 showing that their regularization, pruning, and network re-organization (using bi-partite matching) are better able to yield networks with low energy (total wire length) at a given accuracy level

To my knowledge, the work is novel and opens a new frontier when thinking about network pruning. My main criticism is that the logical network graph is not really physically embodied in today's hardware. You will typically have weights fetched from memory to carry out the network computation. Distance between nodes, etc. thus does not make sense in today's hardware. While we know the brain uses point-to-point axonal connections, current machine learning accelerators most often do not. Even compute-in-memory architecture based on novel devices such as memristors (where you have physical persistent connections) typically use crossbars and other rigid arrangements to connect neurons. These arrangements are dictated by physical considerations and will typically not conform to the arbitrary physical arrangement used in the paper. Maybe the authors can comment more as to how relevant their work is to current technology. 


L33: "lowest-weight nodes" : Should not this be lowest-weight edges. 

Table 1 : @98% accuracy. One would imagine that as p increases, the distance-based regularization would be stronger and thus the final energy should be lower (as long-distance connections are then more likely to be pruned). The trend observed does not show a monotonic reduction of energy as p increases, however. Can you please explain?

Confidence 2/3

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper27/Authors
comment: "Clarification
Step 2 of the algorithm is a little bit confusing. My understanding is as follows:
When d_s = 1, shouldn't it be weight-based pruning? Since it first selects d_s fraction of "lowest-weight" (select the node with lowest weight), and then prune a fix number among them. I just assume fixed number cannot prune more than d_s. In this sense, when d_s = the number of weights, it is distance-based pruning. If my understanding is correct, then the text could be revised since it is the other way around."

The reviewers are incorrect. Let's review what is said in the text: "To do this, first sort the edges by weight and select all $d_s$ fraction of lowest-weight nodes [comment: this should say edges. We will update this in the final document, and thank the other reviewer for pointing this out]. Then, sort these edges by distance, and prune a fixed number of the longest remaining wires." 

When $d_s$ is equal to the number of wires pruned, what happens is we sort the nodes by weight, select all $d_s$ lowest weight nodes, sort all of those nodes by distance, but then simply prune all of them. This is equivalent to merely pruning the fraction $d_s$ lowest weight wires (i.e., weight-based pruning). Now, suppose $d_s$ is equal to $1$. We select all the wires and of those sort all of them by distance and then prune the longest of those wires. If we sort all the nodes by weight, and then all of those by distance, that is exactly distance-based pruning (as the initial weight-based sort is irrelevant because we remove none of the higher weight wires from being candidates for pruning). As we state in the text: "The parameter $d_s$ is called distance sensitivity, when it is equal to the fraction of the wires that are pruned, this algorithm is equivalent to pure weight based pruning; when it is $1$ this corresponds to pure distance-based pruning."

To clarify the technique, we propose re-wording the sentence to: "To do this, first sort the edges by weight and select all $d_s$ fraction of lowest-weight nodes. Then, sort these lowest-weight edges by distance, and prune a fixed number of the longest remaining wires."  Note that we added the "these lowest-weight edges" to the sentence for clarity.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper27/Authors
comment: 
"Cons
- Limiting to specialized hardware."

As we state in the experiments section, "For MNIST, our best performing set of hyperparameters results in a compression ratio of 1.64 percent at $98\%$, comparable to state-of-the art results for this initial architecture and dataset." This indicates that our best performing algorithms, though designed for compressing neural networks for specialized hardware implementation, are also effective for obtaining small (indeed, state-of-the art) compression ratios. We thank the reviewers for pointing out this lack of clarity. Inspired by this, we propose placing in our conclusion: "...and show through ablation studies that all of these algorithms are effective, and can even reach state-of-the-art compression ratios. The results suggests that these techniques may be worth the computational effort if the neural network is to be widely deployed, if significantly lower energy is worth the slight decrease in accuracy, or if the application is to be deployed on either a specialized circuit \emph{or} general purpose processor."

"- No discussion about the pros and cons of such specialized hardware compared to neural network accelerators such as [19,20]. As a result, the discussed specialized hardware itself is not well-motivated. Concretely, if the motivation is stronger than "however, as transistor sizes are approaching fundamental limitations, it may be more sensible to construct specialized neural network circuits", it would be much better."

We thank the reviewer for this point. This work is inspired by the work of Grover (see citation 10). That wiring length dominates energy consumotion in specialized circuitry is also well established in the literature (consider as well the work of Thompson cited below). The basic motivating idea behind this paper is that: (1) modern neural networks are computations on graphs and (2) the fundamental energy costs of a computation on a graph flows from the properties of the graph, not merely the number of edges of the graph. This result is robust, is informed by experiment, and firmly justified by the laws of physics. As stated in chapter 8 of the Blake thesis that we cite, our model flows from what the author calls "the information friction hypothesis." The basic idea of this hypothesis is that communicating information in our universe is fundamentally proportional to distance. From the analysis both in the Grover paper and in the Blake paper, this model captures not only fundamental energy costs of existing digital circuits (as a result of resistive-capacitive effects of the transistors and wires) it also captures a fundamental energy cost of any conceivable practical technology.  In the works cited the authors consider a number of different conceivable communication paradigms, and demonstrates that all of them still have energy at least proportional to distance, or are otherwise utterly impractical.

We will place the suggested citations in the final version of the paper and we thank the reviewers for this suggestion. We note that for the technologies described in these paper, the fundamental information friction energy costs  will dominate in the limit. This is a consequence of the computational model used being informed by the fundamental laws of physics.

The reviewers state: "Concretely, if the motivation is stronger than "however, as transistor sizes are approaching fundamental limitations, it may be more sensible to construct specialized neural network circuits", it would be much better." However, as is stated in the same paragraph of this statement: "Moreover, wiring length is also optimized for in biological systems \cite{chklovskiiWiringOptimization, chklovskiiWiringOptimizationCortical} and is a fundamental engineering limitation for computation in the non-frictionless environments of the physical world \cite[Chapter 8]{blakePhDThesis}." These are two different motivations: one that flows from biological inspiration and another that flows from fundamental engineering limitations. 


C. D. Thompson, “A complexity theory for VLSI,” Ph.D. Thesis, Carnegie-Mellon, 

"- No connection to prior work in pruning. It is important to justify whether we need new algorithms or what we need is just adapting existing algorithms. For example, [21] might be useful in this application."

We thank the reviewer for this point, although we note that our ablation studies considers an adaptation of the widely cited Han et al. paper which focuses on weight-based pruning. Also consider citations [1-7] which is previous work on pruning to which we make a connection: specifically, we note that this prior work considered minimizing the number of non-zero weights, but not wiring length.

We thank the reviewer for illuminating reference 21 and we will incorporate a citation and discussion of this paper in our camera-ready verison.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper27/Authors
comment: We thank the reviewers for this feedback and address each comment in detail below and over the next few comments given character limits. 

"This work proposes algorithms to prune neural networks at the objective of optimizing the wiring length on specialized hardware. Concretely, the proposed algorithm follows three steps: (1) training with wiring length regularizer, (2) prune based on magnitude of weights as well as wiring length (first look at magnitude then look at distance), and (3) conduct layer-wise bipartite matching to optimize for wiring. This work is novel in the sense that it is targeting new objectives, however, the technical contribution is actually the wiring length regularization. The empirical results do not justify the need for (2) since the black concrete line is actually performing well. Moreover, (3) is an optimization that does not participate in the pruning process but rather a post-processing step. If the motivation for the specialized hardware is more quantitative and the connection to previous pruning methods are established, and hence the technical contributions of this paper can be better justified, this paper could be good for the workshop since the application of pruning in this context is novel and "might" entail new algorithms."

We thank the reviewer for this expeditious and clear review. The reviewers summarize the results of our study well; though we have a few points.  The reviewers state: " The empirical results do not justify the need for (2) since the black concrete line is actually performing well. " This statement is not correct. If we look at Figures 1 and 2, we see that nested rank pruning combined with distance regularization for some target accuracies outperforms the best performing distance-based regularization with weight pruning technique. This is true across both datasets. Of course, the differences between the performance of these two algorithms is very close. However, slight differences in performance may result in significant overall differences in energy consumptions when an applicaiton is widely deployed. As we say in our conclusion: "The results suggests that these techniques may be worth the computational effort if the neural network is to be widely deployed, or if significantly lower energy is worth the slight decrease in accuracy." Nonetheless, the reviewers make a good point: more or less distance based regularization with pure weight based pruning seems to perform very close to the best. We however consider it relevant to include the results for nested-rank pruning as a point of comparison to show the efficacy of the regularization technique, and to show that sometimes the combined techniques result in slightly better results. 

The reviewers state: " If the motivation for the specialized hardware is more quantitative and the connection to previous pruning methods are established, and hence the technical contributions of this paper can be better justified, this paper could be good for the workshop since the application of pruning in this context is novel and "might" entail new algorithms." However, as we detail in our responses below, our ablation studies already include a comparison to existing weight-based techniques. We also demonstrated on the standard LeNet-300-100 task that our algorithms (both the weight-based standard technique) and the distance-based techniques, reach state of the art compression ratios, indicating that we are comparing algorithms that are at the cutting edge. 

"Pros
- Novel application of pruning (specialized hardware with different objective)
- Comprehensive ablation study"
We thank the reviewers for these positive coments.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 3 / 5

Review:
This work proposes algorithms to prune neural networks at the objective of optimizing the wiring length on specialized hardware. Concretely, the proposed algorithm follows three steps: (1) training with wiring length regularizer, (2) prune based on magnitude of weights as well as wiring length (first look at magnitude then look at distance), and (3) conduct layer-wise bipartite matching to optimize for wiring. This work is novel in the sense that it is targeting new objectives, however, the technical contribution is actually the wiring length regularization. The empirical results do not justify the need for (2) since the black concrete line is actually performing well. Moreover, (3) is an optimization that does not participate in the pruning process but rather a post-processing step. If the motivation for the specialized hardware is more quantitative and the connection to previous pruning methods are established, and hence the technical contributions of this paper can be better justified, this paper could be good for the workshop since the application of pruning in this context is novel and "might" entail new algorithms.

Pros
- Novel application of pruning (specialized hardware with different objective)
- Comprehensive ablation study

Cons
- Limiting to specialized hardware.
- No discussion about the pros and cons of such specialized hardware compared to neural network accelerators such as [19,20]. As a result, the discussed specialized hardware itself is not well-motivated. Concretely, if the motivation is stronger than "however, as transistor sizes are approaching fundamental limitations, it may be more sensible to construct specialized neural network circuits", it would be much better.
- No connection to prior work in pruning. It is important to justify whether we need new algorithms or what we need is just adapting existing algorithms. For example, [21] might be useful in this application.

Clarification
Step 2 of the algorithm is a little bit confusing. My understanding is as follows:
When d_s = 1, shouldn't it be weight-based pruning? Since it first selects d_s fraction of "lowest-weight" (select the node with lowest weight), and then prune a fix number among them. I just assume fixed number cannot prune more than d_s. In this sense, when d_s = the number of weights, it is distance-based pruning. If my understanding is correct, then the text could be revised since it is the other way around.

Confidence: 2 / 3

[19] Chen, Yu-Hsin, et al. "Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks." IEEE Journal of Solid-State Circuits 52.1 (2017): 127-138.
[20] Jouppi, Norman P., et al. "In-datacenter performance analysis of a tensor processing unit." Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on. IEEE, 2017.
[21] Gordon, Ariel, et al. "Morphnet: Fast & simple resource-constrained structure learning of deep networks." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.