Original: On Learning Wire-Length Efﬁcient Neural Networks
Revised: On Learning Wire-Length Efﬁcient Neural Networks

Original: Anonymous Author(s)
Revised: Christopher G. Blake

Revised: Luyu Wang

Original: AfﬁliationAddress email
Revised: Giuseppe Castiglione Christopher Srinivasa Marcus A. Brubaker Borealis AI {chris.blake, luyu.wang, giuseppe.castiglione, christopher.srinivasa, marcus.brubaker}@borealisai.com

Original: Much recent work on computationally efﬁcient image recognition has focused on minimizing thenumber of non-zero weights in the neural network [1–7].
Revised: Much recent work on computationally efﬁcient image recognition has focused on minimizing the number of non-zero weights in the neural network [1–8].

Original: This is a good energy measure forcomputation on a general purpose processor, however, as transistor sizes are approaching fundamentallimitations, it may be more sensible to construct specialized neural network circuits [8].
Revised: This is a good energy measure for computation on a general purpose processor, however, as transistor sizes are approaching fundamental limitations, it may be more sensible to construct specialized neural network circuits [9, 10].

Original: In specializedcircuits, unlike in general purpose processors, energy consumption is often dominated by wiringlength, and not number of memory accesses [9–13].
Revised: In specialized circuits, unlike in general purpose processors, energy consumption is often dominated by wiring length, and not number of memory accesses [11–15].

Original: Moreover, wiring length is also optimized forin biological systems [14, 15] and is a fundamental engineering limitation for computation in thenon-frictionless environments of the physical world [16, Chapter 8].
Revised: Moreover, wiring length is also optimized for in biological systems [16, 17], it correlates in practice to actual specialized circuit energy consumption [18], and is a fundamental engineering limitation for computation in the nonfrictionless environments of the physical world [19, Chapter 8].

Original: Inspired by this, in this paperwe consider pruning algorithms to optimize for this wiring length measure using three techniques,each of which are used in different steps of the training-pruning process: weight-distance nested rankpruning, weight-distance based regularization, and layer-by-layer bipartite matching.
Revised: Inspired by this, in this paper we consider pruning algorithms to optimize for this wiring length measure using three techniques, each of which are used in different steps of the training-pruning process: weight-distance nested rank pruning, weight-distance based regularization, and layer-by-layer bipartite matching.

Original: First, consider the neural network as a graph in the natural way, in which each neuron is a node andedges connect neurons associated with non-zero weights.
Revised: First, consider the neural network as a graph in the natural way, in which each neuron is a node and edges connect neurons associated with non-zero weights.

Original: A placement of nodes is a set of positionsfor each node of the neural network such that the distance between each node is at least 1 unit distance.
Revised: A placement of nodes is a set of positions for each node of the neural network such that the distance between each node is at least 1 unit distance.

Original: The wiring length of an edge is the Manhattan distance between the nodes that it connects.
Revised: The wiring length of an edge is the Manhattan distance between the nodes that it connects.

Original: Theenergy of the neural network is the total wiring length of all the edges that are not pruned.
Revised: The energy of the neural network is the total wiring length of all the edges that are not pruned.

Original: The algorithm starts with a large neural network with a ﬁxed number of nodes and some labeledtraining data.
Revised: The algorithm starts with a large neural network with a ﬁxed number of nodes and some labeled training data.

Original: First, place the nodes in a stretched-square grid layout .
Revised: First, place the nodes in a stretched-square grid layout .

Original: In such a layout, we conceptuallyplace the nodes of the neural network, layer by layer, within a square prism, which we call the opticalchannel.
Revised: In such a layout, we conceptually place the nodes of the neural network, layer by layer, within a square prism, which we call the optical channel.

Original: We let the width of the prism be just big enough to hold the nodes of the biggest layer in asingle plane.
Revised: We let the width of the prism be just big enough to hold the nodes of the biggest layer in a single plane.

Original: Every other layer is placed consecutively on a square grid stretched out over the widthof the optical channel, in a plane at unit distance away from the nodes of the previous layer.
Revised: Every other layer is placed consecutively on a square grid stretched out over the width of the optical channel, in a plane at unit distance away from the nodes of the previous layer.

Original: Thisensures that all nodes are at least unit distance apart and that consecutive layers are adjacent.
Revised: This ensures that all nodes are at least unit distance apart and that consecutive layers are adjacent.

Original: We list the three steps of our algorithm below, and provide a rough outline of the technique.
Revised: We list the three steps of our algorithm below, and provide a rough outline of the technique.

Original: Train the neural network using stochastic gradient descent and weight-distance regulariza- tion .
Revised: Train the neural network using stochastic gradient descent and weight-distance regularization .

Original: This involves using a cross-entropy cost function and a term deﬁned as:
Revised: This involves using a cross-entropy cost function and a term deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: In this notation, d i denotes the length of the wire i and the w i denotes it associated weight.
Revised: In this notation, d i denotes the length of the wire i and the w i denotes it associated weight.

Original: Note that the d i s are ﬁxed during this procedure, but the w i s are variables that are to be trained.
Revised: Note that the d i s are ﬁxed during this procedure, but the w i s are variables that are to be trained.

Original: The variables p and α are hyperparameters to search over.
Revised: The variables p and α are hyperparameters to search over.

Original: When p is non-zero, the regularization term is meant to bias the circuit to make long wires have lower weight.
Revised: When p is non-zero, the regularization term is meant to bias the circuit to make long wires have lower weight.

Original: Prune the edges of the neural network using a pruning criteria called weight-distance nested rank pruning to prune the edges.
Revised: Prune the edges of the neural network using a pruning criteria called weight-distance nested rank pruning .

Original: To do this, ﬁrst sort the edges by weight and select all d s fraction of lowest-weight nodes.
Revised: To do this, ﬁrst sort the edges by weight and select all d s fraction of lowest-weight edges.

Original: Then, sort these  edges by distance, and prune a ﬁxednumber of the longest remaining wires.
Revised: Then, sort these lowest-weight edges by distance, and prune a ﬁxed number of the longest remaining wires.

Original: The parameter d s is called distance sensitivity, whenit is equal to the fraction of the wires that are pruned, this algorithm is equivalent to pure weight based pruning; when it is 1 this corresponds to pure distance-based pruning.
Revised: The parameter d s is called distance sensitivity, when it is equal to the fraction of the wires that are pruned, this algorithm is equivalent to pure weight based pruning; when it isthis corresponds to pure distance-based pruning.

Original: Whenan edge is pruned, its associated weight is ﬁxed to 0 , it’s distance is then considered zero incomputing wiring length, and it’s weight is no longer adjusted during training.

Original: After iterating between the previous two steps a sufﬁcient number of times, apply layer-bylayer bipartite matching to further optimize the energy of the layouts.
Revised: After iterating between the previous two steps a sufﬁcient number of times, apply layer-by-layer bipartite matching to further optimize the energy of the layouts.

Original: The algorithm usesthe realization that ﬁnding the optimal permutation of nodes in one layer that minimizes thewiring length to the nodes of other layers assuming their positions are ﬁxed is equivalent tothe bipartite matching problem, for which the Hungarian algorithm is polynomial-time andexact.
Revised: The algorithm uses the realization that ﬁnding the optimal permutation of nodes in one layer that minimizes the wiring length to the nodes of other layers assuming their positions are ﬁxed is equivalent to the weighted bipartite matching problem, for which the Hungarian algorithm is polynomial-time and exact [20].

Original: Apply this optimization algorithm layer by layer to the nodes of the pruned network.
Revised: Apply this optimization algorithm layer by layer to the nodes of the pruned network.

Original: Experiments
Revised: Experiments

Original: We run pruning experiments on a fully-connected neural network for MNIST, which contains twohidden layers of 300 and 100 units, respectively (this is the standard LeNet-300-100 architecture thathas been widely studied in the pruning literature).
Revised: We run pruning experiments on a fully-connected neural network for MNIST, which contains two hidden layers of 300 and 100 units, respectively (this is the standard LeNet-300-100 architecture that has been widely studied in the pruning literature).

Original: We also try pruning the fully connected layersa 10-layer convolutional network trained on the street-view house numbers dataset [17].
Revised: We also try pruning the fully connected layers of a 10-layer convolutional network trained on the street-view house numbers dataset [21].

Original: We showenergy-accuracy curves for one setting of hyperparameters for each of these datasets in Figure 1.
Revised: We show energy-accuracy curves for one setting of hyperparameters for each of these datasets in Figure 1.

Original: In Tables 1 and 2 we show a subset of the results of a hyperparameter grid search for these twodatasets.
Revised: In Tables 1 and 2 we show a subset of the results of a hyperparameter grid search for these two datasets.

Original: We deﬁne our task as pruning  and then for each set of hyperparameters choose the modelwith the lowest energy greater than some threshold accuracy.
Revised: We record the accuracy and energy after each pruning iteration, and then for each set of hyperparameters choose the model with the lowest energy greater than some threshold accuracy.

Original: For each target accuracy we show theweight-based result (which is comparable to the technique of [3] and forms a baseline) and the resultson the distance-based regularization technique.
Revised: For each target accuracy we show the weight-based result (which is comparable to the technique of [3] and forms a baseline) and the results on the distance-based regularization technique.

Original: We found that nested rank pruning can performbetter than pure weight based pruning, however distance-based regularization tends to outperformtechniques that use nested-rank pruning, although sometimes distance-based regularization withnested-rank pruning performs best in the lower accuracy, low energy regime as can be seen in the rightgraph of Figure 1.
Revised: We found that nested rank pruning can perform better than pure weight based pruning, however distance-based regularization tends to outperform techniques that use nested-rank pruning, although sometimes distance-based regularization with nested-rank pruning performs best in the lower accuracy, low energy regime as can be seen in the right graph of Figure 1.

Original: In these tables we obtain a wide range of values at the highest accuracy (which wesuspect is due to randomness in initial accuracy) but more consistency at the lower accuracies.
Revised: In these tables we obtain a wide range of values at the highest accuracy (which we suspect is due to randomness in initial accuracy) but more consistency at the lower accuracies.

Original: For MNIST, our best performing set of hyperparameters results in a compression ratio of 1.64 percent at98% , comparable to state-of-the art results for this initial architecture and dataset [18].
Revised: For MNIST, our best performing set of hyperparameters results in a compression ratio of 1.64 percent at 98% , comparable to state-of-the art results for this initial architecture and dataset [22].

Original: In Table 3 we apply the bipartite matching heuristic to the best performing network obtained usingweight-based regularization  the best performing network using weight-distance based regularizationthat meets the target accuracy.
Revised: In Table 3 we apply the bipartite matching heuristic to the best performing network obtained using weight-based regularization and the best performing network using weight-distance based regularization for each target accuracy.

Original: Across both datasets the distance-based regularization outperformsweight-based regularization.
Revised: Across both datasets the distance-based regularization outperforms weight-based regularization on average across four trials, in some cases by close to 70% .

Original: Conclusion
Revised: Conclusion

Original: In this paper we consider the novel problem of learning accurate neural networks that have low totalwiring length because this corresponds to energy consumption in the fundamental limit.
Revised: In this paper we consider the novel problem of learning accurate neural networks that have low total wiring length because this corresponds to energy consumption in the fundamental limit.

Original: We introduceweight-distance regularization, nested rank pruning, and layer-by-layer bipartite matching and showthrough ablation studies that all of these algorithms are effective.
Revised: We introduce weight-distance regularization, nested rank pruning, and layer-by-layer bipartite matching and show through ablation studies that all of these algorithms are effective, and can even reach state-of-the-art compression ratios.

Original: The results suggests that thesetechniques may be worth the computational effort if the neural network is to be widely deployed, orif signiﬁcantly lower energy is worth the slight decrease in accuracy.
Revised: The results suggests that these techniques may be worth the computational effort if the neural network is to be widely deployed, if signiﬁcantly lower energy is worth the slight decrease in accuracy, or if the application is to be deployed on either a specialized circuit or general purpose processor.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]