Paper:
SIGMA-DELTA QUANTIZED NETWORKS
Deep neural networks can be obscenely wasteful. When processing video, a convolutional network expends a fixed amount of computation for each frame with no regard to the similarity between neighbouring frames. As a result, it ends up repeatedly doing very similar computations. To put an end to such waste, we introduce SigmaDelta networks. With each new input, each layer in this network sends a discretized form of its change in activation to the next layer. Thus the amount of computation that the network does scales with the amount of change in the input and layer activations, rather than the size of the network. We introduce an optimization method for converting any pre-trained deep network into an optimally efficient Sigma-Delta network, and show that our algorithm, if run on the appropriate hardware, could cut at least an order of magnitude from the computational cost of processing video data.
Deep neural networks can be obscenely wasteful. When processing video, a convolutional network expends a fixed amount of computation for each frame with no regard to the similarity between neighbouring frames. As a result, it ends up repeatedly doing very similar computations. To put an end to such waste, we introduce SigmaDelta networks. With each new input, each layer in this network sends a discretized form of its change in activation to the next layer. Thus the amount of computation that the network does scales with the amount of change in the input and layer activations, rather than the size of the network. We introduce an optimization method for converting any pre-trained deep network into an optimally efficient Sigma-Delta network, and show that our algorithm, if run on the appropriate hardware, could cut at least an order of magnitude from the computational cost of processing video data.
1 INTRODUCTION
For most deep-learning architectures, the amount of computation required to process a sample of input data is independent of the contents of that data.
Natural data tends to contain a great deal of spatial and temporal redundancy. Researchers have taken advantage of such redundancy to design encoding schemes, like jpeg and mpeg, which introduce small compromises to image fidelity in exchange for substantial savings in the amount of memory required to store images and videos.
In neuroscience, it seems clear that that some kind of sparse spatio-temporal coding is going on. Koch et al. (2006) estimate that the human retina transmits 8.75Mbps, which is about the same as compressed 1080p video at 30FPS.
Thus it seems natural to think that perhaps we should be doing this in deep learning. In this paper, we propose a neural network where neurons only communicate discretized changes in their activations to one another. The computational cost of running such a network would be proportional to the amount of change in the input. Neurons send signals when the change in their input accumulates past some threshold, at which point they send a discrete “spike” notifying downstream neurons of the change. Such a system has at least two advantages over the conventional way of doing things.
1. When extracting features from temporally redundant data, it is much more efficient to communicate the changes in activation than it is to re-process each frame.
2. When receiving data asynchronously from different sources (e.g. sensors, or nodes in a distributed network) at different rates, it no longer makes sense to have a global network update. We could recompute the network with every new input, reusing the stale inputs from the other sources, but this requires doing a great deal of repeated computation for only small differences in input data. We could keep a history of all inputs and update the network periodically, but then we lose the ability to respond immediately to new inputs. Our approach gets around this ugly tradeoff by allowing for efficient approximate updates of the network given a partial update to the input data. The computational cost of the update is proportional to the effect that the new information has on the network’s state.
2 RELATED WORK
This work originated in the study of spiking neural networks, but treads into the territory of discretizing neural nets. The most closely related work is that of Zambrano and Bohte (2016). In this work, the authors describe an Adaptive Sigma-Delta modulation method, in which neurons communicate analog signals to one another by means of a “spike-encoding” mechanism, where a temporal signal is encoded into a sequence of weighted spikes and then approximately decoded as a sum of temporally-shifted exponential kernels. The authors create a scheme for being parsimonious with spikes by allowing adaptive scaling of thresholds, at the cost of sending spikes with real values attached to them, rather than the classic “all or nothing” spikes. Their work references a slightly earlier work by Yoon (2016) which reframes common neural models as forms of Asynchronous Sigma-Delta modulation. In a concurrent work, Lee et al. (2016) implement backpropagation in a similar system (but without adaptive threshold scaling), and demonstrate the best-yet performance on MNIST for networks trained with spiking models. This work postdates Diehl et al. (2015), which proposes a scheme for normalizing neuron activations so that a spiking neural network can be optimized for fast classification.
Our model contrasts with all of the above in that it is time-agnostic. Although we refer to sending “temporal differences” between neurons, our neurons have no concept of time - their is no “leak” in neuron potential, and our neurons’ behaviour only depends on the order of the inputs. Our work also separates the concepts of nonlinearity and discretization, uses units that communicate differences rather than absolute signal values, and explicitly minimizes an objective function corresponding to computational cost.
Coming from another corner, Courbariaux et al. describe a scheme for binarizing networks with the aim of achieving reductions in the amount of computation and memory required to run neural nets. They introduce a number of tricks for training binarized neural networks - a normally difficult task due to the lack of gradient information. Esser et al. (2016) use a similar binarization scheme to efficiently implement a spiking neural network on the IBM TrueNorth chip. Ardakani et al. (2015) take another approach - to approximate real-valued operations of a neural net with a sequence of stochastic integer operations, and show how these can lead to cheaper computation.
These discretization approaches differ from ours in that they do not aim to take advantage of temporal redundancy in data, but rather aim to find ways of saving computation by learning in a low-precision regime. Ideas from these works could be combined with the ideas presented in this paper.
The idea of sending quantized temporal differences has been applied to make event-based sensors, such as the Dynamic-Vision Sensor (Lichtsteiner et al., 2008), which quantize changes in analog pixelvoltages and send out pixel-change events asynchronously. The model we propose in this paper could be used to efficiently process the outputs of such sensors.
Finally, our previous work, (O’Connor and Welling, 2016) develops a method for doing backpropagation with the same type of time-agnostic spiking neurons we use here. In this paper, we do not aim to train the network from scratch, but instead focus on how we can compute efficiently by sending temporal differences between neurons.
3 THE SIGMA-DELTA NETWORK
In this Section, we describe how we start with a traditional deep neural network and apply two modifications - temporal-difference communication and rounding - to create the Sigma-Delta network. To explain the network, we follow the Figure 1 from top to bottom, starting with a standard deep network and progressing to our Sigma-Delta network. Here, we will think of the forward pass of a neural network as composition of subfunctions: f(x) = (fL ◦ ... ◦ f2 ◦ f1)(x).
3.1 TEMPORAL DIFFERENCE NETWORK
We now define “temporal difference” (∆T ) and “temporal integration” (ΣT ) modules as follows:
Algorithm 1 Temporal Difference (∆T ): 1: Internal: ~xlast ∈ Rd ← ~0 2: Input: ~x ∈ Rd 3: ~y ← ~x− ~xlast 4: ~xlast ← ~x 5: Return: ~y ∈ Rd
Algorithm 2 Temporal Integration (ΣT ): 1: Internal: ~y ∈ Rd ← ~0 2: Input: ~x ∈ Rd 3: ~y ← ~y + ~x 4: Return: ~y ∈ Rd
So that when presented with a sequence of inputs x1, ...xt, ∆T (xt) = xt − xt−1|x0=0, and ΣT (xt) =∑t τ=1 xτ . It should be noted that when we refer to “temporal differences”, we refer not to the change in the signal over time, but in the change between two inputs presented sequentially. The output of our network only depends on the value and order of inputs, not on the temporal spacing between them. This distinction only matters when dealing with asynchronous inputs such as the Dynamic Vision Sensor, (Lichtsteiner et al., 2008), which are not considered in this paper.
Since ΣT (∆T (xt)) = xt, we can insert ΣT ◦∆T pairs into the network without affecting the function. So we can re-express our network function as: f(x) = (fL◦ΣT ◦∆T ◦...◦f2◦ΣT ◦∆T ◦f1◦ΣT ◦∆T )(x). Now suppose our network consists of alternating linear functions w(x), and nonlinear functions h(x), so that f(x) = (hL ◦wL... ◦ h2 ◦w2 ◦ h1 ◦w1)(x). As before, we can harmlessly insert our ΣT ◦∆T pairs into the network. But this time, note that for a linear function w(x), the operations (ΣT , w,∆T ) all commute with one another. That is:
∆T (w(ΣT (x))) = w(∆T (ΣT (x))) = w(x) (1)
Therefore we can replace all instances of ∆T ◦ w ◦ ΣT with w, yielding f(x) = (hL ◦ ΣT ◦ wL ◦ ... ◦ ∆T ◦h2 ◦ΣT ◦w2 ◦∆T ◦h1 ◦ΣT ◦w1 ◦∆T )(x), which corresponds to the network shown in Figure 1 B. For now this is completely pointless, since we do not change the network function at all, but it will come in handy in the next section, where we discretize the output of the ∆T modules.
3.2 DISCRETIZATING THE DELTAS
When dealing with data that is naturally spatiotemporally redundant, like most video, we expect the output of the ∆T modules to be a vector with mostly low values, with some peaks corresponding to temporal transitions at certain input positions. We expect the data to have this property not only at the input layer, but even more so at higher layers, which encode higher level features (edges, object parts, class labels), which we would expect to vary more slowly over time than pixel values. If we discretize this “peaky” vector, we end up with a sparse vector of integers, which can then be used to cheaply communicate the approximate change in state of a layer to its downstream layer(s).
A sensible approach is to apply rounding before the temporal-difference operation - i.e. round the activation values and then send the temporal differences of these rounded values. It is then easy to show that the network’s function will remain identical to that of the rounding network:
ΣT (w(∆T (round(x)))) = w(ΣT (∆T (round(x)))) = w(round(x)) (2)
It’s worth noting that this is equivalent to doing discrete-time Sigma-Delta modulation to quantize the temporal differences - this connection is explained in Appendix A.
It follows from this result that our Sigma-Delta network depicted in Figure 1 D computes an identical function to that of the rounding network in Figure 1 C. In other words, the output yt of the Sigma-Delta network is solely dependent on the parameters of the network and the current input xt, and not on any of the previous inputs x1..xt−1. The amount of computation required for the update, however, depends on xt−1. Specifically, if xt is similar to xt−1, the Sigma-Delta network should require less computation to perform an update than the Rounding Network.
3.3 SPARSE DOT PRODUCT
Most of the computation in Deep Neural networks is consumed doing matrix multiplications and convolutions. The architecture we propose saves computation by translating the input to these operations into an integer array with a small L1 norm.
With sparse, low-magnitude integer input, we can compute the vector-matrix dot product efficiently by decomposing it into a sequence of vector additions. We can see this by decomposing the vector ~x ∈ Idin into a set of indices 〈(in, sn) : i ∈ [1..len(~x)], s ∈ ±1, n = [1..N ]〉, such that: ~x = ∑N n=1 sn~ein , where ein is a one-hot vector with element in hot, and N = |~x|L1 is the total L1 magnitude of the vector. We can then compute the dot-product as a series of additions, as shown in Equation 3.
u = ~x ·W : W ∈ Rdin×dout
= ( N∑ n=1 sn~ein ) ·W = N∑ n=1 ~snein ·W = N∑ n=1 sn ·Win,· (3)
Computing the dot product this way takes N · dout additions. A normal dense dot-product, by comparison, takes din · dout multiplications and (din − 1) · dout additions. This is where the energy savings come in. Horowitz (2014) estimates that on current 45nm silicon process, a 32-bit floating-point multiplication costs 3.7pJ, vs 0.9pJ for floating-point addition. With integer or fixed-point arithmetic, the difference is even more pronounced, with 3.1pJ for multiplication vs 0.1pJ for addition. This of course ignores the larger cost of processing instructions and moving memory, but gives us an idea of how these operations might compare on optimized hardware. So provided we can approximate the forward pass of a network to a satisfactory degree of precision without doing many more operations than the original network, we can potentially compute much more efficiently.
3.4 PUTTING IT ALL TOGETHER
Figure 1 visually summarizes the four types of network we have described. Inserting the temporal sum and difference modules discussed in Section 3.1 leads to the Temporal Difference Network, which
is functionally identical to the Original Network. Discretizing the output of the temporal difference modules, as discussed in Section 3.2, leads to the Sigma-Delta network. The Sigma-Delta Network is functionally equivalent to the Rounding network, except that it requires less computation per forward pass if it is fed with temporally redundant data.
4 OPTIMIZING AN EXISTING NETWORK
In this work, we do not aim to train a quantized networks from scratch, as we did in O’Connor and Welling (2016). Rather, we will take existing pretrained networks and optimize them as Sigma-Delta networks. In in our situation, we have two competing objectives: Error (with respect to a non-quantized forward pass), and Computation: the number of additions performed in a forward pass.
4.1 RESCALING OUR NEURONS
We can control the trade-off between these objectives by changing the scale of our discretization. We can thus extend our rounding function by adding a scale k ∈ R+:
round(~x, k) ≡ round(~x · k)/k (4)
This scale can either be layerwise or unitwise (in which case we have a vector of scales per layer). Higher k values will lead to higher precision, but also more computation, for the reason mentioned in Section 3.2. Note that the final division-by-k is equivalent to scaling the following weight matrix by 1 k ,. So in practice, our network functions become:
fround(x) = ( hL ◦
wL kL ◦ round ◦ ·kL ◦ ... ◦ h1 ◦ w1 k1 ◦ round ◦ ·k1
) (x) (5)
fΣ∆(x) = ( hL ◦ ΣT ◦
wL kL ◦ round ◦ ·kL ◦∆T ◦ ... ◦ h1 ◦ ΣT ◦ w1 k1 ◦ round ◦ ·k1 ◦∆T
) (x)
(6)
For the Rounding Network and the Sigma-Delta Network, respectively. By adjusting these scales kl, we can affect the tradeoff between computation and error. Note that if we use ReLU activation functions, parameters kl can simply be baked into the parameters of the network (see Appendix C.)
4.2 THE ART OF COMPROMISE
In this section, we aim to find the optimal trade-offs between Error and Computation for the Rounding Network (Network C in Figure 1). We define our loss as follows:
Lerror = D(fround(x), ftrue(x)) (7)
Lcomp = L−1∑ l=1 |sl|L1dl+1 (8)
Ltotal = Lerror + λLcomp (9)
Where D(a, b) is some scalar distance function (We use KL-divergence for softmax output layers and L2-norm otherwise), fround(x) is the output of the Rounding Network, ftrue(x) is the output of the Original Network. Lcomp is the computational loss, defined as the total number of additions required in a forward pass. Each layer performs |sl|L1dl+1 additions, where sl is the discrete output of the l’th layer, dl+1 is the dimensionality of the (l + 1)’th layer. Finally λ is the tradeoff parameter balancing the importance of the two losses.
We aim to use this loss function to optimize our layer-scales, kl to find an optimal tradeoff between accuracy and computation, given the tradeoff parameter λ.
4.3 DIFFERENTIATING THE UNDIFFERENTIABLE
We run into an obvious problem: y = round(k · x) is not differentiable with respect to our scale, k or our input, x. We get around this by using a similar method to Courbariaux et al., who in turn borrowed it from a lecture by Hinton (2012). That is, on the backward pass, when computing the gradient with respect to the error ∂Lerror∂kl , we simply pass the gradient through the rounding function in layers [l + 1, .., .L], i.e. we say ∂∂xround(x) ≈ 1.
When computing the gradient with respect to the computational cost, ∂Lcomp∂kl , we again just pass the gradient through all rounding operations in the backward pass for layers [l + 1, .., .L]. We found instabilities in training when using the computational loss of higher layers: Lcomp,l′ : l′ ∈ [l+1, ..., L], to update the scale of layer l. Since we don’t expect this term to have much effect anyway, we choose to only use the gradient of the computational cost in layer l when updating scale kl, i.e., we approximate: ∂Lcomp ∂kl ≈ ∂Lcomp,l∂kl .
Our scale parameters also must remain in the positive range, and stay well away from zero, where they can cause instability due to the division-by-k (see Equation 5). To handle this, we parametrize our scales in log-space, as κl = log(kl). Our scale-parameter update rule becomes:
∆κl = −η ( ∂Lerror ∂κl ∣∣∣∣ pass:[l+1..L] + λ ∂ ∂κl |~sl|L1dl+1 ∣∣∣∣ pass:l ) (10)
Where ~sl is the rounded signal from layer l, dl+1 is the “fan-out” (equivalent to the dimension of layer l + 1 in a fully-connected network), and pass : [l + 1..L] indicates that, on the backward pass, we simply pass the gradient through the rounding functions on layers [l + 1..L].
5 EXPERIMENTS
5.1 TOY PROBLEM: A RANDOM NETWORK
We start with a very simple toy problem to verify our method. We initialize a 2-layer (100-100-100) ReLU network with random weights using the initialization scheme proposed in Glorot and Bengio (2010), then scaled the weights by ( 1 2 , 8, 1 4 ) . The weight-rescaling does not affect the function of the network but makes it very ill-adapted for discretization (the first layer will be represented too coarsely, causing error; the second too finely, causing wasted computation). We create random input data, and use it to optimize the layer scales according to Equation 10. We verify, by comparing to a large collection of randomly drawn rescalings, that by tuning lambda we land on different places of the Pareto frontier balancing error and computation. Figure 2 shows that this is indeed the case. In this experiment, error and computation are evaluated just on the Rounding network - we test the Sigma-Delta network in the next experiment, which includes temporal data.
5.2 TEMPORAL-MNIST
In order to evaluate our network’s ability to save computation on temporal data, we create a dataset that we call “Temporal-MNIST”. This is just a reshuffling of the standard MNIST dataset so that similar frames tend to be nearby, giving the impression of a temporal sequence (see Appendix D for details). The columns of Figure 3 show eight snippets from the Temporal-MNIST dataset.
We started our experiment with a conventional ReLU network with layer sizes [784-200-200-10] pretrained on MNIST to a test-accuracy of 97.9%. We then apply the same scale-optimization procedure for the Rounding Network used in the previous experiment to find the optimal rescalings under a range of values for λ. This time, we test the learned scale parameters on both the Rounding Network and the Sigma-Delta network. We do not attempt to directly optimize the scales with respect to the amount of computation in the Sigma-Delta network - we assume that the result should be similar to that for the rounding network, but verifying this is the topic of future work.
The results of this experiment can be seen in Figure 4. We see that our discretized networks (Rounding and Sigma-Delta) converge to the error of the original network with fewer computations than are required for a forward pass of the original neural network. Note that the errors of the rounding and
Sigma-Delta networks are identical. This is a consequence of their equivalence, described in Section 3.2. Note also that the errors for all networks are identical between the MNIST and Temporal-MNIST datasets, since for all networks, the prediction function is independent of the order in which inputs are processed. We see that as expected, our Sigma-Delta network does fewer computations than the rounding network on the Temporal-MNIST dataset for the same error, because the update-mechanism of this network takes advantage of the temporal redundancy in the data.
5.3 A DEEP CONVOLUTIONAL NETWORK ON VIDEO
Our final experiment is a preliminary exploration into how Sigma Delta networks could perform on natural video data. We start with “VGG 19” - a 19 layer convolutional network, trained to recognise the 1000 ImageNet categories. The network was trained and made public by Simonyan and Zisserman (2014). We take selected videos from the ILSVRC 2015 dataset (Russakovsky et al., 2015), and apply the rescaling method from Section 4.1 to adjust the scales on a per-layer basis. We initially had some difficulty in optimizing the scale parameters of network to a stable point. The network would either fail to reduce computation when it could afford to, or reduce it to the point where the network’s function was so corrupted that error gradients would be meaningless, causing computation loss to win out and activations to drop to zero. A simple solution was to replace the rounding operation in training with addition of uniform random noise ∼ U(− 12 , 1 2 ). This seemed to prevent the network from pushing itself into a regime where all activations become zero. More work is need to understand why the addition of noise is necessary here. Figure 5 shows some preliminary results, which indicate that for video data we can get about 4-10x savings in the amount of computation required, in exchange for a modest loss in computational accuracy.
6 DISCUSSION
We have introduced Sigma-Delta Networks, which give us a new way compute the forward pass of a deep neural network. In Sigma-Delta Networks, neurons communicate not by telling other neurons about their current level of activation, but about their change in activation. By discretizing these changes, we end up with very sparse communication between layers. The more similar two consecutive inputs (xt, xt+1) are, the less computation is required to update the network. We show that, while the Sigma-Delta Network’s internal state at time-step t depends on past inputs x1..xt−1, the output yt only depends on the current input xt. We show that there is a tradeoff between the accuracy of this network (with respect to the function of a traditional deep net with the same parameters), and the amount of computation required. Finally, we propose a method to jointly optimize error and computation, given a tradeoff parameter λ that indicates how much accuracy we are willing to sacrifice in exchange for fewer computations. We demonstrate that this method substantially reduces the number of computations required to run a deep network on natural, temporally redundant data. However, we observe in our final experiment (Figure 5, bottom) that our assumption that higher-level features would be more temporally stable - and thus require less computation in our Sigma-Delta net - was not true. We suspect that if we were to train the network from scratch on temporal data, we may learn more temporally stable “slow” features, but this is a topic of future work.
A huge amount of data (eg. video, audio) comes in the form of temporal sequences, and there is an increasingly obvious need to be able to process this data efficiently. There is much to be gained by only doing processing when necessary, based on the contents of the data, and we provide one method for doing that. Further work is needed to determine whether this method would be of use on modern computing hardware, namely GPUs. The problem is that these devices are designed for large, fixed-size array operations, and tend not to be good at taking advantage of sparsity in the data, which requires many random memory accesses to parameters. Fortunately, other devices such as the the IBM TrueNorth (Cassidy et al., 2013) are being designed which keep memory close to processing, and such handle sparse data (and random memory access) much more efficiently.
This work opens up an interesting door. In asynchronous, distributed neural networks, a node may receive input from many different nodes asynchronously. Recomputing the function of the network every time a new input signals arrives may be prohibitively expensive. Our scheme deals with this by making the computational cost of an update proportional to the amount of change in the input. The next obvious step is to extend this approach to communicating changes in gradients, which may be helpful in setting up distributed, asynchronous schemes for training neural networks.
Code for our experiments can be found at: https://github.com/petered/sigma-delta/
ACKNOWLEDGMENTS
This work was supported by Qualcomm, who we’d also like to thank for discussing their past work in the field with us. We’d also like to thank fellow lab members, especially Changyong Oh and Matthias Reisser, for fruitful discussions contributing to this work.
A DELTA-HERDING PROOF
Algorithm 3 Herding
1: Internal: ~φ ∈ Rd ← ~0 2: Input: ~xt ∈ Rd 3: ~φ← ~φ+ ~xt 4: ~s← round(~φ) 5: ~φ← ~φ− ~s 6: Return: ~s ∈ Id
Algorithm 4 Delta-Herding 1: Internal: ~slast ∈ Id ← ~0 2: Input: ~xt ∈ Rd 3: ~s← round(xt) 4: ∆~s← ~s− ~slast 5: ~slast ← ~s 6: Return: ∆~s ∈ Id
In previous work (O’Connor and Welling, 2016), we used a quantization scheme which we refer to as herding for brevity and because of its relation to the deterministic sampling scheme in (Welling, 2009), but could otherwise be called Discrete-Time Bidirectional Sigma-Delta Modulation. The procedure is described in Algorithm 3. The input is summed into a potential φ over time until crossing a quantization threshold (in this case the ± 12 at which the round function changes value), and then resets. Here we prove that Algorithm 4 is equivalent to applying Algorithm 3 to the output of a temporal difference modules. i.e. herd(∆t(xt)) = ∆T (round(xt))∀t. First start by observing the following equivalence:
b = round(a)⇔ |a− b| < 1 2 : b ∈ I (11)
We can apply this to the update rule in Algorithm 3:
st = round(φt−1 + xt) ∈ I φt = (φt−1 + xt)− st
(12)
⇒ |φt| < 1
2 (13)
Now, if we unroll the two Equations 12 over time, with initial condition φ0 = 0, we see that.
φt = t∑ τ=1 xτ − t∑ τ=1 sτ : φt ∈ R, xτ ∈ R, sτ ∈ I (14)
Using Equations 13 and 11, respectively, we can say:
|φ| = ∣∣∣∣∣ t∑
τ=1
xt − t∑
τ=1
sτ ∣∣∣∣∣ < 12 (15) ⇒
t∑ τ=1 sτ = round
( t∑
τ=1
xτ
) (16)
Which can be rearranged to solve for st.
st = round
( t∑
τ=0
xτ
) − round ( t−1∑ τ=0 xτ ) (17)
Now if we receive inputs from a ∆T unit: xτ = uτ − uτ−1 with initial condition u0 = 0, then:
~st = round
( t∑
τ=0
(uτ − uτ−1) ) − round ( t−1∑ τ=0 (uτ − uτ−1) ) (18)
= round (ut)− round (ut−1) (19) = ∆T (round(ut)) (20)
Leaving us with the Delta-Herding algorithm (Algorithm 4).
Therefore, if we have a linear function w(x), and make use of Equation 1, then we can see that the following is true:
ΣT (w(herd(∆T (x)))) = ΣT (w(∆T (round(x)))) = w(ΣT (∆T (round(x)))) = w(round(x)) (21)
B CALCULATING FLOPS
When computing the number of operations required for a forward pass, we only account for the matrixproducts/convolutions (which form the bulk of computation in done by a neural network), and not hidden layer activations.
We compute the number of operations required for a forward pass of a fully connected network as follows:
For the non-discretized network, the number of flops for a single forward pass of a single data point through the network, the flop count is:
nF lopsdense = L−1∑ l=0 (dl · dl+1 + (dl − 1) · dl+1 + dl+1) = 2 L−1∑ l=0 dl · dl+1 (22)
Where dl is the dimensionality of layer l (with l = 0 indicating the input layer). The first term counts the number of multiplications, the second the number of additions for reducing the dot-product, and the third the addition of the bias.
It can be argued that this is an unfair way to count the number of computations done by the nondiscretized network because of the sparsity of the input layer (due to the zero-background of datasets like MNIST) and the hidden layers (due to ReLU units). Thus we also compute the number of operations for the non-discretized network when factoring in sparsity. The equation is:
nF lopssparse = L−1∑ l=0 ( Nl∑ i=0 ([al]i 6= 0) · dl+1 + ( Nl∑ i=0 ([al]i 6= 0)− 1 ) · dl+1 + dl+1 )
= 2 L−1∑ l=0 Nl∑ i=0 ([al]i 6= 0) · dl+1
(23)
Where al are the layer activations Nl is the number of units in layer l and ([al]i 6= 0) is 1 if unit i in layer l has nonzero activation and 0 otherwise.
For the rounding networks, we count the total absolute value of the discrete activations.
nF lopsRound = L−1∑ l=0 ( Nl∑ i=0 |[sl]i| · dl+1 + dl+1 ) (24)
Where sl is the discrete activations of layer l. This corresponds to the number of operations that would be required for doing a dot product with the “sequential addition” method described in Section 3.2.
Finally, the Sigma-Delta network required slightly fewer flops, because the bias only need to be added once (at the beginning), so its cost is amortized.
nF lopsΣ∆ = L−1∑ l=0 Nl∑ i=0 |[sl]i| · dl+1 (25)
C BAKING THE SCALES INTO THE PARAMETERS
In Section 4.1, we mention that we can “bake the scales into the parameters” for ReLU networks. Here we explain that statement.
Suppose you have a function
f(x) = k2 · h ( x · w
k1
)
If our nonlinearity h is homogeneous (i.e. k ·h(x) = h(k ·x)), as is the case for relu(x) = max(0, x), we can collapse the scales k into the parameters:
f(x) = k2 · relu(x · w/k1 + b) (26) = relu (x · w · k2/k1 + k2 · b) (27)
So that after training scales, for a given network, we can simply incorporate them into the parameters, as: w′ = w · k2/k1, and b′ = k2 · b.
D TEMPORAL MNIST
The Temporal MNIST dataset is a version of MNIST that is reshuffled so that similar frames end up being nearby. We generate this by iterating through the dataset, keeping a fixed-size buffer of candidates for the next frame. On every iteration, we compare all the candidates to the current frame, and select the closest one. The place that this winning candidate occupied in the buffer is then filled by a new sample from the dataset, and the winning candidate becomes the current frame. The process is repeated until we’ve sorted though all frames in the dataset. Code for generating the dataset can be found at: https://github.com/petered/sigma-delta/blob/master/sigma_ delta/temporal_mnist.py
E MNIST RESULTS TABLE
F HIGH-LEVEL FEATURE STABILITY
We had initially expected that, when a convolutional network is tasked with processing subsequent frames of video, high-level features would change much more slowly than the pixels and low-level features. This would give a computational advantage to our Sigma-Delta networks, whose computational cost scales with the amount of change in the feature representations. To our surprise, this appeared not to be the case. See the final plot of Figure 5. To verify that this was a property of the original convolutional network (and not somehow related our discretization scheme), we take the same snippet of video used for Figure 5 and measure the inter-frame differences. Figure 6 shows the results of this small experiment, and confirms that our initial belief - that inter-frame differences should become smaller and smaller at higher layers, was not quite correct.

Review:
This is an interesting paper about quantized networks that work on temporal difference inputs.  The basic idea is that when a network has only to process differences then this is computational much more efficient specifically with natural video data since large parts of an image would be fairly constant so that the network only has to process the informative sections of the image (video stream). This is of course how the human visual system works, and it is hence of interest even beyond the core machine learning community. 
As an aside, there is a strong community interested in event-based vision such as the group of Tobi Delbrück, and it might be interesting to connect to this community. This might even provide a reference for your comments on page 1.
I guess the biggest novel contribution is that a rounding network can be replaced by a sigma-delta network, but that the order of discretization and summation doe make some difference in the actual processing load. I think I followed the steps and 
Most of my questions have already been answers in the pre-review period. My only question remaining is on page 3, “It should be noted that when we refer to “temporal differences”, we refer not to the change in the signal over time, but in the change between two inputs presented sequentially. The output of our network only depends on the value and order of inputs, not on the temporal spacing between them.”
This does not make sense to me. As I understand you just take the difference between two frames regardless if you call this temporal or not it is a change in one frame. So this statement rather confuses me and maybe should be dropped unless I do miss something here, in which case some more explanation would be necessary.
Figure 1 should be made bigger.
An improvement of the paper that I could think about is a better discussion of the relevance of the findings. Yes, you do show that your sigma-delta network save some operation compared to threshold, but is this difference essential for a specific task, or does your solution has relevance for neuroscience?