T EMPORALLY
E FFICIENT
D EEP
L EARNING
WITH
S PIKES
Anonymous authors
Paper under double-blind review
A BSTRACT
The vast majority of natural sensory data is temporally redundant.
For instance, video frames or audio samples which are sampled at nearby points in time tend to have similar values.
Typically, deep learning algorithms take no advantage of this redundancy to reduce computations.
This can be an obscene waste of energy.
We present a variant on backpropagation for neural networks in which computation scales with the rate of change of the data - not the rate at which we process the data.
We do this by implementing a form of Predictive Coding wherein neurons communicate a combination of their state, and their temporal change in state, and quantize this signal using Sigma-Delta modulation.
Intriguingly, this simple communication rule give rise to units that resemble biologically-inspired leaky integrate-and-ﬁre neurons, and to a spike-timing-dependent weight-update similar to Spike-Timing Dependent Plasticity (STDP), a synaptic learning rule observed in the brain.
We demonstrate that on MNIST, on a temporal variant of MNIST, and on Youtube-BB, a dataset with videos in the wild, our algorithm performs about as well as a standard deep network trained with backpropagation, despite only communicating discrete values between layers.
I NTRODUCTION
Currently, most algorithms used in Machine Learning work under the assumption that data points are independent and identically distributed, as this assumption provides good statistical guarantees for convergence.
This is very different from the way data enters our brains.
Our eyes receive a single, never-ending stream of temporally correlated data.
We get to use this data once, and then it’s gone.
Moreover, most sensors produce sequential, highly overlapping, and redundant streams of data.
This can be both a blessing and a curse.
From a statistical learning point of view this redundancy may lead to biased estimators and eventually unnecessarily bad predictions.
However, this temporal redundancy also implies that intuitively not all computations are necessary.
Online Learning is the study of how to learn in this domain - where data becomes available in sequential order and is given to the model only once.
Given the enormous amount of sequential data, mainly videos, that are being produced nowadays, it seems desirable to develop learning systems that simply consume data on-the-ﬂy as it is being generated, rather than collect it into datasets for ofﬂine-training.
There is, however a problem of efﬁciency, which we hope to illustrate with two examples:
CCTV feeds.
CCTV Cameras collect an enormous amount of data from mostly-static scenes.
The amount of new information in a frame, given the previous frame, tends to be low.
That is to say, data tends to be temporally redundant.
If we want to train a model from of this data (for example a pedestrian detector), we need process a large amount of mostly-static frames.
If the frame rate doubles, so does the amount of computation.
Intuitively, it feels that this should not be necessary.
It would be nice to still be able to use all this data, but have the amount of computation scale with the amount of new information in each frame, not just the number of frames and dimensions of the data.
Robot perception .
Robots have no choice but to learn online - their future input data ( e g. camera frames) are dependent on their previous predictions ( i.e. motor actions).
Not only does their data come in nonstationary temporal streams, but it typically comes from several sensor running at different rates.
The camera may produce 1MB images atframes/s, while the gyro might produce 1-byte readings 1000 frames/s.
It’s not obvious, using current methods in deep learning, how we can integrate asynchronous sensory signals into a uniﬁed, trainable, latent representation, without undergoing the horribly inefﬁcient process of recomputing the function of the network every time a new signal arrives.
These examples point to the need for a training method where the amount of computation required to update the model scales with the amount of new information in the data, and not just the dimensionality of the data.
There has been a lot of work on increasing the computational efﬁciency of neural networks by quantizing neural weights or activations (see Section 4), but comparatively little work on exploiting redundancies in the data to reduce the amount of computation.
O’Connor and Welling (2016b), set out to exploit the temporal redundancy in video, by having neurons only send their quantized changes in activation to downstream neurons, and having the downstream neurons integrate these changes over time.
This approach works for efﬁciently approximating the function of the network, but fails for training, because when the weights are changing with time, this approach (take the temporal difference, multiply by weights, temporally integrate) fails to reconstruct the correct activation for the next layer.
In other words, (cid:80) [-tτ =
-]t τ =0 ( x τ − x τ − 1 ) · w τ (cid:54) = x t · w t unless w t = w 0 ∀ t .
Figure 2 describes the problem visually.
In this paper, we correct for this problem by encoding a mixture of two components of the layers activation x t : the proportional component k p x t , and the derivative component k d ( x t − x t − 1 ) .
When we invert this encoding scheme, we get a decoding scheme which corresponds to taking an exponentially decaying temporal average of past inputs.
Interestingly, the resulting neurons begin to resemble models of biological spiking neurons, whose membrane potentials can approximately be modeled as an exponentially decaying temporal average of past inputs.
In this work, we present a scheme wherein the temporal redundancy of input data is used to reduce the computation required to train a neural network.
We demonstrate this on the MNIST and Youtube-BB datasets.
To our knowledge we are the ﬁrst to create a neural network training algorithm which uses less computation as data becomes more temporally redundant.
M ETHODS
We propose a coding scheme where neurons can represent their activations as a temporally sparse series of impulses.
The impulses from a given neuron encode a combination of the value and the rate of change of the neuron’s activation.
While our algorithm is designed to work efﬁciently with temporal data , we do not aim to learn temporal sequences in this work.
We aim to efﬁciently approximate a function y t = f ( x t ) , where the current target y t is solely a function of the current input x t , and not previous inputs x 0 ...x
t − 1 .
The temporal redundancy between neighbouring inputs x t − 1 , x t will however be used to make our approximate computation of this function more efﬁcient.
2. P RELIMINARY
Throughout this paper we will use the notation ( f 3 ◦ f 2 ◦ f 1 )( x ) = f 3 ( f 2 ( f 1 ( x ))) to denote function composition.
We slightly abuse the notion of functions by allowing them to have an internal state which persists between calls.
For example, we deﬁne the ∆ function in Equation 1 as being the difference between the inputs in two consecutive calls (where persistent variable x last is initialized to 0).
The Σ function, deﬁned in Equation 2, returns a running sum of the inputs over calls.
So we can write, for example, that when our composition of functions (Σ ◦ ∆) is called with a sequence of input variables x τ : τ = [1 ..t ] , then (Σ ◦ ∆)( x t )
= x t , because y 0 + ( x 1 − x 0 ) + ( x 2 − x 1 ) + ... + ( x t − x t − 1 ) | x 0 =0 ,y
=
= x t .
In general, when we write y t = f ( x t ) , where f is a function with persistent state, it will be implied that we have previously called f ( x τ ) for τ ∈ [1 , .., t − 1] in sequence. Variable deﬁnitions that are used later will be highlighted in blue.
While all terms are deﬁned in the paper, you can always look up the meaning of any symbol used in this paper in Appendix A.
2. PD E NCODING
∆ : x (cid:55)→ y ; Persistent: x last ←y ← x − x last x last ← x (1) Σ : x (cid:55)→ y ;
Persistent: y ←y ← y + x (2)
Q : x (cid:55)→ y ; Persistent: φ ←φ (cid:48) ← φ + x y ← round ( φ (cid:48) )
φ ← φ (cid:48) − y (3) enc : x (cid:55)→ y ;
Persistent: x last ←y ← k p x + k d ( x − x last ) x last ← x (4) dec : x (cid:55)→ y ; Persistent: y ←y ← x + k d y k p + k d (5)
R : x (cid:55)→ round ( x ) (6)
Suppose a neuron has time-varying activation x τ : τ ∈ [1 ..t ] .
Taking inspiration from Proportional-Integral-Derivative (PID) controllers, we can “encode” this activation at each time step as a combination of its current activation and change in activation as a t (cid:44) enc ( x t ) =
k p x t + k d
x t − x t − 1 ) , (see
Equation 4).
The parameters k p and k d determine what portion of our encoding represents the value of the activation and the rate of change of that value, respectively.
In Section 4, we discuss how this is a form of Predictive Coding and in Appendix F, we discuss the effect our choices for these parameters have on the network.
To get our decoding formula, we can simply solve for x t as x t = a t + k d x t − 1 k p + k d (Equation 5), such that ( dec ◦ enc )( x t ) = x t .
Notice that
Equation 5 corresponds to decaying the previous decoder state by some constant k d / ( k p + k d ) and then adding the input a t / ( k p + k d ) .
We can expand this recursively to see that this corresponds to a temporal convolution a ∗ κ where κ is a causal exponential kernel κ τ = (cid:110)
1 k p + k d (cid:16) k d k d + k p (cid:17) τ if τ ≥ 0 otherwise 0 (cid:111) .
2. Q UANTIZATION
Our motivation for the aforementioned encoding scheme is that we want a sparse signal which can be quantized into a low-bitrate discrete signal.
This will later be used to reduce computation.
We can quantize our signal a t into a sparse, integer signal s t (cid:44) Q ( a t ) , where the quantizer Q is deﬁned in Equation 3.
Equation 3 implements a form of Sigma-Delta modulation, a method widely used in signal processing to approximately communicate signals at low bit-rates (Candy and Temes, 1962).
We can show that that Q ( x t ) =
(∆ ◦ R ◦ Σ)( x t ) (See Supplementary Material Section C), where ∆ ◦ R ◦ Σ indicates applying a temporal summation, a rounding, and a temporal difference, in series.
If x t is temporally redundant, then | a t | (cid:28) 1 ∀ t , and we can expect s t to consist of mostly zeros with a few 1’s and -1’s.
We can now approximately reconstruct our original signal x t as ˆ x t (cid:44) dec ( s t ) by applying our decoder, as deﬁned in Equation 5.
As our coefﬁcients k p , k
d become larger, our reconstructed signal ˆ x t should become closer to the original signal x t .
We illustrate examples of encoded signals and their reconstructions for different k p , k d in Figure 1.
2.3. S PECIAL CASES
We can compactly write the entire reconstruction function as ˆ x = ( dec ◦ ∆ ◦ R ◦ Σ ◦ enc )( x t ) .
k p = 0 :
When k p = 0 , we get dec ( x t ) =
( k − 1 d ◦ Σ)( x t ) and enc ( x t ) =
( k d ◦ ∆)( x t ) , so our reconstruction reduces to ˆ x =
( k − 1 d ◦ Σ ◦ ∆ ◦ R ◦ Σ ◦ k d ◦ ∆)( x t ) .
Because Σ ◦ k d ◦ ∆ all commute with one another, we can simplify this to ˆ x t =
( k − 1 d ◦ R ◦ k d )( x t ) .
so our decoded signal is ˆ x t = round ( x t · k d ) /k
d , with no dependence on x t − 1 .
This is visible in the bottom row of Figure 1.
This was the encoding scheme used in O’Connor and Welling (2016b). [Figure] [Figure]
k d = 0 :
In this case, dec ( x t ) =
k − 1 p x t and enc ( x t ) =
k p x t so our encoding-decoding process becomes ˆ x = ( k − 1 p ◦ ∆ ◦ R ◦ Σ ◦ k p )( x t ) .
In this case neither our encoder nor our decoder have any memory, and we take no advantage of temporal redundancy.
2. S PARSE C OMMUNICATION B ETWEEN L AYERS
The purpose of our encoding scheme is to reduce computation by sparsifying communication between layers of a neural network.
Suppose we are trying to compute the pre-nonlinearity activation of the ﬁrst hidden layer, z t ∈ R d out , given the input activation, x t ∈ R d in .
We approximate z t as: z t (cid:44) x t · w t
≈ ˆ x t · w t (cid:44) dec ( Q ( enc ( x t ))) · w t (cid:44) dec ( s t ) · w t ≈ dec ( s t · w t ) (cid:44) ˆ z t where: x t , ˆ x t ∈ R d in ; s t ∈ Z d in ; w ∈ R d in ×
d out ; z t , ˆ z t ∈ R d out (7)
The ﬁrst approximation comes from the quantization (Q) of the encoded signal, and the second from the fact that the weights change over time, as explained in Figure 2.
The effects of these approximations are further explored in Appendix D.
Computing z t takes d in · d out multiplications and ( d in − 1) · d out additions.
The cost of computing ˆ z t , on the other hand, depends on the contents of s t .
If the data is temporally redundant, s t ∈ Z d in should be sparse, with total magnitude N (cid:44) (cid:80) i | s t,i | .
s t can be decomposed into a sum of one-hot vectors s t =
(cid:80) Nn =1 sign ( s t,i n ) γ i n : i n ∈ [1 .. d i n ] where γ i n is a length- d in one-hot vector with element ( γ i n ) i n = 1 .
The matrix product s t · w can then be decomposed into a series of row additions:
[Equation]
If we include the encoding, quantization, and decoding operations, our matrix product takes a total of 2 d in + 2 d out multiplications, and (cid:80) n | s t,n | · d out + 3 d in + d out additions.
Assuming the (cid:80) n | s t,n | · d out term dominates, we can say that the relative cost of computing ˆ z t vs z t is:
[Equation]
2. A N EURAL N ETWORK
We can implement this encoding scheme on every layer of a neural network.
Given a standard neural net f nn consisting of alternating linear ( · w l ) and nonlinear ( h l ) operations, our network function f pdnn can then be written as:
[Equation]
We can use the same approach to approximately calculate our gradients to use in training.
If we deﬁne our layer activations as ˆ z l (cid:44) ( dec ◦ · w l ◦ Q ◦ enc )( x ) if l = 1 otherwise ( dec ◦ · w l ◦ Q ◦ enc )(ˆ z l − 1 ) , and L (cid:44) (cid:96) ( f pdnn ( x ) , y ) , where (cid:96) is some loss function and y is a target, we can backpropagate the approximate gradients as:
[Equation]
This can be implemented by either executing a (sparse) forward and backward pass at each time-step, or in an “event-based manner”, where neurons ﬁre off “events” whenever incoming events push their activations past a threshold, and these events are in turn sent to downstream neurons. For ease of implementation, we opt for the former in our code. Note that unlike in regular backprop computing these forward and backward passes results in changes to the internal state of the enc , dec , and Q components.
2. P ARAMETER U PDATES
There’s no use having an efﬁcient backward pass if the parameter updates aren’t also efﬁcient.
In a normal neural network trained with backpropagation and simple stochastic gradient descent, the parameter update for weight matrix w has the form w ← w − η ∂
L ∂w where η is the learning rate.
If w connects layer l − 1 to layer l , we can write ∂ L ∂w = x t ⊗ e t where x t (cid:44) h l − 1 ( z l − 1 ,t ) ∈
R d in is the presynaptic (layer l − 1 ) activation, e t (cid:44) (cid:100) ∂
L ∂ ˆ z l,t ∈ R d out is the postsynaptic (layer l ) backpropagating gradient and ⊗ is the outer product.
So we pay d in · d out multiplications to update the parameters for each sample.
x x x spike e e e t wt = x t e t wt = x t e t wt , past wt , future
We want a more efﬁcient way to compute this product, which takes advantage of the sparsity of our encoded signals to reduce computation.
We can start by applying our encoding-quantizing-decoding scheme to our input and error signals as ¯ x t (cid:44) ( Q ◦ enc )( x t )
∈ Z d in and ¯ e t (cid:44) ( Q ◦ enc )( e t ) ∈
Z d out , and approximate our true update as (cid:99) ∂
L ∂wrecon,t (cid:44) ˆ x t ⊗ ˆ e t where ˆ x t (cid:44) dec (¯ x t ) and ˆ e t (cid:44) dec (¯ e t ) .
This doesn’t do any good by itself, because the update rule still is not sparse.
But, we can exactly compute the sum of this value over time using one of two sparse update schemes - past updates and future updates - which are depicted in Figure 3. We give the formula for the Past and Future update rules in
Appendix E, but summarize them here: Past Updates : For a given synapse w i,j , if either the presynaptic neuron spikes ( ¯ x t i (cid:54) = 0 ) or the postsynaptic neuron spikes ( ¯ e t i (cid:54) = 0 ), we increment the w i,j by the total area under ˆ x τ,i ˆ e τ,j since the last spike.
We can do this efﬁciently because between the current time and the time of the previous spike, ˆ x τ,i ˆ e τ,j is a geometric sequence.
Given a known initial value u , ﬁnal value v , and decay rate r , a geometric sequence sums to u − v 1 − r .
The area calculated is shown in pink on the bottom row of
Figure 3, and one algorithm to calculate it is in Equation 19 in Appendix E.
Future Updates : Another approach is to calculate the Present Value of the future area under the integral from the current spike.
This is depicted in the blue-gray area in Figure 3, and the formula is in Equation 20 in Appendix E.
Finally, because the magnitude of our gradient varies greatly over training, we create a scheme for adaptively tuning our k p , k d parameters to match the running average of the magnitude of the data.
This is described in detail in Appendix F.
2. R ELATION TO STDP
An extremely attentive reader might have noted that Equation 20 has the form of an online implementation of Spike-Timing Dependent Plasticity (STDP).
STDP (Markram et al., 2012) emerged from neuroscience, where it was observed that synaptic weight changes appeared to be functions of the relative timing of pre- and post-synaptic spikes.
The empirically observed function usually has the double-exponential form seen on the rightmost plot of Figure 4.
Using the quantized input signal ¯ x and error signal ¯ e , and their reconstructions ˆ x t and ˆ e t as deﬁned in the last section, we deﬁne a causal convolutional kernel κ t = (cid:110)
k β ( k α ) t
if t ≥ 0 otherwise 0 (cid:111) , where k α (cid:44) = k We can then deﬁne a “cross-correlation kernel” g t = { κ t if t ≥ 0 otherwise κ − t } = k β ( k α ) | t | : t ∈ Z which deﬁnes the magnitude of a parameter t post t pre w sign ( x t )= sign ( e t ) t post t pre sign ( x t ) sign ( e t ) t post t pre
Classic STDP Rule update as a function of the difference in timing between pre-synaptic spikes from the forward pass and post-synaptic spikes from the backward pass.
The middle plot of Figure 4 is a plot of g .
We deﬁne our STDP update rule as:
[Equation]
We note that while our version of STDP has the same double-exponential form as the classic STDP rule observed in neuroscience (Markram et al., 2012), our “presynaptic” spikes come from the forward pass while our “postsynaptic” spikes come from the backwards pass. STDP is not normally used to as a learning rule networks trained by backpropagation, so the notion of forward and backward pass with a spike-timing-based learning rule are new.
Moreover, unlike in classic STDP, we do not have the property that sign of the weight change depends on whether the presynaptic spike preceded the postsynaptic spike.
In Section E in the supplementary material we show experimentally that while Equations (cid:99) ∂
L ∂wrecon , (cid:99) ∂
L ∂wpast , (cid:99) ∂
L ∂wfuture , (cid:99) ∂
L ∂wstdp may all result in different updates at different times, the rules are equivalent in that for a given set of pre/post-synaptic spikes ¯ x, ¯ e , the cumulative sum of their updates over time converges exactly.
E XPERIMENTS 3. T EMPORAL MNIST
To evaluate our network’s ability to learn, we run it on the standard MNIST dataset, as well as a variant we created called “Temporal MNIST”.
Temporal MNIST is simply a reshufﬂing of the
MNIST dataset so that so that similar inputs (in terms of L2-pixel distance), are put together.
Figure 6 shows several snippets of consecutive frames in the temporal MNIST dataset.
We compare our
Proportional-Derivative Net against a conventional Multi-Layer Perceptron with the same architecture (one hidden layer of 200 ReLU hidden units and a softmax output).
The results are shown in Figure 5.
Somewhat surprisingly, our predictor slightly outperformed the MLP, getting 98.36% on the test set vs 98.25% for the MLP.
We assume this improvement is due to the regularizing effect of the quantization.
On Temporal MNIST, our network was able to converge with less computation than it required for MNIST (it used 32 · 10 12 operations for MNIST vs 15 · 10 12 for Temporal MNIST), but ended up with a slightly worse test score when compared with the MLP (the PDNN got 97.99% vs 98.28% for the MLP). Appendix G contains a table of results with varying hyperparameters.
3. Y OU T UBE V IDEO D ATASET
Next, we want to simulate the setting of CCTV cameras, discussed in Section 1, where we have a lot of data with only a small amount of new information per frame.
In the absence of large enough public CCTV video datasets, we investigate the surrogate task of frame-based object classiﬁcation on [Figure] wild YouTube videos from the large, recently released Youtube-BB dataset Real et al.
Our subset consists of 358 Training Videos and 89 Test videos with 758,033 frames in total.
Each video is labeled with an object in one of 24 categories.
We start from a VGG19 network (Simonyan and Zisserman, 2014): a 19-layer convolutional network pre-trained on imagenet.
We replace the top three layer with three of our own randomly initialized layers, and train the network both as a spiking network, and as a regular network with backpropagation.
While training the entire spiking network end-to-end works, we choose to only train the top layers, in order to speed up our training time.
We compare our training scores and computation between a spiking and non-spiking implementation.
Figure 7 shows that our spiking network performs comparably to a non-spiking network, and Figure 8 shows how the computation per frame of our spiking network decreases as we increase the frame rate (i.e. as the input data becomes more temporally redundant).
Because our spiking network uses only additions, while a regular deep network does multiply-adds, we use the estimated energy-costs per op of Horowitz (2014) to compare computations to a single scale, which estimates the amount of energy required to do multiplies and adds in ﬁxed-point arithmetic.
R ELATED W ORK
Noise-Shaping is a quantization technique that aims to increase the ﬁdelity of signal reconstructions, per unit of bandwidth of the encoded signal, by quantizing the signal in such a way that the quantization noise is pushed into a higher frequency band which is later ﬁltered out upon decoding.
Sigma-Delta (also known as Delta-Sigma) quantization is a form of noise-shaping.
Shin (2001) ﬁrst suggested that biological neurons may be performing a form of noise shaping, and Yoon (2017) found standard spiking neuron models actually implement a form of Sigma-Delta modulation.
The encoding/decoding scheme we use in this paper can be seen as a form of Predictive Coding.
Predictive coding is a lossless compression technique wherein the predictable parts of a signal are
[Figure] [Figure] subtracted away so that just the unpredictable parts are transmitted. The idea that biological neurons may be doing some form of predictive coding was ﬁrst proposed by Srinivasan et al.
In a predictive-coding neuron (unlike neurons commonly used in Deep Learning) there is a distinction between the signal that a neuron represents and the signal that it transmits .
The neurons we use in this paper can be seen as implementing a simple form of predictive coding where the “prediction” is that the neuron maintains a decayed form of its previous signal - i e. that pred ( x t ) (cid:44)
[Figure]
[Equation]
(See
Appendix B for detail).
Chklovskii and Soudry (2012) suggest that the biological spiking mechanism may be though of as consisting of a sigma-delta modulator within a predictive-coding circuit.
To our knowledge, none of the aforementioned work has yet been used in the context of deep learning.
There has been sparse but interesting work on merging the notions of spiking neural networks and deep learning.
Diehl et al.
(2015) found a way to efﬁciently map a trained neural network onto a spiking network.
Lee et al.
(2016) devised a method for training spiking of integrate-and-ﬁre spiking neurons with backpropagation - though their neurons did not send a temporal difference of their activations.
O’Connor and Welling (2016a) created a method for training event-based neural networks - but their method took no advantage of temporal redundancy in the data.
Binas et al.
(2016) and (O’Connor and Welling, 2016b) both took the approach of sending quantized temporal changes reduce computation on temporally redundant data, but their schemes could not be used to train a neural network.
Bohte et al.
(2000) showed how one could apply backpropagation for training spiking neural networks, but it was not obvious how to apply the method to non-spiking data.
Zambrano and Bohte (2016) developed a spiking network with an adaptive scale of quantization (which bears some resemblance to our tuning scheme described in Appendix F), and show that the spiking mechanism is a form of Sigma-Delta modulation, which we also use here.
Courbariaux et al.
(2015) showed that neural networks could be trained with binary weights and activations (we just quantize activations).
Bengio et al.
(2015) found a connection between the classic STDP rule (Figure 4, right) and optimizing a dynamical neural network, although the way they arrived at an STDP-like rule was quite different from ours.
D ISCUSSION
We set out with the objective of reducing the computation in deep networks by taking advantage of temporal redundancy in data.
We described a simple rule (Equation 4) for sparsifying the communication between layers of a neural network by having our neurons communicate a combination of their temporal change in activation, and the current value of their activation.
We show that it follows from this scheme that neurons should behave as leaky integrators (Equation 5).
When we quantize our neural activations with Sigma-Delta modulation, a common quantization scheme in signal processing, we get something resembling a leaky integrate-and-ﬁre neuron.
We derive efﬁcient update rules for the weights of our network, and show these to be related to STDP - a learning rule ﬁrst observed in neuroscience.
Finally, we train our network, verify that it does indeed compute more efﬁciently on temporal data, and show that it performs about as well as a traditional deep network of the same architecture, but with signiﬁcantly reduced computation.
Finally, we show that our network can train on real video data.
The efﬁciency of our approach hinges on the temporal redundancy of our input data and neural activations.
There is an interesting synergy here with the concept of slow-features (Wiskott, 1999).
Slow-Feature learning aims to discover latent objects that persist over time.
If the hidden units were to speciﬁcally learn to respond to slowly-varying features of the input, the layers in a spiking implementation of such a network would have to communicate less often.
In such a network, the tasks of feature-learning and reducing inter-layer communication may be one and the same. A N OTATION
Here we have a legend of notation used throughout this paper.
While the paper is intended to be self-contained, the reader may want to consult this list if ever there is any doubt about the meaning of a variable used in the paper.
Here we indicate the section in which each symbol is ﬁrst used.
Section 2.
∆ : A “temporal difference” operator. See EquationΣ : A “temporal integration” operator. See EquationQ : Sigma-Delta quantization.
See Equationφ :
The internal state variable of the quantizer Q .
enc : An “encoding” operation, which takes a signal and encodes it into a combination of the signal’s current value and its change in value since the last time step.
See Equation 4.
dec : An “decoding” operation, which takes an encoding signal and attempts to reconstruct the original signal that it was encoded from.
If there was no quantization done on the encoded signal, the reconstruction will be exact, otherwise it will be an approximation.
See Equation 5.
R :
The “rounding” operation, which simply rounds an input to the nearest integer value.
x t : Used throughout the paper to represent the value of a generic analog input signal at time t .
In Sections 2.1, 2.2, and 2.3 it represents a scalar, and thereafter it represents a vector of inputs.
Section 2.
k
p ,
k d
∈
R + : Positive scalar coefﬁcients used in the encoder and decoder, controlling how the extent to which the encoding is proportional to the input ( k p ) vs proportional to the temporal difference of the input ( k d ).
a t (cid:44) enc ( x t ) : Used to represent the encoded version of x t .
Section 2.
s t (cid:44) Q ( a t ) :
Used to represent the quantized, encoded version of x t .
ˆ x t (cid:44) dec ( s t ) : Used to represent the reconstruction of input x t „ obtained by encoding, quantizing, and decoding x t .
Section 2.
w t ∈ R d in × d out is the value of a weight matrix at time t .
z t (cid:44)
x t · w t ∈ R d out is the value of a pre-nonlinearity hidden layer activation in a non-spiking network at time t .
ˆ z t (cid:44) dec ( Q ( enc ( x t )) · w t ) ∈
R d out is the value of a pre-nonlinearity hidden layer activation in the spiking network at time t .
It is an approximation of z t .
Section 2.
( · w l ) indicates applying a function which takes the dot-product of the input with the l ’th weight matrix: ( · w l )( x ) (cid:44) x · w l h l indicates an elementwise nonlinearity (e.g. a ReLU).
Q l indicates the quantization step applied at the l ’th layer (because quantization has internal state, φ , and an associated layer dimension, we use the subscript to distinguish quantizers at different layers.)
dec l , enc l are likewise the (stateful) encoding/decoding functions applied before/after layer l .
∂ L ∂z l is the derivative of the loss with respect the the (pre-nonlinearity) activation of layer l .
( · w Tl ) indicates the dot product with the
transpose
of w l .
This is simply backpropagation across a weight
matrix: If u (cid:44) x · w l , then ∂ L ∂x = ∂
L ∂u · w T l
ˆ
z l is the approximation to the (pre-nonlinearity) activation to layer l (ie the output of dec l ), computed by
the
spiking network.
( (cid:12) h (cid:48) l (ˆ z l ))
is a function that does an elementwise multiplication of the input with the derivative of nonlinearity h l evaluated at ˆ z l .
This is simply backpropagation across a nonlinearity:
If u (cid:44) h l ( x ) , then
∂
L
∂x = ∂
L ∂u
(cid:12)
h (cid:48) l ( x )
dec backl ,
enc
backl , Q backl serve the same functions as dec l , enc l , Q l , but for the backward pass.
(cid:99) ∂
L ∂ ˆ z l ∈
R
d l
Is our approximation to the derivative of the
loss
of our network
with respect to ˆ z l , which is itself an approximation of the activation z l in a non-spiking implementation of the network.
Section 2.
In the updates section we describe how we calculate the weight gradients in layer l .
Because this description holds for any arbitrary layer, we get rid of the layer subscript and use the following notation:
x t (cid:44) h l − 1 ( z l − 1 ,t ) ∈
R d in here is deﬁned as a shorthand for “the input to layer l ”.
e t (cid:44) (cid:100) ∂
L ∂ ˆ z l,t ∈
R
d out is simply a shorthand for “the
approximate backpropagating gradient at layer l
( ¯ x
t
and ¯ e t ) are the encoded and quantized versions of signals ( x t and e t )
( ˆ x
t
and ˆ e t ) are the reconstructed versions of signals ( x t and e t ), taken from the quantized ( ¯ x t and ¯ e t ) (cid:99)
∂ L ∂wrecon,t (cid:44) ˆ x t ⊗ ˆ e
t
∈
R d
in ×
d
out
is the approximate gradient of weight matrix w , as calculated by taking the outer product of the (input, error) reconstructions, ˆ x , ˆ e .
(cid:16) (cid:99)
∂ L ∂wpast,t , (cid:99) ∂
L ∂wfuture,t (cid:17) are the gradient approximations at time t taken using the (past, future) approximation methods, deﬁned in Appendix E .
They are more efﬁcient to calculate than (cid:99) ∂
L ∂wrecon,t but converge to the same value when averaged out over time (see Figure 12).
Section 2.
(cid:99) ∂
L ∂wstdp,t is the gradient approximation taken using the STDP-type update.
It also converges to the same value as (cid:99) ∂
L ∂wrecon,t when averaged out over time.
[Equation]
: A reparametrization of k p , k d in terms of the memory in our decoder k α and the scaling of our encoded signal ( k β ).
This reparametrization is also used when discussing the automatic tuning of k p , k d to match the dynamic range of our data in Appendix F
B R ELATION TO P REDICTIVE C ODING
Our encoding/decoding scheme is an instance of predictive coding - an idea imported from the signal processing literature into neuroscience by Srinivasan et al.
(1982) wherein the power of a transmitted signal is reduced by subtracting away the predictable component of this signal before transmission, then reconstructing it after.
(This requires that the encoder and decoder share the same prediction model)
Bharioke and Chklovskii (2015) formulate feedforward predictive coding as follows (with variables names changed to match the conventions of this paper):
[Equation]
[Equation]
Predictive
[Equation]
[Equation]
Where the reconstruction is done by:
[Equation]
[Equation]
They go on to deﬁne “optimal” liner ﬁlter parameters [ w 1 , w 2 , ... ] that minimize the average magnitude of a t in terms of the autocorrelation and signal-to-noise ratio of x .
Our scheme deﬁnes:
[Equation]
So it is identical to feedforward predictive coding with ω τ =
[Equation]
otherwise up to a scaling constant of ( k p + k d ) .
In our case, the function of this additional constant is to determine the coarseness of the quantization.
From this relationship it is clear that this work could be extended to come up with more efﬁcient predictive coding schemes which could further reduce computation by learning the temporal characteristics of the input signal.
Here we show that Q = ∆ ◦ R ◦ Σ , where Q, ∆ , R, Σ are deﬁned in Equations 3, 2, 6, 1, respectively.
From Equation 3 (Q) we can see that
[Equation]
Now we can unroll for y t and observe use the fact that if s ∈ Z then round ( a + s ) = round ( a ) + s , to say:
[Equation]
At which point it is clear that Q is identical to a successive application of a temporal summation, a rounding, and a temporal difference.
That is why we say Q = ∆ ◦ R ◦ Σ .
D S CANNING THE K- SPACE
Equation 7 shows how we make two approximations when approximating z t = x t · w t with ˆ z t
= ( dec ◦ w ◦ Q ◦ enc )( x t ) .
The ﬁrst is the “nonstationary weight” approximation - arising from the fact that w changes in time, the second is the “quantization” approximation, arising from the quantization of x. Here do a small experiment in which we multiply a time-varying scalar signal x t with a time-varying weight w t for many different values of k p , k d to understand the effects of k p , k d on our approximation error.
[Figure]
E U PDATE A LGORITHMS
In Section 2.6, we visually describe what we call the “Past” and “Future” parameters updates.
Here we give the algorithms for implementing these schemes.
To simplify our expressions in the update algorithms, we re-parametrize our k p , k d coefﬁcients as k α (cid:44) [Equation] Persistent: w
[Equation]
u ∈ R [Equation] x
[Equation]
v (19) [Equation] Persistent
[Equation]
P r e s y n a p t i c
S
i g n a l x t x t
P o s t s y n a p t i c
S
i g n a l e t e t t t = 0 w recon STDP past future
F T UNING k p , k d
The smaller the magnitude of a signal, the more severely distorted it is by our quantizationreconstruction scheme.
We can see that scaling a signal by K has the same effect on the quantized version of the signal, s t , as scaling k p and k d by K: s t =
( Q ◦ enc k p ,k d )( Kx t ) =
Q ( k p
Kx t + k d
( Kx t − Kx t − 1 ))
=
Q ( Kk p x t + Kk d ( x t − x t − 1 ))
=
( Q ◦ enc Kk p ,Kk d )( x t ) .
The fact that the reconstruction quality depends on the signal magnitude presents a problem when training our network, because the error gradients tend to change in magnitude throughout training (they start large, and become smaller as the network learns).
To keep our signal within the useful dynamic range of the quantizer, we apply simple scheme to heuristically adjust k p and k d for the forward and backward passes separately, for each layer of the network.
Instead of directly setting k p , k d as hyperparameters, we ﬁx the ratio
k α (cid:44) k
d k p + k d , and adapt the scale k β (cid:44) 1 k p + k d to the magnitude of the signal.
Our update rule for k β is:
[Equation]
Where η k is the scale-adaptation learning rate, µ t is a rolling average of the L 1 magnitude of signal x t , and k relβ deﬁnes how coarse our quantization should be relative to the signal magnitude (higher means coarser).
We can recover k p , k d for use in the encoders and decoders as k p =
(1 − k α ) /k
β and k d = k
α /k β .
In our experiments, we choose η k = 0 .
001 , k relβ = 0 .
91 , k alpha = 0 .
91 , and initialize µ
= 1 .
Here we show training scores and computation for the PDNN and MLP under different input-orderings (the unordered MNIST vs the ordered Temporal MNIST) and hidden layer depths. We notice no dropoff in performance of the PDNN (as compared to an MLP) with the same architecture as we add hidden layers. For all experiments, the PDNN started with k α = 0 . 5 , and this was increased to k α = 1 after 1 epoch (see Appendix A for the meaning of k α ).
dataset
[Table]
Net
[Table]
mnist [200]
MLP  98.
mnist [200]
PDNN  98.
mnist [200, 200]
MLP 99.
mnist [200, 200]
PDNN 99.
mnist [200, 200, 200]
MLP 99.
mnist [200, 200, 200]
PDNN 99.
temporal_mnist [200]
MLP  98.
temporal_mnist [200]
PDNN  98.
temporal_mnist [200, 200]
MLP 99.
temporal_mnist [200, 200]
PDNN 99.
temporal_mnist [200, 200, 200]
MLP 99.
temporal_mnist [200, 200, 200]
PDNN 99.
H T HE Y OUTUBE D ATASET
[Figure]
I I NSTABILITY IN N EURAL N ETWORK R EPRESENTATIONS
Figure 8 seems to show that computation doesn’t quite approach zero as our frame-rate increases, but ﬂat-lines at a certain point.
We think this may have to do with the fact that hidden layer activations are not necessarily smoother in time than the input.
We demonstrate this with a ﬁgure, which, using 5 video snippets from the Youtube-BB dataset, on the 19 layer VGGNet architectures (Simonyan and Zisserman, 2014), which was pre-trained on ImageNet. Given these 5 snippets, we measures how much the average relative change in layer activation and various layer depths:
[Equation]
) varies as we increase our frame-rate.
We simulate lower frame rates by skipping every N’th frame of video.
(so for example to get a 10FPS frame rate we simply select every 3rd frame of the 30FPS video).
For each selected frame rate, and for the given layers, we measure the average inter-frame change at various layers:
[Equation]
Where: S = 5 is the number of video snippets we average over T is the number of frames in each snippet a t is the activation of a layer at time t n is the number of frames we are skipping over. (Note that we vary our frame-rate artiﬁcially by sub-sampling frames).
This shows something interesting.
While our deeper layers do indeed show less relative change in activation over frames than our input/shallow layers, we note that as frame-rate increases, this seems to approach zero much more slowly than our input/shallow layers.
This is a problem for our method, which relies on temporal smoothness in all layers (especially those hidden layers with large amounts of downstream units) to save computation.
It suggest that methods for learning slow feature detectors - layers that are trained speciﬁcally to look for slowly varying features of the input, may be helpful to us.
[Figure]
R e l a t i v e
C h a n g e i n L a y e r
A c t i v a t i o n
[Figure]