Temporally Efficient Deep Learning with Spikes
Anonymous authors
Paper under double-blind review
Abstract
The vast majority of natural sensory data is temporally redundant.
For instance, video frames or audio samples which are sampled at nearby points in time tend to have similar values.
Typically, deep learning algorithms take no advantage of this redundancy to reduce computations.
This can be an obscene waste of energy.
We present a variant on backpropagation for neural networks in which computation scales with the rate of change of the data - not the rate at which we process the data.
We do this by having neurons communicate a combination of their state, and their temporal change in state.
Intriguingly, this simple communication rule give rise to units that resemble biologically-inspired leaky integrate-and-ﬁre neurons, and to a weight-update rule that is equivalent to a form of Spike-Timing
Dependent Plasticity (STDP), a synaptic learning rule observed in the brain.
We demonstrate that on MNIST, on a temporal variant of MNIST, and on Youtube-BB, a dataset with videos in the wild, our algorithm performs about as well as a standard deep network trained with backpropagation, despite only communicating discrete values between layers.
Introduction
Currently, most algorithms used in Machine Learning work under the assumption that data points are independent and identically distributed, as this assumption provides good statistical guarantees for convergence.
This is very diﬀerent from the way data enters our brains.
Our eyes receive a single, never-ending stream of temporally correlated data.
We get to use this data once, and then it’s gone.
Moreover, most sensors produce sequential, highly overlapping and redundant streams of data.
This can be both a blessing and a curse.
From a statistical learning point of view this redundancy may lead to biased estimators and eventually unnecessarily bad predictions.
However, this temporal redundancy also implies that intuitively not all computations are necessary.
Online Learning is the study of how to learn in this domain - where data becomes available in sequential order and is given to the model only once.
Given the enormous amount of sequential data, mainly videos, that are being produced nowadays, it seems desirable to develop learning systems that simply consume data on-the-ﬂy as it is being generated, rather than collect it into datasets for oﬄine-training.
There is, however a problem of eﬃciency, which we hope to illustrate with two examples:
CCTV feeds.
CCTV Cameras collect an enormous amount of data from mostly-static scenes.
The amount of new information in a frame, given the previous frame, tends to be low.
That is to say, data tends to be temporally redundant.
If we want to train a model from of this data (for example a pedestrian detector), we need process a large amount of mostly-static frames.
If the frame rate doubles, so does the amount of computation.
Intuitively, it feels that this should not be necessary.
It would be nice to still be able to use all this data, but have the amount of computation scale with the amount of new information in each frame, not just the number of frames and dimensions of the data.
Robot perception .
Robots have no choice but to learn online - their future input data ( e.g. camera frames) are dependent on their previous predictions ( i.e. motor actions).
Not only does their data come in nonstationary temporal streams, but it typically comes from several sensor running at diﬀerent rates.
The camera may produce 1MB images at 30 frames/s, while the gyro might produce 1-byte readings 1000 frames/s.
It’s not obvious, using current methods in deep learning, how we can integrate asynchronous sensory signals into a uniﬁed, trainable, latent representation, without undergoing the horribly ineﬃcient process of recomputing the function of the network every time a new signal arrives.
These examples point to the need for a training method where the amount of computation required to update the model scales with the amount of new information in the data, and not just the dimensionality of the data.
There has been a lot of work on increasing the computational eﬃciency of neural networks by quantizing neural weights or activations (see Section 4), but comparatively little work on exploiting redundancies in the data to reduce the amount of computation.
O’Connor and
Welling (2016b), set out to exploit the temporal redundancy in video, by having neurons only send their quantized changes in activation to downstream neurons, and having the downstream neurons integrate these changes over time.
This approach works for eﬃciently approximating the function of the network, but fails for training, because when the weights are changing with time, this approach (take the temporal diﬀerence, multiply by weights, temporally integrate) fails to reconstruct the correct activation for the next layer.
In other words, (cid:80)[-tτ =
-]
[Equation] = x t · w t unless w t = w 0 ∀ t .
Figure 2 describes the problem visually.
In this paper, we correct for this problem by encoding a mixture of two components of the layers activation x t : the proportional component k p x t , and the derivative component k d ( x t − x t − 1 ) .
When we invert this encoding scheme, we get get a decoding scheme which corresponds to taking an exponentially decaying temporal average of past inputs.
Interestingly, the resulting neurons begin to resemble models of biological spiking neurons, whose membrane potentials can approximately be modeled as an exponentially decaying temporal average of past inputs.
In this work, we present a scheme wherein the temporal redundancy of input data is used to reduce the computation required to train a neural network.
We demonstrate this on the
MNIST and Youtube-BB datasets.
To our knowledge we are the ﬁrst to create a neural network training algorithm which uses less computation as data becomes more temporally redundant.
Methods
We propose a coding scheme where neurons can represent their activations as a temporally sparse series of impulses.
The impulses from a given neuron encode a combination of the value and the rate of change of the neuron’s activation.
While our algorithm is designed to work eﬃciently with temporal data , we do not aim to learn temporal sequences in this work.
We aim to eﬃciently approximate a function y t = f ( x t ) , where the current target y t is solely a function of the current input x t , and not previous inputs x 0 ...x
t − 1 .
The temporal redundancy between neighbouring inputs x t − 1 , x t will however be used to make our approximate computation of this function more eﬃcient.
2. Preliminary
Throughout this paper we will use the notation ( f 3 ◦ f 2 ◦ f 1 )( x ) = f 3 ( f 2 ( f 1 ( x ))) to denote function composition.
We slightly abuse the notion of functions by allowing them to have an internal state which persists between calls.
For example, we deﬁne the ∆ function in Equation 1 as being the diﬀerence between the inputs in two consecutive calls (where persistent variable x last is initialized to 0).
The Σ function, deﬁned in Equation 2, returns a running sum of the inputs over calls.
So we can write, for example, that when our composition of functions (Σ ◦ ∆) is called with a sequence of input variables x τ : τ = [1 ..t ] , then (Σ ◦ ∆)( x t )
= x t , because y 0 + ( x 1 − x 0 ) + ( x 2 − x 1 ) + ... + ( x t − x t − 1 ) | x 0 =0 ,y
=
= x t .
In general, when we write y t = f ( x t ) , where f is a function with persistent state, it will be implied that we have previously called f ( x τ ) for τ ∈ [1 , .., t − 1] in sequence.
Variable deﬁnitions that are used later will be highlighted in blue.
2. PD Encoding
∆ : x (cid:55)→ y ; Persistent: x last ←y ← x − x last x last ← x (1) Σ : x (cid:55)→ y ;
Persistent: y ←y ← y + x (2)
Q : x (cid:55)→ y ; Persistent: φ ←φ (cid:48) ← φ + x y ← round ( φ (cid:48) )
φ ← φ (cid:48) − y (3) enc : x (cid:55)→ y ;
Persistent: x last ←y ← k p x + k d ( x − x last ) x last ← x (4) dec : x (cid:55)→ y ; Persistent: y ←y ← x + k d y k p + k d (5)
R : x (cid:55)→ round ( x ) (6)
Suppose a neuron has time-varying activation x τ : τ ∈ [1 ..t ] .
Taking inspiration from
Proportional-Integral-Derivative (PID) controllers, we can “encode” this activation at each time step as a combination of its current activation and change in activation as a t (cid:44) enc ( x t ) = k p x t + k d ( x [Equation]
− x
[Equation]
1 ) , (see
Equation 4).
The parameters k p and k d determine what portion of our encoding represents the value of the activation and the rate of change of that value, respectively.
In Section D, we will discuss the eﬀect our choices for these parameters have on the network.
To get our decoding formula, we can simply solve for x t as x t = a t + k d x t − 1 k p + k d (Equation 5), such that ( dec ◦ enc )( x t ) = x t .
Notice that Equation 5 corresponds to decaying the previous decoder state by some constant k d / ( k p + k d ) and then adding the input a t / ( k p + k d ) .
We can expand this recursively to see that this corresponds to a temporal convolution a ∗ κ where κ is a causal exponential kernel κ τ = (cid:110)
[Equation] if [Equation]
2. Quantization
Our motivation for the aforementioned encoding scheme is that we want a sparse signal which can be quantized into a low-bitrate discrete signal.
This will later be used to reduce computation.
We can quantize our signal a t into a sparse, integer signal s t (cid:44) Q ( a t ) , where the quantizer Q is deﬁned in Equation 3.
Equation 3 implements a form of Sigma-Delta modulation, a method widely used in signal processing to approximately communicate signals at low bit-rates (Candy and Temes, 1962).
We can show that that Q ( x t ) =
(∆ ◦ R ◦ Σ)( x t ) (See Supplementary Material Section A), where ∆ ◦ R ◦ Σ indicates applying a temporal summation, a rounding, and a temporal diﬀerence, in series.
If x t is temporally redundant, then | a t | (cid:28) 1 ∀ t , and we can expect s t to consist of mostly zeros with a few 1’s and -1’s.
We can now approximately reconstruct our original signal x t as ˆ x t (cid:44) dec ( s t ) by applying our decoder, as deﬁned in Equation 5.
As our coeﬃcients k p , k
d become larger, our reconstructed signal ˆ x t should become closer to the original signal x t .
We illustrate examples of encoded signals and their reconstructions for diﬀerent k p , k d in Figure 1.
2.3. Special cases
We can compactly write the entire reconstruction function as ˆ x = ( dec ◦ ∆ ◦ R ◦ Σ ◦ enc )( x t ) .
k p = 0 :
When k p = 0 , we get dec ( x t ) =
( k − 1 d ◦ Σ)( x t ) and enc ( x t ) =
( k d ◦ ∆)( x t ) , so our reconstruction reduces to ˆ x =
( k − 1 d ◦ Σ ◦ ∆ ◦ R ◦ Σ ◦ k d ◦ ∆)( x t ) .
Because Σ ◦ k d ◦ ∆ all [Figure] 2 anticipates this question and answers it visually [Figure]
2000 t 2.
[Figure]
[Figure]
p =
[Figure]
x t w t z t = x t w t z t = dec k p k d ( Q ( enc k p k d ( x t )) w t )
Introducing k p allows us to bring our reconstruction back in line with the correct signal.
commute with one another, we can simplify this to ˆ x t =
( k − 1 d ◦ R ◦ k d )( x t ) .
so our decoded signal is ˆ x t = round ( x t · k d ) /k
d , with no dependence on x t − 1 .
This is visible in the bottom row of Figure 1.
This was the encoding scheme used in O’Connor and Welling (2016b).
k d = 0 :
In this case, dec ( x t ) =
k − 1 p x t and enc ( x t ) =
k p x t so our encoding-decoding process becomes ˆ x = ( k − 1 p ◦ ∆ ◦ R ◦ Σ ◦ k p )( x t ) .
In this case neither our encoder nor our decoder have any memory, and we take not advantage of temporal redundancy.
2. Sparse Communication Between Layers
The purpose of our encoding scheme is to reduce computation by sparsifying communication between layers of a neural network.
Suppose we are trying to compute the pre-nonlinearity activation of the ﬁrst hidden layer, z t ∈ R d out , given the input activation, x t ∈ R d in .
We approximate z t as:
[Equation]
where: x t , ˆ x t ∈ R d in
[Equation]
s t [Equation] in
[Equation]
w [Equation] z [Equation]
The ﬁrst approximation comes from the quantization (Q) of the encoded signal, and the second from the fact that the weights change over time, as explained in Figure 2.
The eﬀects of these approximations are further explored in Section B of the Supplementary Material.
Computing z t takes d in · d out multiplications and ( d in − 1) · d out additions.
The cost of computing ˆ z t , on the other hand, depends on the contents of s t .
If the data is temporally redundant, s t ∈ I d in should be sparse, with total magnitude S (cid:44) (cid:80) i | s t,i | .
s t can be decomposed into a sum of one-hot vectors s t =
(cid:80) Sn =1 sign ( s t,i n ) γ i n : i n ∈ [1 ..d in ] where γ i n ∈ I d in is a onehot vector with element γ i n = 1 .
The matrix product s t · w can then be decomposed into a series of row additions:
[Equation]
If we include the encoding, quantization, and decoding operations, our matrix product takes a total of 2 d in + 2 d out multiplications, and (cid:80) n | s t,n | · d out + 3 d in + d out additions.
Assuming the (cid:80) n | s t,n | · d out term dominates, we can say that the relative cost of computing ˆ z t vs z t is: cost [Equation] ˆ
[Equation]
2. A Neural Network
We can implement this encoding scheme on every layer of a neural network.
Given a standard neural net f nn consisting of alternating linear ( · w l ) and nonlinear ( h l ) operations, our network function f pdnn can then be written as:
[Equation]
We can use the same approach to approximately calculate our gradients to use in training.
If we deﬁne our layer activations as ˆ z l (cid:44) ( dec ◦ · w l ◦ Q ◦ enc )( x ) if l = 1 otherwise ( dec ◦ · w l ◦
Q ◦ enc )
(ˆ z l − 1 ) , and L (cid:44) (cid:96) ( f pdnn ( x ) , y ) , where (cid:96) is some loss function and y is a target, we can backpropagate the approximate gradients as:
[Equation]
On every layer of the forward and backward pass, our quantization scheme corrupts the signals that are being sent between layers.
Nevertheless we ﬁnd experimentally that this does not matter much to the performance of the network.
2. Parameter Updates
There’s no use having an eﬃcient backward pass if the parameter updates aren’t also eﬃcient.
In a normal neural network trained with backpropagation and simple stochastic gradient x [Figure] x spike e e e t wt = x t e t wt = x t e t wt , past wt , future descent, the parameter update for weight matrix w has the form w ← w − η ∂
L ∂w where η is the learning rate.
If w connects layer l − 1 to layer l , we can write ∂ L ∂w = x t ⊗ e t where x t (cid:44) h l − 1 ( z l − 1 ,t ) ∈
R d in is the presynaptic activation, e t (cid:44) ∂
L ∂z l,t ∈ R d out is the postsynaptic (pre-nonlinearity) activation and ⊗ is the outer product.
So we pay d in · d out multiplications to update the parameters for each sample.
We want a more eﬃcient way to compute this product, which takes advantage of the sparsity of our encoded signals to reduce computation.
We can start by applying our encodingquantizing-decoding scheme to our input and error signals as ¯ x t (cid:44) ( Q ◦ enc )( x t ) ∈
I d in and ¯ e t (cid:44) ( Q ◦ enc )( e t ) ∈
I d out , and approximate our true update update as (cid:99)
∂
L ∂wrecon,t (cid:44) ˆ x t ⊗ ˆ e t where ˆ x t (cid:44) dec (¯ x t ) and ˆ e t (cid:44) dec (¯ e t ) .
This doesn’t do any good by itself, because the update rule still is not sparse.
But, we can exactly compute the sum of this value over time using one of two sparse update schemes - past updates and future updates - which are depicted in
Figure 3.
Past Updates : For a given synapse w i,j , if either the presynaptic neuron spikes ( ¯ x t i (cid:54) = 0 ) or the postsynaptic neuron spikes ( ¯ e t i (cid:54) = 0 ), we increment the w i,j by the total area under ˆ x τ,i ˆ e τ,j since the last spike.
We can do this eﬃciently because between the current time and the time of the previous spike, ˆ x τ,i ˆ e τ,j is a geometric sequence.
Given a known initial value u , ﬁnal value v , and decay rate r , a geometric sequence sums to u − v 1 − r .
The area calculated is shown in pink on the bottom row of Figure 3, and one algorithm to calculate it is in
Equation 15.
Future Updates : Another approach is to calculate the Present Value of the future area under the integral from the current spike.
This is depicted in the blue-gray area in Figure 3, and the formula is in Equation 16.
We give the formula for the Past and Future update rules in Appendix C.
Finally, because the magnitude of our gradient varies greatly over training, we create a scheme for adaptively tuning k β to match the running magnitude of the data.
This is described in detail in Appendix D.
[Figure]
[Figure]
2. Relation to STDP
An extremely attentive reader might have noted that Equation 16 has the form of an online implementation of Spike-Timing Dependent Plasticity (STDP).
STDP (Markram et al., 2012) emerged from neuroscience, where it was observed that synaptic weight changes appeared to be functions of the relative timing of pre- and post-synaptic spikes.
The empirically observed function usually has the double-exponential form seen on the rightmost plot of Figure 4.
Using the quantized input signal ¯ x and error signal ¯ e , and their reconstructions ˆ x t and ˆ e t as deﬁned in the last section, we deﬁne a causal convolutional kernel κ t = (cid:110)
k β ( k α ) t
if t ≥ 0 otherwise 0 (cid:111) and g t = { κ t if t ≥ 0 otherwise κ − t } = k β ( k α ) | t | where t ∈ I .
The middle plot of Figure 4 is a plot of g .
We deﬁne our STDP update rule as:
[Equation]
[Equation]
We note that while our version of STDP has the same double-exponential form as the classic
STDP rule observed in neuroscience (Markram et al., 2012), we do not have the property that sign of the weight change depends on whether the presynaptic spike preceded the postsynaptic spike.
In Section C in the supplementary material we show experimentally that while Equations (cid:99) ∂
L ∂wrecon , (cid:99) ∂
L ∂wpast , (cid:99) ∂
L ∂wfuture , (cid:99) ∂
L ∂wstdp may all result in diﬀerent updates at diﬀerent times, the rules are equivalent in that for a given set of pre/post-synaptic spikes ¯ x, ¯ e , the cumulative sum of their updates over time converges exactly.
Experiments 3. Temporal MNIST
To evaluate our network’s ability to learn, we run it on the standard MNIST dataset, as well as a variant we created called “Temporal MNIST”.
Temporal MNIST is simply a reshuﬄing of the MNIST dataset so that so that similar inputs (in terms of L2-pixel distance), are put together.
Figure 6 shows several snippets of consecutive frames in the temporal MNIST dataset.
We compare our Proportional-Derivative Net against a conventional Multi-Layer
Perceptron with the same architecture (one hidden layer of 200 ReLU hidden units and a softmax output).
The results are shown in Figure 5.
Somewhat surprisingly, our predictor slightly outperformed the MLP, getting 98.36% on the test set vs 98.25% for the MLP.
We assume this improvement is due to the regularizing eﬀect of the quantization.
On Temporal
MNIST, our network was able to converge with less computation than it required for MNIST (It used 32 · 10 12 operations for MNIST vs 15 · 10 12 for Temporal MNIST), but ended up with a slightly worse test score when compared with the MLP (the PDNN got 97.99% vs 98.28% for the MLP). [Figure] be at hardware
3. YouTube Video Dataset
Next, we want to simulate the setting of CCTV cameras, discussed in 1, where we have a lot of data with only a small amount of new information per frame.
In the absence of large enough public CCTV video datasets we investigate the surrogate task of frame-based object classiﬁcation on wild YouTube videos from the large, recently released Youtube-BB dataset Real et al.
Our subset consists of 358 Training Videos and 89 Test videos with 758,033 frames in total.
Each video is labeled with an object in one of 24 categories.
We start from a VGG19 network (Simonyan and Zisserman, 2014): a 19-layer convolutional network pre-trained on imagenet.
We replace the top three layer with three of our own randomly initialized layers, and train the network both as a spiking network, and as a regular network with backpropagation.
While training the entire spiking network end-to-end works, we choose to only train the top layers, in order to speed up our training time.
We compare our training scores and computation between a spiking and non-spiking implementation.
Figure 7 shows that our spiking network performs comparably to a non-spiking network, and Figure 8 shows how the computation per frame of our spiking network decreases as we increase the frame rate (i.e. as the input data becomes more temporally redundant).
Because our spiking network uses only additions, while a regular deep network does multiplyadds, we use the estimated energy-costs per op of Horowitz (2014) to compare computations to a single scale, which estimates the amount of energy required to do multiplies and adds in ﬁxed-point arithmetic.
Related Work
There has been sparse but interesting work on merging the notions of spiking neural networks and deep learning.
Diehl et al.
(2015) found a way to eﬃciently map a trained neural [Figure] [Figure] network onto a spiking network.
Lee et al.
(2016) devised a method for training spiking of integrate-and-ﬁre spiking neurons with backpropagation - though their neurons did not send a temporal diﬀerence of their activations.
O’Connor and Welling (2016a) created a method for training event-based neural networks - but their method took no advantage of temporal redundancy in the data.
Binas et al.
(2016) and (O’Connor and Welling, 2016b)
both took the approach of sending quantized temporal changes reduce computation on temporally redundant data but their schemes could not be used to train a neural network.
Bohte et al.
(2000) showed how could apply backpropagation for training spiking neural networks but it was not obvious how to apply the method to non-spiking data.
Zambrano and Bohte (2016) developed a spiking network with an adaptive scale of quantization (which bears some resemblance to our tuning scheme described in Section D), and show that the spiking mechanism is a form of Sigma-Delta modulation, which we also use here.
Similarly, Yoon (2017) found standard spiking neuron models actually implement a form of Sigma-Delta modulation.
Courbariaux et al.
(2015) showed that neural networks could be trained with binary weights and activations (we just quantize activations).
Bengio et al.
(2015) found a connection between the classic STDP rule (Figure 4, right) and optimizing a dynamical neural network, although the way they arrived at an STDP-like rule was quite diﬀerent from ours.
Discussion
We set out with the objective of reducing the computation in deep networks by taking advantage of temporal redundancy in data.
We described a simple rule (Equation 4) for sparsifying the communication between layers of a neural network by having our neurons communicate a combination of their temporal change in activation, and the current value of their activation.
We show that it follows from this scheme that neurons should behave as leaky integrators (Equation 5).
When we quantize our neural activations with Sigma-Delta modulation, a common quantization scheme in signal processing, we get something resembling a leaky integrate-and-ﬁre neuron.
We derive eﬃcient update rules for the weights of our network, and show these to be equivalent to a form of STDP - a learning rule ﬁrst observed in neuroscience.
Finally, we train our network, verify that it does indeed compute more eﬃciently on temporal data, and show that it performs about as well as a traditional deep network of the same architecture, but with signiﬁcantly reduced computation.
Finally, we show that our network can train on real video data.
The eﬃciency of our approach hinges on the temporal redundancy of our input data and neural activations.
There is an interesting synergy here with the concept of slow-features (Wiskott, 1999).
Slow-Feature learning aims to discover latent objects that persist over time.
If the hidden units were to speciﬁcally learn to respond to slowly-varying features of the input, the layers in a spiking implementation of such a network would have to communicate less often.
In such a network, the tasks of feature-learning and reducing inter-layer communication may be one and the same.
A Sigma-Delta Unwrapping
Here we show that Q = ∆ ◦ R ◦ Σ , where Q, ∆ , R, Σ are deﬁned in Equations 3, 2, 6, 1, respectively.
From Equation 3 (Q) we can see that
[Equation]
Now we can unroll for y t and observe use the fact that if s ∈ I then round ( a + s )
= round ( a )+ s , to say:
[Equation]
At which point it is clear that Q is identical to a successive application of a temporal summation, a rounding, and a temporal diﬀerence.
That is why we say Q = ∆ ◦ R ◦ Σ .
B Scanning the K-space
Equation 7 shows how we make two approximations when approximating z t = x t · w t with ˆ z t
= ( dec ◦ w ◦ Q ◦ enc )( x t ) .
The ﬁrst is the “nonstationary weight” approximation - arising from the fact that w changes in time, the second is the “quantization” approximation, arising from the quantization of x. Here do a small experiment in which we multiply a time-varying scalar signal x t with a time-varying weight w t for many diﬀerent values of k p , k d to understand the eﬀects of k p , k d on our approximation error.
Example: k p =0.01, k d =1.
x t Q ( enc ( x t ))
w t     20000 t
[Figure]
x t w t dec ( enc ( x t ) w t ) dec ( Q ( enc ( x t ))
w t ) 0.
k p
S C ( x w , dec ( enc ( x ) w )) 0.40.50.
k p
S C ( x w , dec ( Q ( enc ( x )))
w ) 0.40.50.
[Figure] [Figure]
[Figure]
k p
S C ( x w , dec ( Q ( enc ( x )) w )) 0.40.50.
0.1 0.44 2.3 10 k d log 10 ( N ops ) 1.752.002.252.
[Figure]
Equation
C Update Algorithms
In Section 2.6, we visually describe what we call the “Past” and “Future” parameters updates.
Here we give the algorithms for implementing these schemes.
To simplify our expressions in the update algorithms, we re-parametrize our k p , k d coeﬃcients as k α (cid:44) = k
[Equation]
k β (cid:44) [Equation] + [Equation]
d .
P r e s y n a p t i c
S
i g n a l x t x t
P o s t s y n a p t i c
S
i g n a l e t e t t t = 0 w recon STDP past future
In Section 2.6 and 2.7, we described 4 diﬀerent update rules, and stated that while they do not necessarily produce the same updates same times, they produce the same we demonstrate this empirically.
We generate random
[Equation]
Persistent: w, u
∈
R d in
[Equation]
x
[Equation]
i ← ¯ x (cid:54) = 0 , j ← ¯ e (cid:54) =
ˆ x ← k α ˆ x , ˆ e ← k α ˆ e v ← ˆ x
i ⊗ ˆ e j ∈ R (cid:80) i (cid:48)
[¯ x i (cid:48) (cid:54) =0] × (cid:80) j (cid:48)
[¯ e j (cid:48) (cid:54) =0] (cid:100)
∂
L ∂w
[Equation]
j ← u i,j − v 1 − k 2 α ˆ x ← ˆ x + k β ¯ x,
[Equation]
u i,j
[Equation]
v
[Equation]
D Tuning k p , k d
The smaller the magnitude of a signal, the more severely distorted it is by our quantizationreconstruction scheme.
We can see that scaling a signal by K has the same eﬀect on the quantized version of the signal, s t , as scaling k p and k d by K: s t =
( Q ◦ enc k p ,k d )( Kx t ) =
Q ( k p
Kx t + k d
( Kx t − Kx t − 1 ))
=
Q ( Kk p x t + Kk d ( x t − x t − 1 ))
=
( Q ◦ enc Kk p ,Kk d )( x t ) .
The fact that the reconstruction quality depends on the signal magnitude presents a problem when training our network, because the error gradients tend to change in magnitude throughout training (they start large, and become smaller as the network learns).
To keep our signal within the useful dynamic range of the quantizer, we apply simple scheme to heuristically adjust k p and k d for the forward and backward passes separately, for each layer of the network.
Instead of directly setting k p , k d as hyperparameters, we ﬁx the ratio
k α (cid:44) k
d k p + k d , and adapt the scale k β (cid:44) 1 k p + k d to the magnitude of the signal.
Our update rule for k β is:
[Equation]
Where η k is the scale-adaptation learning rate, µ t is a rolling average of the L 1 magnitude of signal x t , and k relβ deﬁnes how coarse our quantization should be relative to the signal magnitude (higher means coarser).
We can recover k p , k d for use in the encoders and decoders as k p =
(1 − k α ) /k
β and k d = k
α /k β .
In our experiments, we choose η k = 0 .
001 , k relβ = 0 .
91 , k alpha = 0 .
91 , and
[Table]
µ 0 =
E The Youtube Dataset
[Figure]
F Instability in Neural Network Representations
Figure 8 seems to show that computation doesn’t quite approach zero as our frame-rate increases, but ﬂat-lines at a certain point.
We think this may have to do with the fact that hidden layer activations are not necessarily smoother in time than the input.
We demonstrate this with a ﬁgure, which, using 5 video snippets from the Youtube-BB dataset, measures how much the average relative change in layer activation:
[Equation]
varies as we increase our frame-rate.
(Note that we vary our frame-rate artiﬁcially by sub-sampling frames).
[Figure]
R e l a t i v e
C h a n g e i n L a y e r
A c t i v a t i o n
[Figure]