S PIKE G RAD : A N ANN- EQUIVALENT C OMPUTATION M ODEL FOR I MPLEMENTING B ACKPROPAGATION WITH S PIKES
A BSTRACT
Event-based neuromorphic systems promise to reduce the energy consumption of deep neural networks by replacing expensive ﬂoating point operations on dense matrices by low energy, sparse operations on spike events.
While these systems can be trained increasingly well using approximations of the backpropagation algorithm, this usually requires high precision errors and is therefore incompatible with the typical communication infrastructure of neuromorphic circuits.
In this work, we analyze how the gradient can be discretized into spike events when training a spiking neural network.
To accelerate our simulation, we show that using a special implementation of the integrate-and-ﬁre neuron allows us to describe the accumulated activations and errors of the spiking neural network in terms of an equivalent artiﬁcial neural network, allowing us to largely speed up training compared to an explicit simulation of all spike events.
This way we are able to demonstrate that even for deep networks, the gradients can be discretized sufﬁciently well with spikes if the gradient is properly rescaled.
This form of spike-based backpropagation enables us to achieve equivalent or better accuracies on the MNIST and CIFAR10 datasets than comparable state-of-the-art spiking neural networks trained with full precision gradients.
The algorithm, which we call
SpikeGrad , is based on accumulation and comparison operations and can naturally exploit sparsity in the gradient computation, which makes it an interesting choice for a spiking neuromorphic systems with on-chip learning capacities.
I NTRODUCTION
Spiking neural networks (SNNs) are a new generation of artiﬁcial neural network models (Maass, 1997) that try to harness properties of biological neurons to build energy efﬁcient spiking neuromorphic systems.
Processing in traditional artiﬁcial neural networks (ANNs) is based on parallel processing of operations on dense tensors of ﬁxed length.
In contrast to this, spiking neuromorphic systems communicate with asynchronous events, which allows dynamic, data dependent computation that can exploit high temporal and spatial sparsity.
The recent years have seen a large number of approaches devoted to optimization of spiking neural networks with the backpropagation algorithm, either by converting ANNs to SNNs (Diehl et al., 2015;
Esser et al., 2016;
Rueckauer et al., 2017; Sengupta et al., 2019) or by simulating spikes explicitly in the forward pass and optimizing these dynamics with ﬂoating point gradients (Lee et al., 2016;
Yin et al., 2017; Wu et al., 2018b;c; Severa et al., 2019; Jin et al., 2018; Bellec et al., 2018; Zenke & Ganguli, 2018; Shrestha & Orchard, 2018).
These methods aim to optimize SNNs for efﬁcient inference, and backpropagation is performed ofﬂine on a standard computing system.
It would however be desirable to also enable on-chip learning in neuromorphic chips using the power of the backpropagation algorithm, and to maintain the advantages of spike-based processing also in the error propagation phase.
Previous work on the implementation of backpropagation with spikes is mostly concerned with biological plausibility.
A non-spiking version of biologically inspired backpropagation is presented
) introduce spike-based versions of the backpropagation algorithm using variants of (direct) feedback alignment (Lillicrap et al., 2016; Nøkland, 2016).
The exact backpropagation algorithm, which backpropagates through symmetric weights, might however be required to achieve good inference performance on large-scale deep neural networks (Baldi & Sadowski, 2016; Bartunov et al., 2018).
O’Connor & Welling (2016) and Thiele et al.
(2019) present implementations of standard backpropagation where the gradient is coded into spikes and propagated through symmetric weights.
In the same spirit, our work is mostly concerned with exploiting spike based information encoding for energy efﬁcient processing, which means that inference performance and operational simplicity will be preferred over biological plausibility and complex neuron models.
We demonstrate how backpropagation can be seamlessly integrated into the spiking neural network framework by using a second accumulation compartment that discretizes the error into spikes.
By additionally weighting the activity counters by the learning rate, we obtain a system that is able to perform learning and inference based on accumulations and comparisons alone.
As for the forward pass, this allows us to use the dynamic precision computation provided by the discretization of all operations into spike events, and to exploit the sparsity of the gradient.
Using a similar reasoning as
Binas et al.
(2016) and Wu et al.
(2019) have applied to forward propagation in SNNs, we show that the system obtained in this way can be mapped to an integer activation ANN whose activations are equivalent to the accumulated neuron responses for both the forward and the backward propagation phase.
This allows us to simulate training of large-scale SNNs efﬁciently on graphics processing units (GPUs), using their equivalent ANN.
Additionally, in contrast to conversion methods that approximate pre-trained ANNs with SNNs, this method guarantees that the inference precision of the SNN will be equivalent to the ANN.
In contrast to O’Connor & Welling (2016), this is true for any number of spikes and arbitrary spike order.
We demonstrate classiﬁcation accuracies equivalent or superior to existing implementations of SNNs trained with full precision gradients, and comparable to the precision of standard ANNs using similar topologies.
This is the ﬁrst time competitive classiﬁcation performances are reported on the CIFAR10 and CIFAR100 datasets using a large-scale SNN where both training and inference are fully implemented with spikes.
To the best of our knowledge, our work provides for the ﬁrst time a demonstration of how the sparsity of the gradient during backpropagation could be exploited within a large-scale SNN processing structure.
T HE SpikeGrad ALGORITHM
We begin with the description of SpikeGrad , the spike-based backpropagation algorithm.
The algorithm mainly consists of special implementations of the integrate-and-ﬁre (IF) neuron model, that are used to encode and propagate information in the forward and backward pass.
For each training example/mini-batch, integration is performed from t = 0 to t =
T for the forward pass and from t =
T + ∆ t to t = T in the backward pass.
Since no explicit time is used in the algorithm, ∆ t represents symbolically the (very short) time between the arrival of an incoming spike and the response of the neuron, which is only used here to describe causality.
Spike discretization of the forward pass For forward propagation, the architecture is described by multiple layers (labeled by l ∈ [0 , L ] ) of IF neurons with integration variable V li ( t ) and threshold Θ ﬀ :
[Equation]
The variable w lij is the weight and b li a bias value.
The spike activation function s li ( t ) ∈ {− 1 , 0 , 1 } is a function which triggers a signed spike event depending on the internal variables of the neuron.
It will be shown later that the speciﬁc choice of the activation function is fundamental for the mapping to an equivalent ANN.
After a neuron has ﬁred, its integration variable is decremented or incremented by the threshold value Θ ﬀ , which is represented by the second term on the r.h.s.
of equation 1.
As a representation of the neuron activity, we use a trace x li ( t ) which accumulates spike information over a single example:
[Equation]
By weighting the activity with the learning rate η we avoid performing a multiplication when weighting the input with the learning rate for the weight update equation 8.
Implementation of implicit ReLU and surrogate activation function derivative It is possible to deﬁne an implicit activation function based on how the neuron variables affect the spike activation function s li ( t ) .
In our implementation, we use the following fully symmetric function to represent linear activation functions (used for instance in pooling layers):
[Equation]
The following function corresponds to the rectiﬁed linear unit (ReLU) activation function:
[Equation]
The pseudo-derivative of the activation function is denoted symbolically by S (cid:48) li .
We use S (cid:48) l, lin
i
[Equation]
for the linear case.
For the ReLU, we use a surrogate of the form:
[Equation]
These choices will be motivated in the following sections.
Note that the derivatives depend only on the ﬁnal states of the neurons at time T .
Discretization of the error into spikes For gradient backpropagation, we introduce a second compartment with threshold Θ bp in each neuron, which integrates error signals from higher layers.
The process discretizes errors in the same fashion as the forward pass discretizes an input signal into a sequence of signed spike signals:
[Equation]
To this end, we introduce a ternary error spike activation function z li ( t ) ∈ {− 1 , 0 , 1 } which is deﬁned in analogy to equation 3 using the error integration variable U li ( t ) and the backpropagation threshold Θ bp .
The error is then obtained by gating this ternarized variable z li ( t ) with one of the surrogate activation function derivatives of the previous section (linear or ReLU):
[Equation]
This ternary spike signal is backpropagated through the weights to the lower layers and also applied in the update rule of the weight increment
accumulator ω lij :
[Equation]
which is triggered every time an error spike (equation 7) is backpropagated.
The weight updates are accumulated during error propagation and are applied after propagation is ﬁnished to update each weight simultaneously.
In this way, the backpropagation of errors and the weight update will, exactly as forward propagation, only involve additions and comparisons of ﬂoating point numbers.
The SpikeGrad algorithm can also be expressed in an event-based formulation, described in algorithms 1, 2 and 3.
This formulation is closer to how the algorithm would be implemented in an actual SNN hardware implementation of the IF ﬁring dynamics.
Loss function and error scale
We use the cross entropy loss function in the ﬁnal layer applied to the softmax of the total integration V L i ( T ) (no spikes are triggered in the top layer during inference).
This requires more complex operations than accumulations, but is negligible if the number of classes is small.
To make sure that sufﬁcient error spikes are triggered in the top layer, and that error spikes arrive even in the lowest layer of the network, we apply a scaling factor α to the error values before transferring them to U Li .
This scaling factor also implicitly sets the precision of the gradient, since a higher number of spikes means that a large range of values can be represented.
To counteract the relative increase of the gradient scale, the learning rates have to be rescaled by a factor 1 / α .
Input encoding As pointed out in Rueckauer et al.
(2017) and Wu et al.
(2018c)
, it is crucial to maintain the full precision of the input image to obtain good performances on complex standard benchmarks with SNNs.
One possibility is to encode the input in a large number of spikes (Sengupta et al., 2019).
Another possibility, which has been shown to require a much lower number of spikes in the network, is to multiply the input values directly with the weights of the ﬁrst layer (just like in a standard ANN).
The drawback is that the ﬁrst layer then requires multiplication operations.
The additional cost of this procedure may however be negligible if all other layers can proﬁt from spike-based computation.
This problematic does not exist for stimuli which are natively encoded in spikes.
Algorithm 1 Forward function P ROPAGATE ( [ l, i, j ] , s )
V li ← V li + s · w lij s li ← s li ( V li , x li ) (cid:46) spike activation function if s li (cid:54)
=
then
V l i ← V l i − s l
i · Θ ﬀ x li ← x li + ηs li for k in l + 1 connected to i do
P ROPAGATE ( [ l + 1 , k, i ] , s li ) function B ACKPROPAGATE ( [ l, i, k ] , δ )
U li ← U li + δ · w l +1 ki z li ← z li ( U li )
(cid:46) error activation function δ li ← z li · S (cid:48) li if z li (cid:54) =
then
U li ← U li − z li · Θ bp for j in layer l − 1 connected to i do
B ACKPROPAGATE ( [ l − 1 , j, i ] , δ li ) ω
lij ← ω lij − δ li · x l − 1 j
Algorithm init: V ← b , U ← 0 , x ← 0 , ω ←(cid:46) variables in bold describe all neurons in network/layer while input spikes s ini do for k in l = 0 receiving s ini do (cid:46) spikes corresponding to training input
P ROPAGATE ( [0 , k, i ] , s ini )
S (cid:48) ← S (cid:48)
( V , x ) (cid:46) calculate surrogate derivatives
U L ← α · ∂ L / ∂ V L (cid:46) calculate classﬁcation error for i in l =
L do while | U Li | ≥ Θ bp do (cid:46) backpropagate error spikes
B ACKPROPAGATE ( [ L, i, − ] , 0 ) (cid:46) last layer receives no error w ← w + ω
(cid:46) update weights with weight update accumulator
F ORMULATION OF THE EQUIVALENT ANN
The simulation of the temporal dynamics of spikes in SpikeGrad requires a large number of time steps or events if activation or error values are large.
It would therefore be extremely beneﬁcial if we were able to map the SNN to an equivalent ANN that can be trained much faster on standard hardware.
In this section, we demonstrate that it is possible to ﬁnd such an ANN using the forward and backward propagation dynamics of the SpikeGrad IF neuron model described in the previous section.
Spike discretization error We start our analysis with equation 1.
We reorder the terms and sum over the increments ∆ V li ( t ) =
V li ( t + ∆ t ) − V li ( t ) every time the integration variable is changed either by a spike that arrives at time t sj ∈ [0 , T ] via connection j , or by a spike that is triggered at time t si ∈ [0 , T ] .
With the initial conditions V li (0) =
b li , s li (0) =
0 , we obtain the ﬁnal value V li ( T ) :
[Equation]
By deﬁning the total transmitted output of a neuron as S li := (cid:80) t si s li ( t si ) we obtain:
[Equation]
The same reasoning can be applied to backpropagation of the gradient.
We deﬁne the summed responses over error spikes times τ sj ∈
[ T + ∆ t, T ] as Z li := (cid:80) τ si z li
[Equation]
si ) to obtain:
[Equation]
In both equation 10 and equation 11, the terms S li and Z li are equivalent to the output of an ANN with signed integer inputs S l − 1 j and E l +1 k . 1 / Θ ff and 1 / Θ bp are implicit scaling factors of activations and gradients.
If gradients shall not be explicitly rescaled, backpropagation requires Θ bp =
Θ ﬀ .
The values of the residual integrations 1 / Θ ff V li ( T ) and 1 / Θ bp U li ( T ) therefore represent the spike discretization error SDE ﬀ :=
S li − S li or SDE bp :
= Z li − Z li between the ANN outputs S li and
Z li and the accumulated SNN outputs S li and Z li .
Since we know that V li ( T ) ∈ ( − Θ ﬀ , Θ ﬀ ) and
U li ( T ) ∈ ( − Θ bp , Θ bp ) , this gives bounds of | SDE ﬀ | < 1 and | SDE bp | < 1 .
So far we can only represent linear functions.
We now consider an implementation where the ANN applies a ReLU activation function instead.
The SDE in this case is:
[Equation]
We can calculate the error by considering that equation 4 forces the neuron in one of two regimes (note that x li > 0 ⇔ S li > 0 ):
In one case, S li = 0 , V li ( T ) <
Θ ﬀ (this includes V li ( T ) ≤ − Θ ﬀ ).
This implies S li
= 1 / Θ ff V li ( T ) and therefore | SDE ReLUﬀ | < 1 (or even | SDE ReLUﬀ | = 0 if V li ( T ) ≤ 0 ).
In the other case, S li > 0 , V li ( t ) ∈ ( − Θ ﬀ , Θ ﬀ ) , where equation 4 is equivalent to equation 3.
This equivalence motivates the choice of equation 5 as a surrogate derivative for the SNN: the condition ( V li ( T ) > 0 or x li ( T ) > 0) can be seen to be equivalent to S li ( T ) > 0 , which deﬁnes the derivative of a ReLU.
Finally, for the total weight increment ∆ w lij , it can be seen from equationand equation 8 that:
[Equation]
which is exactly the weight update formula of an ANN deﬁned on the accumulated variables.
We have therefore demonstrated that the SNN can be represented by an ANN by replacing all S and Z by S and Z and applying the corresponding activation function directly on these variables.
The error that will be caused by this substitution compared to using the accumulated variables S and Z of an
SNN is described by the SDE.
This ANN can now be used for training of the SNN on GPUs.
The
SpikeGrad algorithm formulated on the variables s , z , δ and x represents the algorithm that would be implemented on a dedicated, event-based spiking neural network hardware implementation of the IF neurons.
We will now demonstrate how the SDE can be further reduced to obtain an ANN and SNN that are exactly equivalent.
Response equivalence For a large number of spikes, the SDE may be negligible compared to the activation of the ANN.
However, in a framework whose objective it is to minimize the number of spikes emitted by each neuron, this error can have a potentially large impact.
One option to reduce the error between the ANN and the SNN output is to constrain the ANN during training to integer values.
One possibility is to round the ANN outputs:
[Equation]
The round function here rounds to the next integer value, with boundary cases rounded away from zero.
This behavior can be implemented in the SNN by a modiﬁed spike activation function which is applied after the full stimulus has been propagated.
To obtain the exact response as the ANN, we have to take into account the current value of S li and modify the threshold values:
[Equation]
Because this spike activation function is applied only to the residual values, we call it the residual spike activation function .
The function is applied to a layer after all spikes have been propagated with the standard spike activation function (equation 3 or equation 4).
We start with the lowest layer and propagate all residual spikes to the higher layers, which use the standard activation function.
We then proceed with setting the next layer to residual mode and propagate the residual spikes.
This is continued until we arrive at the last layer of the network.
By considering all possible rounding scenarios, it can be seen that equation 16 indeed implies:
[Equation]
The same principle can be applied to obtain integer-rounded error propagation:
[Equation]
i [Equation] We have to apply the following modiﬁed spike activation function in the SNN after the full error has been propagated by the standard error spike activation function: [Equation]
otherwise
[Equation]
which implies:
[Equation]
We have therefore shown that the SNN will after each propagation phase have exactly the same accumulated responses as the corresponding integer activation ANN.
The same principle can be applied to obtain other forms of rounding (e.g. ﬂoor and ceil), if equation 16 and equation 19 are modiﬁed accordingly.
Computational complexity estimation Note that we have only demonstrated the equivalence of the accumulated neuron responses.
However, for each of the response values, there is a large number of possible combinations of 1 and − 1 values that lead to the same response.
The computational complexity of the event-based algorithm depends therefore on the total number n of these events.
The best possible case is when the accumulated response value S li is represented by exactly | S li | spikes.
In the worst case, a large number of additional redundant spikes is emitted which sum up to 0 .
The maximal number of spikes in each layer is bounded by the largest possible integration value that can be obtained.
This depends on the maximal absolute weight value w l max , the number of connections
N l in and the number of spike events n l − 1 each connection receives, which is given by the maximal value of the previous layer (or the input in the ﬁrst layer):
[Equation]
The same reasoning applies to backpropagation.
Our experiments show that for input encodings where the input is provided in a continuous fashion, and weight values that are much smaller than the threshold value, the deviation from the best case scenario is rather small.
This is because in this case the sub-threshold integration allows to average out the ﬂuctuations in the signal.
This way the ﬁring rate stays rather close to its long term average and few redundant spikes are emitted.
For the total number of spikes n in the full network on the CIFAR10 test set, we obtain empirically n − n min / n min < 0 . 035 .
E XPERIMENTS
Classiﬁcation performance Tables 1 and 2 compare the state-of-the-art results for SNNs on the
MNIST and CIFAR10 datasets.
It can be seen that in both cases, our results are competitive with respect to the state-of-the-art results of other SNNs trained with high precision gradients.
Compared to results using the same topology, our algorithm performs at least equivalently.
[Table] [Table]
The ﬁnal classiﬁcation performance of the network as a function of the error scaling term α in the ﬁnal layer can be seen in ﬁgure 1.
Previous work on low bitwidth gradients (Zhou et al., 2018) found that gradients usually require a higher precision than both weights and activations.
Our results also indicate that a certain minimum number of error spikes is necessary to achieve convergence.
This strongly depends on the depth of the network and if enough spikes are triggered to provide sufﬁcient gradient signal in the bottom layers.
For the CIFAR10 network, convergence becomes unstable for approximately α < 300 .
If the number of operations is large enough for convergence, the required precision for the gradient does not seem to be extremely high.
On the MNIST task, the difference in test performance between a gradient rescaled by a factor of 50 and a gradient rescaled by a factor of 100 becomes insigniﬁcant.
In the CIFAR10 task, this is true for a rescaling by 400 or 500.
Also the results obtained with the ﬂoat precision gradients in tables 1 and 2 demonstrate the same performance, given the range of the error.
To investigate the performance of SpikeGrad on databases with a larger number of classes, we also ran an additional experiment on the CIFAR100 dataset.
Using the exactly same architecture as for
CIFAR10 with α = 500 (besides the ﬁnal layer that is increased to 100 classes), we obtain a maximal classiﬁcation score of 64 .
Running the same experiment with only the forward pass encoded in spikes, but ﬂoating point gradients, we obtain 64 .
This result could probably be improved by adapting the architecture better to the dataset.
However, it demonstrates that coding the gradient into spikes does not lead to a large precision loss even in this scenario.
Sparsity in backpropagated gradient To evaluate the potential efﬁciency of the spike coding scheme relative to an ANN, we use the metric of relative synaptic operations.
A synaptic operation corresponds to a multiply-accumulate (MAC) in the case of an ANN, and a simple accumulation (ACC) in the case of an SNN.
This metric allows us to compare networks based on their fundamental operation.
The advantage of this metric is the fact that it does not depend on the exact implementation of the operations (for instance the number of bits used to represent each number).
Since an ACC is however generally cheaper and easier to implement than a MAC, we can be sure that an SNN is more efﬁcient in terms of its operations than the corresponding ANN if the number of ACCs is smaller than the number of MACs (without considering potential hardware overheads of event-based computing).
Numbers were obtained with the integer activations of the equivalent ANN to keep simulation times tractable.
As previously explained, the integer response of the equivalent ANN represents the best case scenario for the SNN, i.e. the activation encoding with the lowest number of spikes.
The actual
[Figure] [Figure]
In ﬁgure 1, it can be seen that the number of operations decreases with increasing inference precision of the network.
This is a result of the decrease of error in the classiﬁcation layer, which leads to the emission of a smaller number of error spikes.
Figure 2 demonstrates how the number of operations during the backpropagation phase is distributed in the layers of the network (the ﬂoat input bottom layer and average pooling layers were omitted).
While propagating deeper down the network, the relative number of operations decreases and the error becomes increasingly sparse.
This tendency is consistent during the whole training process for different epochs.
D ISCUSSION AND CONCLUSION
Using spike-based propagation of the gradient, we demonstrated that the paradigm of event-based information propagation can be easily translated to the backpropagation algorithm.
We have not only shown that competitive inference performance can be achieved, but also that gradient propagation seems particularly suitable to leverage spike-based processing by exploiting high signal sparsity.
For both forward and backward propagation, SpikeGrad requires a similar event-based communication infrastructure between neurons, which simpliﬁes a possible spiking hardware implementation.
One restriction of our algorithm is the need for negative spikes, which could be problematic for some neuromorphic hardware platforms.
In particular the topology used for CIFAR10 classiﬁcation is rather large for the given task.
We decided to use the same topologies as other state-of-the-art SNNs to allow for better comparison.
The relatively large number of parameters of these topologies may to a certain extent explain the very low number of relative synaptic operations we observe during backpropagation.
It has been demonstrated that the number of parameters in many ANNs can be strongly reduced without a signiﬁcant impact on performance (Han et al., 2016).
It remains to show that the spike coding approach provides sufﬁcient gains also in topologies that are optimized for minimal computation and memory requirements.
This seems at least possible, since the reduction of operation count in SNNs is mainly caused by two factors: the ﬁrst factor is actual sparsity, that means the number of zeros that are produced by thresholding and the ReLU activation function.
The second factor is the more ﬁne grained distribution of activation and error values.
If most values are represented by small integers, only a few spikes are necessary to encode them.
The consequence is that even topologies that are compressed and therefore have lower sparsity may be able to proﬁt from spike coding if most of the activation or error values are small.
This is typically the case for networks quantized to low precision integer values (see for instance (Zhou et al., 2018) or Wu et al.
(2018a))
and could lead to interesting synergies between our spike-based computation model and ANN quantization methods.
Our analysis of the potential energy efﬁciency of SpikeGrad in a dedicated SNN architecture is based on counting the number of basic operations that are performed at the synapses of a neuron.
This does not take into account the potential hardware overhead that is introduced by the implementation of the
IF neuron model and the event-based processing scheme.
If the low operation count can in the end be translated into higher processing speed or increased energy efﬁciency, compared to a normal ANN, depends on the particular SNN hardware implementation.
Such dedicated neuromorphic hardware is the subject of current research efforts (see for instance Merolla et al.
and Davies et al.
We assumed in this work that such a system might be built and used operation count as our principal metric, since it enables us to easily approximate the energy consumption of a spiking neuromorphic chip if the average cost of such an operation is known.
Our work shows that a hardware implementation of SpikeGrad might proﬁt sufﬁciently from the low operation count that energy or speed improvements could be possible at least in certain cases.
A A PPENDIX B A DDITIONAL EXPLANATIONS B. R OUND FUNCTION
It can be seen that the residual spike activation function covers indeed all possible cases.
In particular, the boundary cases are rounded correctly: for V i
=
Θ ﬀ , we obtain S
= S i + 0 .
For S i ≥ 0 , this should be rounded to round[ S ]
= S i + 1 and for S i < 0 , we should obtain round
[ S ] =
S i .
Similarly, for V i = − Θ ﬀ , we obtain S
= S
i − 0 .
For S i ≤ 0 , this should be rounded to round[ S ]
=
S
i −and for S i > 0 , we should obtain round
[ S ] =
S i .
The same reasoning applies to error propagation.
C A DDITIONAL ARCHITECTURE DETAILS C. A VERAGE P OOLING
All pooling layers are average pooling layers, since these are much easier to implement in a spiking neural network.
Average pooling can simply be implemented as an IF neuron with constant weights: w lij =
[Equation]
p li , (22) where p li is the number of neurons in the pooling window of the neuron.
C. D ROPOUT
Dropout can be implemented by an additional gating variable which randomly removes some neurons during forward and backward propagation.
During learning, the activations of the other neurons have to be rescaled by the inverse of the dropout probability 1 / (1 − p drop ) .
This can be implemented in the SNN framework by rescaling the threshold values by (1 − p drop ) .
C. M OMENTUM
While momentum is not required for the basic algorithm, we used it to speed up training.
In a SNN implementation, this could be implemented by adding additional accumulators that save past gradient information.
C. P ARAMETER PRECISION
All real valued variables are coded with 32 bit ﬂoating point variables.
D E XPERIMENTS D. C OMPUTING FRAMEWORK
All experiments are performed with custom CUDA/cuDNN accelerated C++ code.
Training is performed on RTX 2080 Ti graphic cards.
D. E RROR BARS
For all experiments, the means, errors and maximal values are calculated over 20 simulation runs.
D. P REPROCESSING
No preprocessing is used on the MNIST dataset.
We separate the training set of size 60000 into 50000 training and 10000 validation examples, which are used to monitor convergence.
Testing is performed on the test set of 10000 examples.
For CIFAR10, the values of all color channels are divided by 255 and then rescaled by a factor of 20 to trigger sufﬁcient activation in the network.
The usual preprocessing and data augmentation is applied.
For data augmentation images are padded with the image mean value by two pixels on each side and random slices of 32 × 32 are extracted.
Additionally, images are ﬂipped randomly along the vertical axis.
We separate the training set of size 50000 into 40000 training and 10000 validation examples, which are used to monitor convergence.
Testing is performed on the test set ofexamples.
Final scores were obtained without retraining on the validation set.
[Table] [Table]
D. H YPERPARAMETERS
The hyperparameters for training can be seen in tables 3 and
[Table]
The maximal inference performances in the results were achieved with α = 100 for MNIST and α = 400 for CIFAR
[Table]