Original: R OBUST AND ACCELERATED SINGLE - SPIKE SPIKING NEURAL NETWORK TRAINING WITH APPLICABILITY TO CHALLENGING TEMPORAL TASKS
Revised: R OBUST AND ACCELERATED SINGLE - SPIKE SPIKING NEURAL NETWORK TRAINING WITH APPLICABILITY TO CHALLENGING TEMPORAL TASKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Spiking neural networks (SNNs), particularly the single-spike variant in which neurons spike at most once, are considerably more energy efﬁcient than standard artiﬁcial neural networks (ANNs).
Revised: Spiking neural networks (SNNs), particularly the single-spike variant in which neurons spike at most once, are considerably more energy efﬁcient than standard artiﬁcial neural networks (ANNs).

Original: However, single-spike SSNs are difﬁcult to train due to their dynamic and non-differentiable nature, where current solutions are either slow or suffer from training instabilities.
Revised: However, single-spike SSNs are difﬁcult to train due to their dynamic and non-differentiable nature, where current solutions are either slow or suffer from training instabilities.

Original: These networks have also been critiqued for their limited computational applicability such as being unsuitable for time-series datasets.
Revised: These networks have also been critiqued for their limited computational applicability such as being unsuitable for time-series datasets.

Original: We propose a new model for training single-spike SNNs which mitigates the aforementioned training issues and obtains competitive results across various image and neuromorphic datasets, with up to a 13.98 × training speedup and up to an 81% reduction in spikes compared to the multi-spike SNN.
Revised: We propose a new model for training single-spike SNNs which mitigates the aforementioned training issues and obtains competitive results across various image and neuromorphic datasets, with up to a 13.98 × training speedup and up to an 81% reduction in spikes compared to the multi-spike SNN.

Original: Notably, our model performs on par with multispike SNNs in challenging tasks involving neuromorphic time-series datasets, demonstrating a broader computational role for single-spike SNNs than previously believed.
Revised: Notably, our model performs on par with multispike SNNs in challenging tasks involving neuromorphic time-series datasets, demonstrating a broader computational role for single-spike SNNs than previously believed.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Artiﬁcial neural networks (ANNs) have achieved impressive feats over recent years, obtaining human-level performance on visual and auditory tasks (Hinton et al., 2012;
Revised: Artiﬁcial neural networks (ANNs) have achieved impressive feats over recent years, obtaining human-level performance on visual and auditory tasks (Hinton et al., 2012;

Original: He et al., 2016), natural language processing (Brown et al., 2020) and challenging games (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019).
Revised: He et al., 2016), natural language processing (Brown et al., 2020) and challenging games (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019).

Original: However, as the difﬁculty and complexity of the tasks increase, so has the size of the networks required to solve them, demanding a substantial and unsustainable amount of energy (Strubell et al., 2019; Schwartz et al., 2020).
Revised: However, as the difﬁculty and complexity of the tasks increase, so has the size of the networks required to solve them, demanding a substantial and unsustainable amount of energy (Strubell et al., 2019; Schwartz et al., 2020).

Original: Inspired by the extreme energy efﬁciency of the brain (Sokoloff, 1960), spiking neural networks (SNNs) emulated on neuromorphic computers attempt to solve this dilemma, requiring signiﬁcantly less energy than ANNs (Wunderlich et al., 2019).
Revised: Inspired by the extreme energy efﬁciency of the brain (Sokoloff, 1960), spiking neural networks (SNNs) emulated on neuromorphic computers attempt to solve this dilemma, requiring signiﬁcantly less energy than ANNs (Wunderlich et al., 2019).

Original: These networks are of growing interest, obtaining noteworthy results on visual (Fang et al., 2021;
Revised: These networks are of growing interest, obtaining noteworthy results on visual (Fang et al., 2021;

Original: Zhou & Li, 2021), auditory (Yin et al., 2020; Yao et al., 2021) and reinforcement learning problems (Patel et al., 2019; Tang et al., 2020; Bellec et al., 2020).
Revised: Zhou & Li, 2021), auditory (Yin et al., 2020; Yao et al., 2021) and reinforcement learning problems (Patel et al., 2019; Tang et al., 2020; Bellec et al., 2020).

Original: A particular class of SNNs in which individual neurons respond with at most one spike aims to further amplify the energy and scaling advantages of SNNs.
Revised: A particular class of SNNs in which individual neurons respond with at most one spike aims to further amplify the energy and scaling advantages of standard SNNs and ANNs.

Original: Inspired by the sparse spike processing shown to exist at least for certain stimuli in the auditory and visual systems (Heil, 2004; Gollisch & Meister, 2008), and forming a class of universal function approximator (Comsa et al., 2020), these networks obtain extreme energy efﬁciency due to their single-spike nature (Oh et al., 2021; Liang et al., 2021).
Revised: Inspired by the sparse spike processing shown to exist at least for certain stimuli in the auditory and visual systems (Heil, 2004; Gollisch & Meister, 2008), and forming a class of universal function approximator (Comsa et al., 2020), these networks obtain extreme energy efﬁciency due to their singlespike nature (Oh et al., 2021; Liang et al., 2021).

Original: Although providing a promising path toward building very large and energy-efﬁcient networks, we are yet to understand how to properly train these SNNs.
Revised: Although providing a promising path toward building very large and energy-efﬁcient networks, we are yet to understand how to properly train these SNNs.

Original: The success of the backprop training algorithm in ANNs does not naturally transfer to single- and multispike SNNs due to their non-differentiable activation function.
Revised: The success of the backprop training algorithm in ANNs does not naturally transfer to single- and multi-spike SNNs due to their non-differentiable activation function.

Original: Current attempts at training are either slow (as time is sequentially simulated) or suffer from training instabilities (e.g. the dead neuron problem) and idiosyncrasies (e.g. requiring particular regularisation)
Revised: Current attempts at training are either slow (as time is sequentially simulated) or suffer from training instabilities (e.g. the dead neuron problem) and idiosyncrasies (e.g. requiring particular regularisation)

Original: (Eshraghian et al., 2021).
Revised: (Eshraghian et al., 2021).

Original: Additionally, it has been argued that single-spike networks have limited applicability and are not suited for temporal problems, as recently pointed out by Eshraghian et al.
Revised: Additionally, it has been argued that single-spike networks have limited applicability and are not suited for temporal problems, as recently pointed out by Eshraghian et al.

Original: (2021): "[...] it enforces stringent priors upon the network (e.g., each neuron must ﬁre only once) that are incompatible with dynamically changing input data" and Zenke et al.
Revised: (2021): "[...] it enforces stringent priors upon the network (e.g., each neuron must ﬁre only once) that are incompatible with dynamically changing input data" and Zenke et al.

Original: (2021): "[...] only using single spikes in each neuron has its limits and is less suitable for processing temporal stimuli, such as electroencephalogram (EEG) signals, speech, or videos".
Revised: (2021): "[...] only using single spikes in each neuron has its limits and is less suitable for processing temporal stimuli, such as electroencephalogram (EEG) signals, speech, or videos".

Original: In this work we address these shortcomings by proposing a new model for training single-spike networks, for which the main contributions are summarised as follows.
Revised: In this work we address these shortcomings by proposing a new model for training single-spike networks, for which the main contributions are summarised as follows

Original: Our model for training single-spike SNNs eschews all sequential dependence on time and exclusively relies on GPU parallelisable non-sequential operations.
Revised: Our model for training single-spike SNNs eschews all sequential dependence on time and exclusively relies on GPU parallelisable non-sequential operations.

Original: We experimentally validate this to obtain faster training times over sequentially trained control models on synthetic benchmarks (up to 16.77 × speedup) and real datasets (up to 13.98 × speedup).
Revised: We experimentally validate this to obtain faster training times over sequentially trained control models on synthetic benchmarks (up to 16.77 × speedup) and real datasets (up to 13.98 × speedup).

Original: We obtain competitive accuracies on various image and neuromorphic datasets with extreme spike sparsity (up to 81% fewer spikes than standard multi-spike SNNs), with our model being insensitive to the dead neuron problem and not requiring careful network regularisation.
Revised: We obtain competitive accuracies on various image and neuromorphic datasets with extreme spike sparsity (up to 81% fewer spikes than standard multi-spike SNNs), with our model being insensitive to the dead neuron problem and not requiring careful network regularisation.

Original: In other single-spike training methods, but not in our model, the dead neuron problem tends to halt learning due to reduced network activity.
Revised: In other single-spike training methods, but not in our model, the dead neuron problem tends to halt learning due to reduced network activity.

Original: We showcase our model’s applicability in deeper and convolutional networks, and through the inclusion of trainable membrane time constants manage to solve difﬁcult temporal problems otherwise thought to be unsolvable by single-spike networks.
Revised: We showcase our model’s applicability in deeper and convolutional networks, and through the inclusion of trainable membrane time constants manage to solve difﬁcult temporal problems otherwise thought to be unsolvable by single-spike networks.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: 2.1 S INGLE - SPIKE MODEL
Revised: 2.1 S INGLE - SPIKE MODEL

Original: A spiking neural network (SNN) consists of artiﬁcial neurons which output binary signals known as spikes (Figure 1a).
Revised: A spiking neural network (SNN) consists of artiﬁcial neurons which output binary signals known as spikes (Figure 1a).

Original: Assume a feedforward network architecture of L fully connected layers, where each layer l consists of N ( l ) neurons that are fully connected to the next layer l + 1 via synaptic weights W ( l + 1) ∈ R N
Revised: Assume a feedforward network architecture of L fully connected layers, where each layer l consists of N ( l ) neurons that are fully connected to the next layer l + 1 via synaptic weights W ( l + 1) ∈ R N

Original: ( l + 1) × N ( l ) .
Revised: ( l + 1) × N ( l ) .

Original: Neuron i in layer l emits a spike S ( l ) i [ t ] ∈ {0,1} at time t if its membrane potential V ( l ) i [ t ] ∈ R reaches ﬁring threshold
Revised: Neuron i in layer l emits a spike S ( l ) i [ t ] ∈ {0,1} at time t if its membrane potential V ( l ) i [ t ] ∈ R reaches ﬁring threshold

Original: V th .
Revised: V th .

Original: [Equation]
Revised: [Equation]

Original: Membrane potentials evolve according to the leaky integrate and ﬁre (LIF) model
Revised: Membrane potentials evolve according to the leaky integrate and ﬁre (LIF) model

Original: [Equation]
Revised: [Equation]

Original: where τ ∈ R is the membrane time constant and R ∈ R is the input resistance (Gerstner et al., 2014).
Revised: where τ ∈ R is the membrane time constant and R ∈ R is the input resistance (Gerstner et al., 2014).

Original: 1 Without loss of generality the LIF model is normalised ( V ( l ) i ( t ) ∈ [0,1] by V rest = 0, V th = 1, R = 1; see Appendix) and discretised using the forward Euler method (see Appendix), from which the membrane potential can be computed at every discrete simulation time step t ∈ {1,..., T } for T ∈ N using the difference equation below.
Revised: 1 Without loss of generality the LIF model is normalised ( V ( l ) i ( t ) ∈ [0,1] by V rest = 0, V th = 1, R = 1; see Appendix) and discretised using the forward Euler method (see Appendix), from which the membrane potential can be computed at every discrete simulation time step t ∈ {1,..., T } for T ∈ N using the difference equation below.

Original: [Equation]
Revised: [Equation]

Original: The membrane potential is charged from the current induced by the incoming presynaptic spikes
Revised: The membrane potential is charged from the current induced by the incoming presynaptic spikes

Original: S ( l − 1) [ t ] ∈ R N ( l − 1) and from the constant bias current source b ( l ) i .
Revised: S ( l − 1) [ t ] ∈ R N ( l − 1) and from the constant bias current source b ( l ) i .

Original: Over time, this potential dissipates, and the degree of dissipation is captured by 0 ≤ β = exp( − ∆ t τ ) ≤ 1 (for simulation time-step size ∆ t ∈ R ).
Revised: Over time, this potential dissipates, and the degree of dissipation is captured by 0 ≤ β = exp( − ∆ t τ ) ≤ 1 (for simulation time-step size ∆ t ∈ R ).

Original: The neuron’s membrane potential is at resting state V rest = 0 in the absence of any input current and emits a spike S ( l ) i [ t ] = 1 if the potential rises above ﬁring threshold
Revised: The neuron’s membrane potential is at resting state V rest = 0 in the absence of any input current and emits a spike S ( l ) i [ t ] = 1 if the potential rises above ﬁring threshold

Original: V th = 1 (after which it is reduced back close to resting state).
Revised: V th = 1 (after which it is reduced back close to resting state).

Original: To enforce the single-spike constraint, we keep track if a neuron has spiked prior to time t using the variable d ( l ) i [ t ] = max( S ( l ) i
Revised: To enforce the single-spike constraint, we keep track if a neuron has spiked prior to time t using the variable d ( l ) i [ t ] = max( S ( l ) i

Original: [ t − 1], d ( l ) i
Revised: [ t − 1], d ( l ) i

Original: [ t − 1]), which is zero before the ﬁrst spike and one thereafter ( d ( l ) i
Revised: [ t − 1]), which is zero before the ﬁrst spike and one thereafter ( d ( l ) i

Original: t = 0] = 0).
Revised: t = 0] = 0).

Original: We then redeﬁne the output spikes as ˜ S ( l ) i [ t ] = (1 − d ( l ) i [ t ]) · S ( l ) i [ t ], thus ensuring that no more than a single spike is emitted during simulation (Figure 1b).
Revised: We then redeﬁne the output spikes as ˜ S ( l ) i [ t ] = (1 − d ( l ) i [ t ]) · S ( l ) i [ t ], thus ensuring that no more than a single spike is emitted during simulation (Figure 1b).

Original: 2.2 S INGLE - SPIKE TRAINING TECHNIQUES
Revised: 2.2 S INGLE - SPIKE TRAINING TECHNIQUES

Original: The main problem with training single- and multi-spike SNNs is the non-differentiable nature of their activation function.
Revised: The main problem with training single- and multi-spike SNNs is the non-differentiable nature of their activation function.

Original: This precludes the direct use of the backprop algorithm
Revised: This precludes the direct use of the backprop algorithm

Original: (Rumelhart et al., 1986), which has underpinned the successful training of ANNs.
Revised: (Rumelhart et al., 1986), which has underpinned the successful training of ANNs.

Original: Various SNN training solutions have been proposed, which we group into three categories.
Revised: Various SNN training solutions have been proposed, which we group into three categories.

Original: Shadow training Instead of directly training a SNN, an already trained ANN is mapped to a
Revised: Shadow training Instead of directly training a SNN, an already trained ANN is mapped to a

Original: SNN.
Revised: SNN.

Original: This approach has actively been explored in the multi-spike setting (O’Connor et al., 2013;
Revised: This approach has actively been explored in the multi-spike setting (O’Connor et al., 2013;

Original: Esser et al., 2015; Rueckauer et al., 2016; 2017), with recent work extending this to single-spike networks (Stöckl & Maass, 2019; Park et al., 2020).
Revised: Esser et al., 2015; Rueckauer et al., 2016; 2017), with recent work extending this to single-spike networks (Stöckl & Maass, 2019; Park et al., 2020).

Original: Although these approaches permit the training of large networks, they come with various shortcomings.
Revised: Although these approaches permit the training of large networks, they come with various shortcomings.

Original: Some shortcomings are method speciﬁc, such as Stöckl & Maass (2019) who outline how a single ANN unit can be represented as a network of spiking units.
Revised: Some shortcomings are method speciﬁc, such as Stöckl & Maass (2019) who outline how a single ANN unit can be represented as a network of spiking units.

Original: However, this leads to an undesirable blowup of network parameters in their conversion process (which is avoided by our approach).
Revised: However, this leads to an undesirable blowup of network parameters in their conversion process (which is avoided by our approach).

Original: Other shortcomings are more general, such as the lack of support for training neural parameters besides synaptic weights (which our approach permits) or inference accuracy being lost in the conversion process, where mapped
Revised: Other shortcomings are more general, such as the lack of support for training neural parameters besides synaptic weights (which our approach permits) or inference accuracy being lost in the conversion process, where mapped

Original: SNNs perform worse than the original ANNs (which we avoid).
Revised: SNNs perform worse than the original ANNs (which we avoid).

Original: Training using the spike times An approach used to directly train SNNs using backprop involves passing gradients through the time of spiking, which sidesteps the aforementioned nondifferentiability issue (Bohte et al., 2002; Mostafa, 2017; Comsa et al., 2020; Kheradpisheh &
Revised: Training using the spike times An approach used to directly train SNNs using backprop involves passing gradients through the time of spiking, which sidesteps the aforementioned nondifferentiability issue (Bohte et al., 2002; Mostafa, 2017; Comsa et al., 2020; Kheradpisheh &

Original: Masquelier, 2020; Zhang et al., 2021; Zhou & Li, 2021; Zhou et al., 2021).
Revised: Masquelier, 2020; Zhang et al., 2021; Zhou & Li, 2021; Zhou et al., 2021).

Original: Although commonly used for training single-spike SNNs, this approach suffers from various shortcomings, such as 1.
Revised: Although commonly used for training single-spike SNNs, this approach suffers from various shortcomings, such as 1.

Original: the dead neuron problem, where a lack of spiking activity halts the learning process (which we overcome), 2.
Revised: the dead neuron problem, where a lack of spiking activity halts the learning process (which we overcome), 2.

Original: being usually constrained to integrate and ﬁre (IF) neurons (where we support both the IF and LIF model), 3.
Revised: being usually constrained to integrate and ﬁre (IF) neurons (where we support both the IF and LIF model), 3.

Original: having performance dependent on the computationally costly processing of presynaptic spikes using postsynaptic potential (PSP) kernels (which we show not to be necessary) and 4.
Revised: having performance dependent on the computationally costly processing of presynaptic spikes using postsynaptic potential (PSP) kernels (which we show not to be necessary) and 4.

Original: requiring careful network regularisation (which we avoid).
Revised: requiring careful network regularisation (which we avoid).

Original: Training using the membrane potentials Another approach to directly training SNNs using backprop is by replacing the undeﬁned gradient of the non-differentiable spike function with a surrogate gradient (Esser et al., 2016; Hunsberger & Eliasmith, 2015; Zenke & Ganguli, 2018;
Revised: Training using the membrane potentials Another approach to directly training SNNs using backprop is by replacing the undeﬁned gradient of the non-differentiable spike function with a surrogate gradient (Esser et al., 2016; Hunsberger & Eliasmith, 2015; Zenke & Ganguli, 2018;

Original: Lee et al., 2016), which permits the ﬂow of gradient through every membrane potential in time (Bellec et al., 2018; Shrestha & Orchard, 2018; Neftci et al., 2019).
Revised: Lee et al., 2016), which permits the ﬂow of gradient through every membrane potential in time (Bellec et al., 2018; Shrestha & Orchard, 2018; Neftci et al., 2019).

Original: This method has been shown to circumvent the dead neuron problem and permit the training of other neural parameters besides synaptic connectivity (such as membrane time constants) that have been shown to improve network performance (Perez-Nieves et al., 2021).
Revised: This method has been shown to circumvent the dead neuron problem and permit the training of other neural parameters besides synaptic connectivity (such as membrane time constants) that have been shown to improve network performance (Perez-Nieves et al., 2021).

Original: However, these results have not been replicated in the single-spike setting (which we do).
Revised: However, these results have not been replicated in the single-spike setting (which we do).

Original: A shortcoming of this method is its slow training speed, as the network needs to sequentially be simulated at every point in time (which we overcome).
Revised: A shortcoming of this method is its slow training speed, as the network needs to sequentially be simulated at every point in time (which we overcome).

Original:  P ROPOSED TRAINING SPEEDUP ALGORITHM
Revised:  P ROPOSED TRAINING SPEEDUP ALGORITHM

Original: S (0) [1]
Revised: S (0) [1]

Original: S (0) [2] S (0) [3]
Revised: S (0) [2] S (0) [3]

Original: S (0) [4]
Revised: S (0) [4]

Original: I (1) [1] I (1) [2]
Revised: I (1) [1] I (1) [2]

Original: I (1) [3]
Revised: I (1) [3]

Original: I (1) [4]
Revised: I (1) [4]

Original: V (1) [1]
Revised: V (1) [1]

Original: S (1) [1] S (1) [2]
Revised: S (1) [1] S (1) [2]

Original: S (1) [3]
Revised: S (1) [3]

Original: S (1) [4] z (1) [1] z (1) [2] z (1) [3] z (1) [4] ! "
Revised: S (1) [4] z (1) [1] z (1) [2] z (1) [3] z (1) [4] ! "

Original: V (1) [2]
Revised: V (1) [2]

Original: V (1) [3]
Revised: V (1) [3]

Original: V (1) [4]
Revised: V (1) [4]

Original: S (1) [1] S (1) [2]
Revised: S (1) [1] S (1) [2]

Original: S (1) [3]
Revised: S (1) [3]

Original: S (1) [4]
Revised: S (1) [4]

Original: We propose a new model for training SNNs in which individual neurons spike at most once.
Revised: We propose a new model for training SNNs in which individual neurons spike at most once.

Original: Our solution overcomes the slow training speeds of prior training algorithms by eschewing all sequential dependence and recasting the standard single-spike model to exclusively rely on parallelisable non-sequential operations.
Revised: Our solution overcomes the slow training speeds of prior training algorithms by eschewing all sequential dependence and recasting the standard single-spike model to exclusively rely on nonsequential operations.

Original: This recast model provides the same transformation from input to output spikes as the standard single-spike model, but is substantially faster to train.
Revised: Although our model performs more calculations than the standard singlespike model, all these calculations are highly parallelisable and thus substantially faster to train (see Appendix).

Original: Our model is comprised of three main steps which are readily implementable in modern auto differentiation frameworks (Abadi et al., 2016;
Revised: Our model is comprised of three main steps which are readily implementable in modern auto differentiation frameworks (Abadi et al., 2016;

Original: Paszke et al., 2017; Bradbury et al., 2018).
Revised: Paszke et al., 2017; Bradbury et al., 2018).

Original: For illustration purposes, we provide a diagram of the model’s computational graph (Figure 2a) and an example of how input spikes are transformed throughout the model’s different layers of processing (Figure 2b).
Revised: For illustration purposes, we provide a diagram of the model’s computational graph (Figure 2a) and an example of how input spikes are transformed throughout the model’s different layers of processing (Figure 2b).

Original: Convert presynaptic spikes to input current As in the standard model, we map the time series of presynaptic spikes S ( l − 1)
Revised: Convert presynaptic spikes to input current As in the standard model, we map the time series of presynaptic spikes S ( l − 1)

Original: j 2 to a time series of input currents I ( l ) i , which is achieved using a tensor multiplication.
Revised: j 2 to a time series of input currents I ( l ) i , which is achieved using a tensor multiplication.

Original: [Equation]
Revised: [Equation]

Original: Calculate membrane potentials without reset In contrast to the standard model, we calculate modiﬁed membrane potentials ˜ V ( l ) i from the input current I ( l ) i by excluding the reset mechanism.
Revised: Calculate membrane potentials without reset In contrast to the standard model, we calculate modiﬁed membrane potentials ˜ V ( l ) i from the input current I ( l ) i by excluding the reset mechanism.

Original: By dropping the reset term − S ( l ) i [ t ] in Equation 3 and unrolling this altered equation (see Appendix), we obtain a convolutional form allowing us to calculate these no-reset membrane potentials ˜ V ( l ) i without any sequential operations (where β = [ β 0 , β 1 , ··· , β T − 1 ]).
Revised: By dropping the reset term − S ( l ) i [ t ] in Equation 3 and unrolling this altered equation (see Appendix), we obtain a convolutional form allowing us to calculate these no-reset membrane potentials ˜ V ( l ) i without any sequential operations (where β = [ β 0 , β 1 , ··· , β T − 1 ]).

Original: [Equation]
Revised: [Equation]

Original: Map no-reset membrane potentials to output spikes We map the time series of no-reset membrane potentials ˜ V ( l ) i to output spikes S ( l ) i (which contains at most one spike).
Revised: Map no-reset membrane potentials to output spikes We map the time series of no-reset membrane potentials ˜ V ( l ) i to output spikes S ( l ) i (which contains at most one spike).

Original: We obtain a
Revised: We obtain a

Original: time series of erroneous output spikes ˜ S
Revised: time series of erroneous output spikes ˜ S

Original: ( l ) i by passing no-reset membrane potentials ˜ V ( l ) i through the spike function f (Equation 1)
Revised: ( l ) i by passing no-reset membrane potentials ˜ V ( l ) i through the spike function f (Equation 1)

Original: [Equation]
Revised: [Equation]

Original: Due to the removal of the spike reset mechanism, only the ﬁrst spike occurrence in ˜ S ( l ) i follows the dynamics set out by the LIF model (Equation 3) and thus all spikes succeeding the ﬁrst spike occurrence are removed (compliant with the single spike assumption).
Revised: Due to the removal of the spike reset mechanism, only the ﬁrst spike occurrence in ˜ S ( l ) i follows the dynamics set out by the LIF model (Equation 3) and thus all spikes succeeding the ﬁrst spike occurrence are removed (compliant with the single spike assumption).

Original: We achieve this by constructing correct output spikes S ( l ) i with S ( l ) i [ t ] = 0 for t ∈ {1,2,..., T } except S ( l ) i [ t ] = 1 for the smallest t satisfying ˜ V ( l ) i
Revised: We achieve this by constructing correct output spikes S ( l ) i with S ( l ) i [ t ] = 0 for t ∈ {1,2,..., T } except S ( l ) i [ t ] = 1 for the smallest t satisfying ˜ V ( l ) i

Original: [ t ] > 1 (if such ˜ V ( l ) i [ t ] exists).
Revised: [ t ] > 1 (if such ˜ V ( l ) i [ t ] exists).

Original: A straightforward solution would be to iterate over all elements in ˜ S ( l ) i and set all spikes succeeding the ﬁrst to zero, but such sequential calculation is the very problem we set out to remediate.
Revised: A straightforward solution would be to iterate over all elements in ˜ S ( l ) i and set all spikes succeeding the ﬁrst to zero, but such sequential calculation is the very problem we set out to remediate.

Original: We propose a vectorised solution to this problem which is comprised of two steps:
Revised: We propose a vectorised solution to this problem which is comprised of two steps:

Original: Map the erroneous output spikes ˜ S
Revised: Map the erroneous output spikes ˜ S

Original: ( l ) i to a latent representation z ( l )
Revised: ( l ) i to a latent representation z ( l )

Original: i =
Revised: i =

Original: φ (˜ S ( l ) i ), where every element therein encodes an ordering of the spikes.
Revised: φ (˜ S ( l ) i ), where every element therein encodes an ordering of the spikes.

Original: This is achieved by passing the erroneous output spikes ˜ S ( l ) i through proposed function φ (Proposition 1), which maps all elements besides the ﬁrst spike occurrence to a value other than one ( z ( l ) i [ t ] ∕ = 1 for all t except for the smallest t satisfying ˜ S
Revised: This is achieved by passing the erroneous output spikes ˜ S ( l ) i through proposed function φ (Proposition 1), which maps all elements besides the ﬁrst spike occurrence to a value other than one ( z ( l ) i [ t ] ∕ = 1 for all t except for the smallest t satisfying ˜ S

Original: ( l ) i
Revised: ( l ) i

Original: [ t ] = 1 if such t exists).
Revised: [ t ] = 1 if such t exists).

Original: Obtain the correct output spikes S ( l ) i
Revised: Obtain the correct output spikes S ( l ) i

Original: = g ( z ( l ) i ) by passing the latent representation z
Revised: = g ( z ( l ) i ) by passing the latent representation z

Original: ( l ) i through function g , which uses the encoded spike ordering to produce the correct outputs spikes S ( l ) i by mapping every value besides one to zero.
Revised: ( l ) i through function g , which uses the encoded spike ordering to produce the correct outputs spikes S ( l ) i by mapping every value besides one to zero.

Original: [Equation]
Revised: [Equation]

Original: otherwise (
Revised: otherwise (

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Function φ (˜ S ( l ) i )
Revised: Function φ (˜ S ( l ) i )

Original: [ t ] = + tk = [Equation]
Revised: [ t ] = + tk = 1 ˜ S ( l ) i

Original: ˜ S ( l ) i [ k ]( t − k + 1) acting on ˜ S ( l ) i ∈ {0,1} T contains at most one element equal to one φ (˜ S ( l ) i )
Revised: [ k ]( t − k + 1) acting on ˜ S ( l ) i ∈ {0,1} T contains at most one element equal to one φ (˜ S ( l ) i )

Original: [ t ] = 1 for the smallest t satisfying ˜ S ( l ) i
Revised: [ t ] = 1 for the smallest t satisfying ˜ S ( l ) i

Original: [ t ] = 1 (if such t exists).
Revised: [ t ] = 1 (if such t exists).

Original: Proof.
Revised: Proof.

Original: Firstly, if ˜ S ( l ) i [ t ] = 0 for all t ∈ [1, T ] then φ (˜ S ( l ) i )[ t ] = 0 for all t ∈ [1, T ] (follows from substitution).
Revised: Firstly, if ˜ S ( l ) i [ t ] = 0 for all t ∈ [1, T ] then φ (˜ S ( l ) i )[ t ] = 0 for all t ∈ [1, T ] (follows from substitution).

Original: Secondly, if ˜ S ( l ) i [ t 1 ] = 1 for smallest t 1 ∈ [1, T ] then φ (˜ S ( l ) i )
Revised: Secondly, if ˜ S ( l ) i [ t 1 ] = 1 for smallest t 1 ∈ [1, T ] then φ (˜ S ( l ) i )

Original: [ t 1 ] = 1 (follows from substitution) and there can exist no t 2 > t 1 such that φ (˜ S ( l ) i )
Revised: [ t 1 ] = 1 (follows from substitution) and there can exist no t 2 > t 1 such that φ (˜ S ( l ) i )

Original: [ t 2 ] = 1 as
Revised: [ t 2 ] = 1 as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus φ (˜ S ( l ) i )[ t 2 ] > φ (˜ S
Revised: Thus φ (˜ S ( l ) i )[ t 2 ] > φ (˜ S

Original: ( l ) i )
Revised: ( l ) i )

Original: [ t 1 ] for all t 2 > t 1 as
Revised: [ t 1 ] for all t 2 > t 1 as

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS AND RESULTS
Revised:  E XPERIMENTS AND RESULTS

Original: We investigate our model’s speedup advantages and performance on real datasets in comparison to prior work.
Revised: We investigate our model’s speedup advantages and performance on real datasets in comparison to prior work.

Original: All models were implemented using PyTorch (Paszke et al., 2017) with benchmarks and training conducted on a cluster of NVIDIA Tesla A100 GPUs.
Revised: All models were implemented using PyTorch (Paszke et al., 2017) with benchmarks and training conducted on a cluster of NVIDIA Tesla A100 GPUs.

Original: 4.1 S PEEDUP BENCHMARKS
Revised: 4.1 S PEEDUP BENCHMARKS

Original: We evaluate the speedup advantages of our model over the standard single-spike model trained using surrogate gradients, by simulating the forward and backward passes for different numbers of hidden units, layers, simulation steps and batch sizes on a synthetic spike dataset (see Appendix).
Revised: We evaluate the speedup advantages of our model over the standard single-spike model trained using surrogate gradients, by simulating the forward and backward passes for different numbers of hidden units, layers, simulation steps and batch sizes on a synthetic spike dataset (see Appendix).

Original: Robust speedup for different numbers of hidden units and simulation steps We observe a considerable training speedup across a range of hidden units and simulation steps in a single layer (Figure 3a).
Revised: Robust speedup for different numbers of hidden units and simulation steps We observe a considerable training speedup across a range of hidden units and simulation steps in a single layer (Figure 3a).

Original: We obtain an optimal speedup of 16.77 × for n = 100 units and t = 2 7 time steps, where our model takes 4.34 ± 0.9ms compared to the 72.82 ± 2.7ms it takes the standard model to complete a training pass (Figure 3b).
Revised: We obtain an optimal speedup of 16.77 × for n = 100 units and t = 2 7 time steps, where our model takes 4.34 ± 0.9ms compared to the 72.82 ± 2.7ms it takes the standard model to complete a training pass (Figure 3b).

Original: Our model still obtains a reasonable speedup of 3.40 × for largest benchmarked n = 1000 units and t = 2 11 time steps (albeit the forward pass speedup being slower).
Revised: Our model still obtains a reasonable speedup of 3.40 × for largest benchmarked n = 1000 units and t = 2 11 time steps (albeit the forward pass speedup being slower).

Original: These speedups are even more pronounced when the membrane time constants are ﬁxed (obtaining a maximum speedup of 17.42 × ) or when using smaller batch sizes (with batch sizes b
Revised: These speedups are even more pronounced when the membrane time constants are ﬁxed (obtaining a maximum speedup of 17.42 × ) or when using smaller batch sizes (with batch sizes b

Original: = 32 and b = 64 obtaining a maximum speedup of 35.05 × and 25.16 × , respectively; See Appendix).
Revised: = 32 and b = 64 obtaining a maximum speedup of 35.05 × and 25.16 × , respectively; See Appendix).

Original: Applicability to deeper networks We ﬁnd our model to obtain substantial training speedups when using multiple layers (Figure 3c) and layers containing thousands of neurons (Figure 3d).
Revised: Applicability to deeper networks We ﬁnd our model to obtain substantial training speedups when using multiple layers (Figure 3c) and layers containing thousands of neurons (Figure 3d).

Original: The training speedups remain similar across an increasing numbers of layers for different number of hidden units (Figure 3c).
Revised: The training speedups remain similar across an increasing numbers of layers for different number of hidden units (Figure 3c).

Original: Furthermore, we obtain a speedup of ∼ 3 × when using a large number of neurons (ranging between 2 · 10 3 to 2 · 10 4 neurons) in a single layer (Figure 3d).
Revised: Furthermore, we obtain a speedup of ∼ 3 × when using a large number of neurons (ranging between 2 · 10 3 to 2 · 10 4 neurons) in a single layer (Figure 3d).

Original: Inter- estingly, these speedups remain approximately the same across the different number of neurons, even when the batch size is changed.
Revised: Inter- estingly, these speedups remain approximately the same across the different number of neurons, even when the batch size is changed.

Original: Speedup advantages and room for improvement
Revised: Speedup advantages and room for improvement

Original: Previous attempts at accelerating SNN training either speed up the backward pass (Perez-Nieves & Goodman, 2021) or remove it completely (Bellec et al., 2020).
Revised: Previous attempts at accelerating SNN training either speed up the backward pass (Perez-Nieves & Goodman, 2021) or remove it completely (Bellec et al., 2020).

Original: These methods however still sequentially compute the forward pass, which our model is able to accelerate (Figure 3e).
Revised: These methods however still sequentially compute the forward pass, which our model is able to accelerate (Figure 3e).

Original: Furthermore, we observe the backward pass to slow down relative to the forward pass for increasing time steps (Figure 3f).
Revised: Furthermore, we observe the backward pass to slow down relative to the forward pass for increasing time steps (Figure 3f).

Original: Further training speedup may therefore be achieved using sparse gradient descent, as auto differentiation frameworks are not optimised for the sparse nature of SNNs (Perez-Nieves & Goodman, 2021).
Revised: Further training speedup may therefore be achieved using sparse gradient descent, as auto differentiation frameworks are not optimised for the sparse nature of SNNs (Perez-Nieves & Goodman, 2021).

Original: 4.2 P ERFORMANCE ON REAL DATASETS
Revised: 4.2 P ERFORMANCE ON REAL DATASETS

Original: We investigate the applicability of our model to classify real data from different domains and of varying complexity (Table 1).
Revised: We investigate the applicability of our model to classify real data from different domains and of varying complexity (Table 1).

Original: These include the Yin-Yang dataset (Kriener et al., 2022) in which the goal is to classify spatial coordinates belonging to different groups, and the MNIST (LeCun, 1998) and Fashion-MNIST (F-MNIST)
Revised: These include the Yin-Yang dataset (Kriener et al., 2022) in which the goal is to classify spatial coordinates belonging to different groups, and the MNIST (LeCun, 1998) and Fashion-MNIST (F-MNIST)

Original: (Xiao et al., 2017) image datasets, where the objective is to classify images of handwritten digits and fashion items.
Revised: (Xiao et al., 2017) image datasets, where the objective is to classify images of handwritten digits and fashion items.

Original: All these analog datasets were converted into a spike representation using the time-to-ﬁrst-spike encoding (see Appendix).
Revised: All these analog datasets were converted into a spike representation using the time-to-ﬁrst-spike encoding (see Appendix).

Original: We also test performance on two neuromorphic datasets, being the vision N-MNIST (Orchard et al., 2015) and the more difﬁcult auditory SHD dataset (Cramer et al., 2020).
Revised: We also test performance on two neuromorphic datasets, being the vision N-MNIST (Orchard et al., 2015) and the more difﬁcult auditory SHD dataset (Cramer et al., 2020).

Original: The N-MNIST dataset is the MNIST dataset mapped onto a spike code using a neuromorphic vision sensor and the SHD dataset comprises spoken digit waveforms converted into spikes using a model of auditory bushy neurons in the cochlear nucleus.
Revised: The N-MNIST dataset is the MNIST dataset mapped onto a spike code using a neuromorphic vision sensor and the SHD dataset comprises spoken digit waveforms converted into spikes using a model of auditory bushy neurons in the cochlear nucleus.

Original: Obtaining competitive results across different image and neuromorphic datasets The results of our model across all datasets are comparable or superior to prior reported results using singlespike SNNs.
Revised: Obtaining competitive results across different image and neuromorphic datasets The results of our model across all datasets are comparable or superior to prior reported results using singlespike SNNs.

Original: We reach an accuracy of 98.02%, 97.91% and 89.05% using a single hidden layer network on the Yin-Yang, MNIST and F-MNIST datasets respectively, where best performing prior work reported an accuracy of 95.90%, 98.50% and 88.1% respectively.
Revised: We reach an accuracy of 98.02%, 97.91% and 89.05% using a single hidden layer network on the Yin-Yang, MNIST and F-MNIST datasets respectively, where best performing prior work reported an accuracy of 95.90%, 98.50% and 88.1% respectively.

Original: Furthermore, our singlespike model nearly obtains the same accuracies to those obtained in the standard multi-spike
Revised: Furthermore, our singlespike model nearly obtains the same accuracies to those obtained in the standard multi-spike

Original: SNN on these datasets (Yin-Yang and MNIST < 0.3% difference; F-MNIST ∼ 2% difference; Figure 4a).
Revised: SNN on these datasets (Yin-Yang and MNIST < 0.3% difference; F-MNIST ∼ 2% difference; Figure 4a).

Original: [Table]
Revised: [Table]

Original: Single-spike neurons solve challenging temporal problems using neural heterogeneity It has been noted that single-spike SNNs are well suited for static datasets (such as spike encoded images) and less suited for processing temporally complex stimuli (such as audio or video) due to the single-spike constraint (Zenke et al., 2021; Eshraghian et al., 2021).
Revised: Single-spike neurons solve challenging temporal problems using neural heterogeneity It has been noted that single-spike SNNs are well suited for static datasets (such as spike encoded images) and less suited for processing temporally complex stimuli (such as audio or video) due to the single-spike constraint (Zenke et al., 2021; Eshraghian et al., 2021).

Original: Prior single-spike SNN training techniques have attempted to optimise network connectivity without learning other neural parameters, such as membrane time-constants, which have shown to improve performance in multi-spike SNNs (Perez-Nieves et al., 2021).
Revised: Prior single-spike SNN training techniques have attempted to optimise network connectivity without learning other neural parameters, such as membrane time-constants, which have shown to improve performance in multi-spike SNNs (Perez-Nieves et al., 2021).

Original: We explored the effect of learning the membrane time-constants in our single-spike model.
Revised: We explored the effect of learning the membrane time-constants in our single-spike model.

Original: We obtained an accuracy of 44.50% using a network trained with ﬁxed time constants on the temporally-complex auditory SHD dataset.
Revised: We obtained an accuracy of 44.50% using a network trained with ﬁxed time constants on the temporally-complex auditory SHD dataset.

Original: However, by including learnable time constants we were able to obtain a much higher accuracy of 70.32%, which is similar to the performance obtained by a standard SNN with trainable time constants 70.81% or recurrent connections 71.40%.
Revised: However, by including learnable time constants we were able to obtain a much higher accuracy of 70.32%, which is similar to the performance obtained by a standard SNN with trainable time constants 70.81% or recurrent connections 71.40%.

Original: Drastic speedup in training We obtain over a four-fold training speedup across all datasets, with a maximum speedup of 13.98 × on the Yin-Yang dataset (Figure 4b). 
Revised: Drastic speedup in training We obtain over a four-fold training speedup across all datasets over the standard single-spike SNN, with a maximum speedup of 13.98 × on the Yin-Yang dataset (Figure 4b). We observe similar training speedups over the multi-spike SNN (see Appendix).

Original: Differences in speedups are due to the different temporal lengths and input dimensions of the datasets, as well the different network architectures employed (see section 4.1).
Revised: Differences in speedups are due to the different temporal lengths and input dimensions of the datasets, as well the different network architectures employed (see section 4.1).

Original: Increased spike sparsity Our single-spike SNN is able to solve various datasets with a large reduction in spikes compared to a standard multi-spike SNN (Figure 4c), with over a 44% and up to a 81% reduction in spikes.
Revised: Increased spike sparsity Our single-spike SNN is able to solve various datasets with a large reduction in spikes compared to a standard multi-spike SNN (Figure 4c), with over a 44% and up to a 81% reduction in spikes.

Original: This corroborates the value of obtaining more energy-efﬁcient computations using single- rather than multi-spike neuromorphic systems (Liang et al., 2021;
Revised: This corroborates the value of obtaining more energy-efﬁcient computations using single- rather than multi-spike neuromorphic systems (Liang et al., 2021;

Original: Oh et al., 2021; Zhou et al., 2021), as energy consumption scales approximately proportional to the number of emitted spikes (Panda et al., 2020).
Revised: Oh et al., 2021; Zhou et al., 2021), as energy consumption scales approximately proportional to the number of emitted spikes (Panda et al., 2020).

Original: Training deeper convolutional architectures We evaluate our model in deeper convolutional architectures, which to date remains largely unexplored in single-spike SNNs (Mirsadeghi et al., 
Revised: Training deeper convolutional architectures We evaluate our model in deeper convolutional architectures, which to date remains largely unexplored in single-spike SNNs (Mirsadeghi et al., 2022).

Original: We trained a multi-layer convolutional network on the MNIST and F-MNIST datasets, obtaining accuracies (MNIST: 99.32% and F-MNIST: 90.57%) similar to best performing prior work (MNIST: 99.4% and F-MNIST: 92.8%), whilst being faster to train in comparison to the control (MNIST-speedup ∼ 1.37 × and F-MNIST-speedup ∼ 1.39 × ).
Revised: We trained a multi-layer convolutional network on the MNIST and F-MNIST datasets, obtaining accuracies (MNIST: 99.32% and F-MNIST: 90.57%) similar to best performing prior work (MNIST: 99.4% and F-MNIST: 92.8%), whilst being faster to train in comparison to the control (MNIST-speedup ∼ 1.37 × and F-MNIST-speedup ∼ 1.39 × ).

Original: Robust learning and bypassing the dead neuron problem A limitation of current single-spike
Revised: Robust learning and bypassing the dead neuron problem A limitation of current single-spike

Original: SNN training methods is the dead neuron problem, referring to the hinderance in learning when neurons do not spike, as the learning signal is dependent on the occurrence thereof (Eshraghian et al., 2021).
Revised: SNN training methods is the dead neuron problem, referring to the hinderance in learning when neurons do not spike, as the learning signal is dependent on the occurrence thereof (Eshraghian et al., 2021).

Original: Our model is able to overcome this problem as we use surrogate gradients for training, in which the learning signal is instead passed through the membrane potentials.
Revised: Our model is able to overcome this problem as we use surrogate gradients for training, in which the learning signal is instead passed through the membrane potentials.

Original: We experimentally veriﬁed this by showing how networks instantiated with zero starting activity (fatal to other single-spike training methods) still manage to solve different datasets (Figure 4d).
Revised: We experimentally veriﬁed this by showing how networks instantiated with zero starting activity (fatal to other single-spike training methods) still manage to solve different datasets (Figure 4d).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: SNNs emulated on neuromorphic hardware are a promising avenue towards addressing the energy and scaling constraints of ANNs (Wunderlich et al., 2019).
Revised: SNNs emulated on neuromorphic hardware are a promising avenue towards addressing the energy and scaling constraints of ANNs (Wunderlich et al., 2019).

Original: Single-spike SNNs further amplify these energy improvements through extreme spike sparsity, as energy consumption scales approximately proportionally to the number of emitted spikes (Panda et al., 2020).
Revised: Single-spike SNNs further amplify these energy improvements through extreme spike sparsity, as energy consumption scales approximately proportionally to the number of emitted spikes (Panda et al., 2020).

Original: To date, SNN training remains challenging due to the non-differentiable nature of the spike function, prohibiting the direct use of the backprop training algorithm which underpins the success of ANNs.
Revised: To date, SNN training remains challenging due to the non-differentiable nature of the spike function, prohibiting the direct use of the backprop training algorithm which underpins the success of ANNs.

Original: Various extensions of backprop for SNNs have been proposed, but fall short in particular aspects.
Revised: Various extensions of backprop for SNNs have been proposed, but fall short in particular aspects.

Original: Gradients can be passed through the timing of spikes (Bohte et al., 2002; Mostafa, 2017;
Revised: Gradients can be passed through the timing of spikes (Bohte et al., 2002; Mostafa, 2017;

Original: Kheradpisheh & Masquelier, 2020), yet this method suffers from the dead neuron problem, requires careful regularisation or imposes computationally-expensive modelling constraints.
Revised: Kheradpisheh & Masquelier, 2020), yet this method suffers from the dead neuron problem, requires careful regularisation or imposes computationally-expensive modelling constraints.

Original: Alternatively, gradients can be passed through the membrane potentials using surrogate gradients (Shrestha & Orchard, 2018; Neftci et al., 2019), and although this method improves upon the problems of passing gradients through the spike times, it is painfully slow.
Revised: Alternatively, gradients can be passed through the membrane potentials using surrogate gradients (Shrestha & Orchard, 2018; Neftci et al., 2019), and although this method improves upon the problems of passing gradients through the spike times, it is painfully slow.

Original: In this work, we address these problems by proposing a new general model (e.g. neurons can be
Revised: In this work, we address these problems by proposing a new general model (e.g. neurons can be

Original: IF or LIF) for training single-spike SNNs, without imposing any modelling (e.g. requiring PSP kernels) or training constraints (e.g. requiring careful regularisation) and support training of neural parameters other than synaptic connectivity (e.g. membrane time constants).
Revised: IF or LIF) for training single-spike SNNs, without imposing any modelling (e.g. requiring PSP kernels) or training constraints (e.g. requiring careful regularisation) and support training of neural parameters other than synaptic connectivity (e.g. membrane time constants).

Original: We mathematically show how training can be sped up by replacing the slow sequential operations with faster convolutional ones.
Revised: We mathematically show how training can be sped up by replacing the slow sequential operations with faster convolutional ones.

Original: We experimentally validate this speedup across various numbers of units, time steps, layers and batch sizes, obtaining up to a 16.77 × speedup.
Revised: We experimentally validate this speedup across various numbers of units, time steps, layers and batch sizes, obtaining up to a 16.77 × speedup.

Original: We show that our model can be trained across different network architectures (e.g. feedforward, hierarchical and convolutional) and obtain competitive results on different image and neuromorphic datasets.
Revised: We show that our model can be trained across different network architectures (e.g. feedforward, hierarchical and convolutional) and obtain competitive results on different image and neuromorphic datasets.

Original: Our results compare well against multi-spike SNNs ( < 2% accuracy difference on all datasets) and obtain up to an 81% reduction in spike counts.
Revised: Our results compare well against multi-spike SNNs ( < 2% accuracy difference on all datasets) and obtain up to an 81% reduction in spike counts.

Original: Furthermore, our method circumvents the dead neuron problem and, for the ﬁrst time, we show how single-spike SNNs can solve temporally-complex datasets on a par with multi-spike SNNs by including trainable membrane time constants.
Revised: Furthermore, our method circumvents the dead neuron problem and, for the ﬁrst time, we show how single-spike SNNs can solve temporally-complex datasets on a par with multi-spike SNNs by including trainable membrane time constants.

Original: Our ﬁndings therefore challenge the dogma that single-spike SNNs are only suited to non-temporal problems (Eshraghian et al., 2021; Zenke et al., 2021).
Revised: Our ﬁndings therefore challenge the dogma that single-spike SNNs are only suited to non-temporal problems (Eshraghian et al., 2021; Zenke et al., 2021).

Original: We obtain training speedups on all datasets, however, ﬁnd that the backward pass slows down relative to the forward pass for longer timespans.
Revised: We obtain training speedups on all datasets, however, ﬁnd that the backward pass slows down relative to the forward pass for longer timespans.

Original: Future work could mitigate this bottleneck and accelerate training using sparse gradient descent, which has shown to accelerate the backward pass in standard SNNs by taking advantage of spike sparsity (Perez-Nieves & Goodman, 2021).
Revised: Future work could mitigate this bottleneck and accelerate training using sparse gradient descent, which has shown to accelerate the backward pass in standard SNNs by taking advantage of spike sparsity (Perez-Nieves & Goodman, 2021).

Original: Currently, our single-spike model performs slightly worse compared to its multi-spike counterpart, where better performance could be achieved by extending our model to the multi-spike setting and permitting recurrent connectivity.
Revised: Currently, our single-spike model performs slightly worse compared to its multi-spike counterpart, where better performance could be achieved by extending our model to the multi-spike setting and permitting recurrent connectivity.

Original: Although we obtain impressive network performance without synaptic dynamics, future investigations could also benchmark the effect of including different PSP kernels on network performance. Finally, it remains an open question how the inclusion of trainable membrane time constants in our single-spike network obtains performance on a par with an equivalent multi-spike network on challenging temporal datasets such as the SHD dataset. Heterogenous time constants permit neurons to integrate information over various time scales (Perez-Nieves et al., 2021), but how such dynamics coupled with extreme spike sparsity solve challenging temporal tasks requires more rigorous theoretical analysis.
Revised: Finally, it remains an open question how the inclusion of trainable membrane time constants in our model boost performance, requiring further theoretical analysis.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: The theoretical construction and derivations of our model are outlined in section 3 and we provide accompanying derivations in the Appendix.
Revised: The theoretical construction and derivations of our model are outlined in section 3 and we provide accompanying derivations in the Appendix.

Original: All code is publicly available at https://github.com/webstorms/FastSNN under the BSD 3-Clause Licence.
Revised: All code is publicly available at https://github.com/webstorms/Block under the BSD 3-Clause Licence.

Original: This includes instructions on installation, data processing and running experiments to reproduce all results and ﬁgures portrayed in the paper.
Revised: This includes instructions on installation, data processing and running experiments to reproduce all results and ﬁgures portrayed in the paper.

Original: Training details are also provided in the Appendix.
Revised: Training details are also provided in the Appendix.

Original: A A PPENDIX A.1 S PIKING NEURAL NETWORK DERIVATIONS A.1.1 N ORMALISING THE LEAKY INTEGRATE AND F IRE MODEL
Revised: A A PPENDIX A.1 S PIKING NEURAL NETWORK DERIVATIONS A.1.1 N ORMALISING THE LEAKY INTEGRATE AND F IRE MODEL

Original: Proposition 2.
Revised: Proposition 2.

Original: Any leaky integrate and ﬁre (LIF) model τ dV ( t ) dt = − V ( t )
Revised: Any leaky integrate and ﬁre (LIF) model τ dV ( t ) dt = − V ( t )

Original: + V rest + RI ( t ) (with membrane potential V , resting potential V rest , ﬁring threshold
Revised: + V rest + RI ( t ) (with membrane potential V , resting potential V rest , ﬁring threshold

Original: V th , resistance R, input current I and membrane time constant τ ) can be normalised to a LIF model of the form τ d ˜ V ( t ) dt = − ˜ V ( t ) + ˜ I ( t ) (such that 0 ≤ ˜ V ( t ) ≤ 1 , with ﬁring threshold ˜
Revised: V th , resistance R, input current I and membrane time constant τ ) can be normalised to a LIF model of the form τ d ˜ V ( t ) dt = − ˜ V ( t ) + ˜ I ( t ) (such that 0 ≤ ˜ V ( t ) ≤ 1 , with ﬁring threshold ˜

Original: V
Revised: V

Original: th = 1 , resting potential ˜ V rest
Revised: th = 1 , resting potential ˜ V rest

Original: = 0 and a resistance equal to one).
Revised: = 0 and a resistance equal to one).

Original: Proof.
Revised: Proof.

Original: This mapping from any LIF model to the normalised LIF model is achieved using the following transformation (taken from Hunsberger (2018)).
Revised: This mapping from any LIF model to the normalised LIF model is achieved using the following transformation (taken from Hunsberger (2018)).

Original: [Equation]
Revised: [Equation]

Original: Rearranging this expression with respect to V ( t ) =
Revised: Rearranging this expression with respect to V ( t ) =

Original: ˜ V ( t )( V th − V rest )
Revised: ˜ V ( t )( V th − V rest )

Original: + V rest and substituting this into the LIF model we obtain
Revised: + V rest and substituting this into the LIF model we obtain

Original: [Equation]
Revised: [Equation]

Original: This new LIF form has a resting potential ˜ V rest = 0 and ﬁring threshold ˜
Revised: This new LIF form has a resting potential ˜ V rest = 0 and ﬁring threshold ˜

Original: V th = 1 (obtained by substituting V ( t ) =
Revised: V th = 1 (obtained by substituting V ( t ) =

Original: V rest and V ( t )
Revised: V rest and V ( t )

Original: =
Revised: =

Original: V th in Equation 9 respectively).
Revised: V th in Equation 9 respectively).

Original: Thus, without loss of generality, any LIF model can be mapped to a normalised form using linear transformation Equation 9.
Revised: Thus, without loss of generality, any LIF model can be mapped to a normalised form using linear transformation Equation 9.

Original: [Equation]
Revised: [Equation]

Original: THE LEAKY INTEGRATE AND F IRE MODEL
Revised: THE LEAKY INTEGRATE AND F IRE MODEL

Original: Proposition 3.
Revised: Proposition 3.

Original: The normalised continuous time leaky integrate and ﬁre model τ dV ( t ) dt
Revised: The normalised continuous time leaky integrate and ﬁre model τ dV ( t ) dt

Original: = − V ( t ) +
Revised: = − V ( t ) +

Original: I ( t ) (with membrane potentialV , input current I and membrane time constant τ ) can numerically be approximated by discrete time difference equation V [ t + 1] =
Revised: I ( t ) (with membrane potentialV , input current I and membrane time constant τ ) can numerically be approximated by discrete time difference equation V [ t + 1] =

Original: β V [ t ] + (1 − β )
Revised: β V [ t ] + (1 − β )

Original: I
Revised: I

Original: [ t + 1] , where β = exp( ∆ t τ ) (for simulation time resolution ∆ t).
Revised: [ t + 1] , where β = exp( ∆ t τ ) (for simulation time resolution ∆ t).

Original: Proof.
Revised: Proof.

Original: We proceed using the forward Euler method.
Revised: We proceed using the forward Euler method.

Original: Let I ( t ) =
Revised: Let I ( t ) =

Original: I be constant with respect to time, for which the ordinary differential equation becomes separable.
Revised: I be constant with respect to time, for which the ordinary differential equation becomes separable.

Original: [Equation]
Revised: [Equation]

Original: For initial solution V ( t 0 ) at time t 0 we derive k
Revised: For initial solution V ( t 0 ) at time t 0 we derive k

Original: =
Revised: =

Original: ( V ( t 0 ) − I )exp( t 0 τ ).
Revised: ( V ( t 0 ) − I )exp( t 0 τ ).

Original: Then for constant I and initial solution V ( t 0 ) we obtain solution.
Revised: Then for constant I and initial solution V ( t 0 ) we obtain solution.

Original: [Equation]
Revised: [Equation]

Original: To obtain the discretised update equation, we deﬁne simulation update time step ∆ t = t − t 0 , decay factor β = exp( − ∆ t τ ), assign continuous time points to discretised time steps t ← t 0 and t + 1 ← t 0 + ∆ t and assume the input current to be approximately constant and equal to I [ t + 1] between discretised update steps t to t + [Equation]
Revised: To obtain the discretised update equation, we deﬁne simulation update time step ∆ t = t − t 0 , decay factor β = exp( − ∆ t τ ), assign continuous time points to discretised time steps t ← t 0 and t + 1 ← t 0 + ∆ t and assume the input current to be approximately constant and equal to I [ t + 1] between discretised update steps t to t + 1.

Original: [Equation]
Revised: [Equation]

Original: A.1.3 U NROLLING THE LEAKY INTEGRATE AND F IRE MODEL WITHOUT THE RESET TERM
Revised: A.1.3 U NROLLING THE LEAKY INTEGRATE AND F IRE MODEL WITHOUT THE RESET TERM

Original: Proposition 4.
Revised: Proposition 4.

Original: Equation V [ t ] =
Revised: Equation V [ t ] =

Original: β t V [0]
Revised: β t V [0]

Original: + (1 − β ) t
Revised: + (1 − β ) t

Original: # i =
Revised: # i =

Original: β t − i
Revised: β t − i

Original: I [ i ] is equivalent to difference equation
Revised: I [ i ] is equivalent to difference equation

Original: V [ t ] =
Revised: V [ t ] =

Original: β V [ t − 1]
Revised: β V [ t − 1]

Original: + (1 − β ) I
Revised: + (1 − β ) I

Original: [ t ] for t ≥ 1 .
Revised: [ t ] for t ≥ 1 .

Original: Proof.
Revised: Proof.

Original: We proceed to proof equivalence by induction.
Revised: We proceed to proof equivalence by induction.

Original: For t = 1 we obtain
Revised: For t = 1 we obtain

Original: [Equation]
Revised: [Equation]

Original: Hence the relation holds true for the base case t = 1.
Revised: Hence the relation holds true for the base case t = 1.

Original: Assume the relation holds true for t = k ≥ 1, then for t
Revised: Assume the relation holds true for t = k ≥ 1, then for t

Original: = k
Revised: = k

Original: + 

Revised: +

Original: we derive
Revised: we derive

Original: [Equation]
Revised: [Equation]

Original: This implies equivalence for t = k
Revised: This implies equivalence for t = k

Original: + 

Revised: +

Original: if t = k holds true.
Revised: if t = k holds true.

Original: By the principle of induction, equivalence is established given that both the base case and inductive step hold true.
Revised: By the principle of induction, equivalence is established given that both the base case and inductive step hold true.

Original: A.2 A DDITIONAL DATASET DETAILS A.2.1 S YNTHETIC SPIKE DATASET FOR THE SPEED BENCHMARKS
Revised: A.2 A DDITIONAL MODEL THEORY : W HY IS OUR MODEL FASTER ?

Revised: To address the question why our model is faster than the standard single-spike model, we analyse their respective computational complexities. Consider a single neuron with N presynaptic neurons simulated for T time steps. Our model has a computational complexity of O ( NT 2 ) and the computational complexity of the standard model is O ( NT ).

Revised: However, the sequential complexity of our model is constant time O (1) (as our model eschews all sequential dependence) and the sequential complexity of the standard model is linear O ( T ).

Revised: Our model performs more calculations than the standard single-spike model, yet is able to obtain faster training speeds, as - unlike the standard single-spike model - all these calculations are highly parallelisable.

Revised: A.3 A DDITIONAL DATASET DETAILS A.3.1 S YNTHETIC SPIKE DATASET FOR THE SPEED BENCHMARKS

Original: We generated binary input spike tensors of shape B × N × T ( B being the batch size, N the number of input neurons and T the number of simulation steps).
Revised: We generated binary input spike tensors of shape B × N × T ( B being the batch size, N the number of input neurons and T the number of simulation steps).

Original: For every batch dimension b a ﬁring rate r b ∼ U ( u min , u max ) was uniformly sampled (with u min = 0Hz and u max = 200Hz), from which a random binary spike matrix of shape N × T was constructed, such that every input neuron in this matrix had an expected ﬁring rate of r b Hz.
Revised: For every batch dimension b a ﬁring rate r b ∼ U ( u min , u max ) was uniformly sampled (with u min = 0Hz and u max = 200Hz), from which a random binary spike matrix of shape N × T was constructed, such that every input neuron in this matrix had an expected ﬁring rate of r b Hz.

Original: A.2.2 T IME - TO - F IRST - SPIKE ENCODING
Revised: A.3.2 T IME - TO - F IRST - SPIKE ENCODING

Original: We encoded all analog non-spiking input data into a spike raster using the time-to-ﬁrst-spike coding method (Kheradpisheh & Masquelier, 2020).
Revised: We encoded all analog non-spiking input data into a spike raster using the time-to-ﬁrst-spike coding method (Kheradpisheh & Masquelier, 2020).

Original: Here, every scalar value I i ∈ {0, I max } within an input tensor is converted into a spike train with a single spike, where the time of spike t i ∈ {0, T } is determined by the following equation
Revised: Here, every scalar value I i ∈ {0, I max } within an input tensor is converted into a spike train with a single spike, where the time of spike t i ∈ {0, T } is determined by the following equation

Original: [Equation]
Revised: [Equation]

Original: A.3 T RAINING DETAILS AND HYPERPARAMETERS A.3.1 R EADOUT NEURONS
Revised: A.4 T RAINING DETAILS AND HYPERPARAMETERS A.4.1 R EADOUT NEURONS

Original: The output layer L of every trained network contained the same number of neurons as the number of classes contained within the dataset being trained on.
Revised: The output layer L of every trained network contained the same number of neurons as the number of classes contained within the dataset being trained on.

Original: As suggested by Zenke & Vogels (2021), every neuron had a ﬁring threshold set to inﬁnity ( i.e. the spiking and reset mechanism was removed) from which the output o b , c of readout neuron c to input sample b was either taken to be the maximum membrane potential over time o b , c = max t V Lb , c
Revised: As suggested by Zenke & Vogels (2021), every neuron had a ﬁring threshold set to inﬁnity ( i.e. the spiking and reset mechanism was removed) from which the output o b , c of readout neuron c to input sample b was either taken to be the maximum membrane potential over time o b , c = max t V Lb , c

Original: [ t ] or the summated membrane potential over time o b , c
Revised: [ t ] or the summated membrane potential over time o b , c

Original: = + t [Equation] Lb , c [ t ] (see table 2).
Revised: = + t V Lb , c [ t ] (see table 2).

Original: A.3.2 B ETA CLIPPING
Revised: A.4.2 B ETA CLIPPING

Original: As the beta β ( l ) i (a transformation of the membrane time constant) of every neuron was optimised, we had to enforce correct neuron dynamics by clipping the values into the range [0,1].
Revised: As the beta β ( l ) i (a transformation of the membrane time constant) of every neuron was optimised, we had to enforce correct neuron dynamics by clipping the values into the range [0,1].

Original: Note that β ( l ) i = 0 implies no memory i.e. a binary neuron, 0 < β ( l ) i < 1 implies decaying memory i.e. a LIF neuron and β ( l ) i = 1 implies full memory i.e. an IF neuron.
Revised: Note that β ( l ) i = 0 implies no memory i.e. a binary neuron, 0 < β ( l ) i < 1 implies decaying memory i.e. a LIF neuron and β ( l ) i = 1 implies full memory i.e. an IF neuron.

Original: [Equation]
Revised: [Equation]

Original: A.3.3 W EIGHT INITIALISATION
Revised: A.4.3 W EIGHT INITIALISATION

Original: The network weights in a layer were sampled from a uniform distribution U ( −+ N − 1 , + N − 1 ), except for the Yin-Yang dataset for which the weights were sampled from U ( −+ 2 N − 1 , + 2 N − 1 ).
Revised: The network weights in a layer were sampled from a uniform distribution U ( −+ N − 1 , + N − 1 ), except for the Yin-Yang dataset for which the weights were sampled from U ( −+ 2 N − 1 , + 2 N − 1 ).

Original: For the feedforward layers N was set to the number of afferent connections to the layer and for the convolutional layers N = k 2 for kernel shape k × k .
Revised: For the feedforward layers N was set to the number of afferent connections to the layer and for the convolutional layers N = k 2 for kernel shape k × k .

Original: The bias terms were initialised to 0 in all networks.
Revised: The bias terms were initialised to 0 in all networks.

Original: All neurons in the hidden layers were initialised with a membrane time constant τ = 10ms and τ = 20ms for the readout neurons.
Revised: All neurons in the hidden layers were initialised with a membrane time constant τ = 10ms and τ = 20ms for the readout neurons.

Original: A.3.4 S UPERVISED TRAINING LOSS
Revised: A.4.4 S UPERVISED TRAINING LOSS

Original: All networks were trained to minimise a cross-entropy loss
Revised: All networks were trained to minimise a cross-entropy loss

Original: [Equation]
Revised: [Equation]

Original: 18) with B and C being the number of batch samples and dataset classes respectively, and y b , c ∈ {0,1} C and p b , c being the one hot target vector and network prediction probabilities respectively.
Revised: with B and C being the number of batch samples and dataset classes respectively, and y b , c ∈ {0,1} C and p b , c being the one hot target vector and network prediction probabilities respectively.

Original: The prediction probabilities p b , c were obtained by passing the readout neuron outputs o b , c through the softmax function.
Revised: The prediction probabilities p b , c were obtained by passing the readout neuron outputs o b , c through the softmax function.

Original: [Equation]
Revised: [Equation]

Original: A.3.5 S URROGATE GRADIENT
Revised: A.4.5 S URROGATE GRADIENT

Original: The backprop algorithm requires all nodes within the computational graph of optimisation to be differentiable.
Revised: The backprop algorithm requires all nodes within the computational graph of optimisation to be differentiable.

Original: This requirements is however violated in a SNN due to the non-differentiable heavy stepwise spike function f .
Revised: This requirements is however violated in a SNN due to the non-differentiable heavy stepwise spike function f .

Original: To permit the use of backprop, we replaced the undeﬁned derivate dfdV of the function f with a surrogate gradient df sur dV (Zenke & Ganguli, 2018), which has been shown to work well in practice (Zenke & Vogels, 2021).
Revised: To permit the use of backprop, we replaced the undeﬁned derivate dfdV of the function f with a surrogate gradient df sur dV (Zenke & Ganguli, 2018), which has been shown to work well in practice (Zenke & Vogels, 2021).

Original: Here hyperparameter β sur (which we set to 10 in all experiments) deﬁnes the slope of the gradient.
Revised: Here hyperparameter β sur (which we set to 10 in all experiments) deﬁnes the slope of the gradient.

Original: [Equation]
Revised: [Equation]

Original: A.3.6 T RAINING PROCEDURE
Revised: A.4.6 T RAINING PROCEDURE

Original: All models were trained using the Adam optimiser (with default parameters) (Kingma & Ba, 2014).
Revised: All models were trained using the Adam optimiser (with default parameters) (Kingma & Ba, 2014).

Original: Training started with an initial learning rate, which was decayed by a factor of 10 every time the number of epochs reached a new milestone, after which the best performing model (that achieved lowest training loss) was loaded and training continued.
Revised: Training started with an initial learning rate, which was decayed by a factor of 10 every time the number of epochs reached a new milestone, after which the best performing model (that achieved lowest training loss) was loaded and training continued.

Original: A.3.7 T RAINING HYPERPARAMETERS
Revised: A.4.7 T RAINING HYPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: Dataset classes

Original: [Figure]
Revised: Epochs

Original: [Table] [Figure] [Table] [Figure]
Revised: Learning rate 0.

Original: [Table] [Figure] [Table]
Revised: Batch size B Simulation steps T

Original: [Figure] [Table] [Figure]
Revised: Time resolution ∆ t (ms)

Original: Milestones (50,100) (15,90,120) (30,60,90) (15,90,120) (30,60,90) (30,60,90) (30,60,90)
Revised: Milestones (50,100) (15,90,120) (30,60,90) (15,90,120) (30,60,90) (30,60,90) (30,60,90)

Revised: [Table]

Original: Output function sum sum max sum sum sum sum
Revised: function sum sum max sum sum sum sum

Original: A.4 E XTENDED SPEEDUP RESULTS
Revised: A.5 E XTENDED SPEEDUP RESULTS