Training Spiking Neural Networks with Local Tandem Learning
Abstract
Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efﬁcient over their predecessors.
However, there is a lack of an efﬁcient and generalized training method for deep SNNs, especially for deployment on analog computing substrates.
In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL).
The LTL rule follows the teacherstudent learning approach by mimicking the intermediate feature representations of a pre-trained ANN.
By decoupling the learning of network layers and leveraging highly informative supervisor signals, we demonstrate rapid network convergence within ﬁve training epochs on the CIFAR-10 dataset while having low computational complexity.
Our experimental results have also shown that the SNNs thus trained can achieve comparable accuracies to their teacher ANNs on CIFAR-10,
CIFAR-100, and Tiny ImageNet datasets.
Moreover, the proposed LTL rule is hardware friendly.
It can be easily implemented on-chip to perform fast parameter calibration and provide robustness against the notorious device non-ideality issues.
It, therefore, opens up a myriad of opportunities for training and deployment of
SNN on ultra-low-power mixed-signal neuromorphic computing chips.
Introduction
Over the last decade, artiﬁcial neural networks (ANNs) have improved the perceptual and cognitive capabilities of machines by leaps and bounds, and have become the de-facto standard for many pattern recognition tasks including computer vision [30, 52, 53, 62, 35], speech processing [60, 39], language understanding [3] and robotics [51].
Despite their superior performance, ANNs are computationally expensive to be deployed on ubiquitous mobile and edge computing devices due to high memory and computation requirements
Spiking neural networks (SNNs), the third-generation artiﬁcial neural networks, have gained growing research attention due to their greater biological plausibility and potential to realize ultra-lowpower computation as observed in biological neural networks.
Leveraging the sparse, spike-driven computation and ﬁne-grain parallelism, the fully digital neuromorphic computing (NC) chips like
TrueNorth [2], Loihi [11], and Tianjic [42], that support the efﬁcient inference of SNNs, have indeed demonstrated orders of magnitude improved power efﬁciency over GPU-based AI solutions.
Moreover, the emerging in situ mixed-signal NC chips [54, 47], enabled by nascent non-volatile technologies, can further boost the hardware efﬁciency by a large margin over the aforementioned digital chips.
Despite remarkable progress in neuromorphic hardware development, how to efﬁciently and effectively train the core computational model, spiking neural network, remains a challenging research topic.
It, therefore, impedes the development of efﬁcient neuromorphic training chips as well as the wide adoption of neuromorphic solutions in mainstream AI applications.
The existing training algorithms for deep SNNs can be grouped into two categories: ANN-to-SNN conversion and gradient-based direct training.
For ANN-to-SNN conversion methods, they propose to reuse network weights from more easily trainable ANNs.
This can be viewed as a speciﬁc example of Teacher-Student (T-S) learning that transfers the knowledge from a teacher ANN to a student SNN in the form of network weights.
By properly determining the neuronal ﬁring threshold and initial membrane potentials for SNNs, recent studies show that the activation values of ANNs can be well approximated with the ﬁring rate of spiking neurons, achieving near-lossless network conversion on a number of challenging AI benchmarks [7, 49, 14, 22, 21, 46, 34, 12, 4, 5, 63].
Nevertheless, these network conversion methods are developed solely based on the non-leaky integrate-and-ﬁre (IF) neuron model and typically require a large time window so as to reach a reliable ﬁring rate approximation.
It is, therefore, not straightforward and efﬁcient to deploy these converted SNNs onto the existing neuromorphic chips.
In another vein of research, the gradient-based direct training methods explicitly model each spiking neuron as a self-recurrent neural network and leverage the canonical Backpropagation Through Time (BPTT) algorithm to optimize the network parameters.
The non-differentiable spiking activation function is typically circumvented with continuous surrogate gradient (SG) functions during error backpropagation [50, 58, 59, 38, 8, 15, 45, 61].
Despite their compatibility with event-based inputs and different spiking neuron models, they are computational and memory inefﬁcient to operate in practice.
Moreover, the gradient approximation error introduced by these SG functions tends to accumulate over layers, causing signiﬁcant performance degradation in the face of the deep network structure and short time window [57].
In general, SNN learning algorithms can be categorized into off-chip learning [16, 64] and onchip learning [9, 41, 43].
Almost all of the direct SNN training methods discussed above belong to the off-chip learning category.
Due to the lack of effective ways to exploit the high level of sparsity in spiking activities and the requirement to store non-local information for credit assignment, these off-chip methods exhibit very low training efﬁciency.
Moreover, due to notorious device non-ideality problems [6], the actual network dynamics will deviate from the off-chip simulated ones, causing the accuracy of off-chip trained SNNs degrades signiﬁcantly when deployed onto the analog computing substrates [44, 24, 37, 1].
To address these problems, recent work proposes on-chip learning algorithms in the form of local Hebbian learning [11, 41, 28] and approximation of gradient-based learning [32, 40, 10, 19], while the effectiveness of these algorithms had only been demonstrated on simple benchmarks, such as MNIST and N-MNIST datasets.
To address the aforementioned problems in SNNs training and hardware deployment, we put forward a generalized SNN learning rule in this paper, which we referred to as the Local Tandem Learning (LTL) rule.
The LTL rule takes the best of both ANN-to-SNN conversion and gradient-based training methods.
On one hand, it makes good use of highly effective intermediate feature representations of ANNs to supervise the training of SNNs.
By doing so, we show that it can achieve rapid network convergence within ﬁve training epochs on the CIFAR-10 dataset with low computational complexity.
On the other hand, the LTL rule adopts the gradient-based approach to perform knowledge transfer, which can support different neuron models and achieve rapid pattern recognition.
By propagating gradient information locally within a layer, it can also alleviate the compounding gradient approximation errors of the SG method and lead to near-lossless knowledge transfer on CIFAR-10,
CIFAR-100, and Tiny ImageNet datasets.
Moreover, the LTL rule is designed to be hardware friendly, which can perform efﬁcient on-chip learning using only local information.
Under this on-chip setting, we demonstrate that the LTL rule is capable of addressing the notorious device non-ideality issues of analog computing substrates, including device mismatch, quantization noise, thermal noise, and neuron silencing.
Layer Layer
Layer Layer
Layer Layer
ANN (teacher)
Data
SNN (student) (a)
T=
T= T=
Layer (b)
T=
T= T=
Layer (c)
Synapse
Membrane Potential
Spike Accumulator Error
Boxcar()
SpikeGenerator
Sign()
Reset
[Figure]
(d)  Methods 2.1 Spiking Neuron Model
To demonstrate the proposed LTL rule is compatible with different spiking neuron models, we base our study on both non-leaky integrate-and-ﬁre (IF) [48] and leaky integrate-and-ﬁre (LIF) neuron models [18], whose neuronal dynamics can be described by the following discrete-time formulation:
[Equation]
(1) with [Equation]
[Equation]
where U li [ t ] and I li [ t ] refer to the subthreshold membrane potential of and input current to neuron i at layer l , respectively.
↵ ⌘ exp (  dt/ ⌧ m ) is the membrane potential decaying constant, wherein ⌧ m is the membrane time constant and dt is the simulation time step.
For IF neuron, ↵ takes a value of 1 .
# denotes the neuronal ﬁring threshold.
w l  1 ij represents the connection weight from neuron j of the preceding layer l  1 and b li denotes the constant injecting current to neuron i .
S li [ t  1] indicates the occurrence of an output spike from neuron i at time step t  1 , which is determined according to the spiking activation function as per
[Equation]
2.2 Local Tandem Learning
As illustrated in Figure 1(a), the LTL rule follows the T-S learning approach [20], whereby the intermediate feature representation of a pre-trained ANN is transferred to SNN through layer-wise local loss functions.
In contrast to the ANN-to-SNN conversion methods, we establish the feature representation equivalence at the neuron level rather than at the synapse level, which provides the ﬂexibility for choosing any neuron model to be used in the SNN.
On the other hand, with the proposed spatially local loss function, we simplify the spatial-temporal credit assignment required in the endto-end direct training methods, which can dramatically improve the network convergence speed and meanwhile reduce the computational complexity.
In the following, we introduce two versions of the LTL rule, ofﬂine and online, depending on whether the temporal locality constraint is imposed.
Ofﬂine Learning Following the T-S learning approach, we consider the intermediate feature representation of a pre-trained ANN as the knowledge and train an SNN to reproduce an equivalent feature representation via layer-wise loss functions.
In particular, we establish an equivalence between the normalized activation values of an ANN and the global average ﬁring rates of an SNN.
To reduce the discrepancy between these two quantities, we adopt the mean square error (MSE) loss function and apply it separately for each layer.
Thus, for any layer l , the local loss function L l is deﬁned as follows
[Equation]
where ˆ y l is the output of ANN layer l .
y norm is a normalization constant that takes the value of 99 th or 99 .
9 th percentile across all ˆ y li .
This can alleviate the effect of outliers compared to using the maximum activation value [48].
T w is the time window size.
C l [ T w ] = ⌃ T w t =1 S
[ t ] is total spike count.
As the computational graph shown in Figure 1(b), we adopt the BPTT algorithm to resolve the temporal credit assignment problem, and the weight gradients can be derived as @
[Equation]
with
[Equation]
where
[Equation]
To resolve the problem of the non-differentiable spiking activation function, we apply the surrogate gradient {+method, i.e., ⇥ 
+}
( x ) ⇡ ✓ 0 ( x ) .
Speciﬁcally, we adopt the boxcar function for ✓ 0 ( x ) that supports convenient and efﬁcient on-chip implementation.
[Equation]
where p controls the permissible range of membrane potentials that allow gradients to pass through, and we tune this hyperparameter separately for each dataset.
By substituting Eq.
(8) into Eqs.
(6) and (7), we can yield the ultimate form of weight gradients, and we can update the weights according to the stochastic gradient descent method or its adaptive variants.
See Supplementary Materials Section
A.1 for a more detailed derivation of the gradients to weight and bias terms.
Online Learning The ofﬂine LTL rule requires storing intermediate synaptic and membrane state variables so as to be used during error backpropagation, which is prohibited for on-chip learning where memory resources are limited.
To address this problem, we introduce an online LTL rule, whose loss function is designed to be both spatially and temporally local.
To achieve the temporal locality, we use the moving average ﬁring rate, which can be calculated at each time step, to replace the global ﬁring rate used in Eq.
It hence yields the following local loss function
[Equation]
Compared to the ofﬂine version, the gradient update is much simpler now:
[Equation]
where ⇣ l [ t ] can be directly computed from Eq.
[Equation]
The computational graph of the online LTL rule is provided in Figure 1(c).
It is worth noting that the ﬁrst few time steps of the ﬁring rate calculation are relatively noisy.
Nevertheless, this issue can be easily addressed by treating the ﬁrst few steps as the warm-up period, during which the parameter updates are not allowed (see Supplementary Materials Section C for a study on the effect of the warm-up period).
By doing so, it can also reduce the overall training cost.
As will be discussed in
Sections 3.1 and 3.4, this online version can signiﬁcantly reduce the computational complexity, while achieving a comparable test accuracy to that of the ofﬂine version.
On-chip Implementation To allow a convenient and efﬁcient on-chip implementation of the proposed online LTL rule, we carefully designed the learning circuits as illustrated in Figure 1(d).
The output spike count C l is updated at the spike accumulator, and it is compared to the local target ˆ y l following the layer-wise loss function deﬁned in Eq.
This error term is further feedback to the neuron to update the synaptic parameters.
Note that the synaptic updates are gated by sign ( · ) and boxcar ( · ) functions, which can signiﬁcantly reduce the overall number of parameter updates.
We would like to highlight that the proposed LTL learning rule is more hardware-friendly than the recently introduced hardware-in-the-loop (HIL) training approach [10, 19].
The HIL training approaches require two-way information communication, that is, (1) reading intermediate neuronal states from the NC chip to the host computer to perform off-chip training and (2) writing the updated weights from the host computer to the NC chip.
Given the sequential nature of these two processes and the high implementation cost for reading neuronal states (e.g., requiring to implement costly analog-to-digital converters for analog spiking neurons), HIL training approaches are expensive to deploy in practice.
In contrast, the LTL training rule can be implemented efﬁciently on-chip by simultaneously extracting the layerwise targets from ANNs, running on the host computer, for data batch i + 1 and performing on-chip SNN training for data batch i .
This is similar to conventional ANN training, where the data preprocessing of the next data batch is performed on the CPU and meanwhile the current data batch is used for ANN training on the GPU. The only difference is that the input data is preprocessed by the pre-trained ANN to extract the targets for intermediate layers for the proposed LTL rule. Given the inference of ANN can be performed in parallel on the host computer, the overall training time is bottlenecked by the NC chip that operates in a sequential mode, where only one sample is been processed at a time. Therefore, our method has much lower hardware and time complexity.
Experiments
In this section, we evaluate the effectiveness of the proposed LTL rule on the image classiﬁcation task with CIFAR-10 [29], CIFAR-100 [29], and Tiny ImageNet [55] datasets.
We perform a comprehensive study to demonstrate its superiority in: 1.
accurate, rapid, and efﬁcient pattern recognition; 2.
rapid network convergence with low computational complexity; 3. provide robustness against hardwarerelated noises.
More details about the experimental datasets and implementation details are provided in the Supplementary Materials Section B, and the source code can be found at 2 .
3.1 Accurate and Scalable Image Classiﬁcation
Here, we report the classiﬁcation results of LTL trained SNNs on CIFAR-10, CIFAR-100 and Tiny
ImageNet datasets against other SNN learning rules, including ANN-to-SNN conversion [57, 49, 46, 34, 22, 21, 12, 4, 5, 31, 17] and direct SNN training [45, 13] methods.
Given the network architectures and data preparation processes vary slightly across different work, therefore, we focus our discussions on the conversion or transfer errors between the ANNs and SNNs whenever the data is available.
[Figure]
On the CIFAR-10 dataset, as reported in Table 1, our LTL ofﬂine trained spiking VGG-11 has surpassed the accuracy of teacher ANN at T w = 16 , this result suggests high effectiveness of the proposed T-S learning approach in transferring the knowledge through local objectives.
Moreover, the LTL rule can take advantage of a larger time window that offers a greater representation power, and further improves the performance of SNNs at T w = 32 .
It is also worth mentioning that by transferring the knowledge in a learnable way, the proposed LTL rule is compatible with different neuron models, manifested through high accuracies achieved by both IF and LIF neuron models.
Although slightly degrades from the ofﬂine version, the online trained SNNs can still achieve comparable results to their teacher models with accuracies dropping by less than 1%, which indicates that removing temporal dependency between time steps has little impact on the training of SNNs (see
Section D for a detailed study on why the online LTL rule can perform as effectively as the ofﬂine version).
For the VGG-16 model, we notice a modest accuracy drop that is more than 1% from the teacher model, which can be explained by the compounding layerwise representation errors.
Fortunately, such errors can be well addressed with a larger time window.
Speciﬁcally, at T w = 32 , the accuracy gap to the teacher ANN narrows down to only 0.13% and 0.33% respectively for LIF and IF neuron models under the ofﬂine training setting.
When compared against other SOTA learning algorithms, the proposed LTL rule can achieve comparable performance in terms of both the ﬁnal SNN model accuracies and the accuracy drop from ANN models.
The same conclusions can also be drawn from the results of CIFAR-100 and Tiny ImageNet datasets, it hence suggests the proposed LTL rule is a generalized, scalable, and highly effective method for constructing high-performance SNNs.
3.2 Rapid and Efﬁcient Pattern Recognition
To realize the goal as a power-efﬁcient alternative to the mainstream ANNs in solving pattern recognition tasks, SNNs should make accurate and rapid predictions with a low ﬁring rate.
Here, we study whether the proposed LTL rule is able to train SNNs that can cope with a short time window.
Speciﬁcally, we design experiments that progressively increase the time window T w from 4 toduring training, and we compare the ofﬂine LTL rule against the STBP [58] learning rule on the
CIFAR-10 dataset.
To ensure a fair comparison, we use the VGG-11 architecture with the same experimental settings, except that the LTL rule uses local objectives and the STBP rule uses a global objective.
We note that the discrete neural representation of SNNs is equivalent to ANNs with quantized activation values [57].
Hence, we also compare our results to quantized ANNs obtained through quantization-aware training [25].
As shown in Figure 2(a), we observe that the accuracy of the quantized ANN degrades rapidly when the total quantization level is lower than 16 , while both LTL and STBP trained SNNs can [Table] maintain a high accuracy even at T w = 4 .
It suggests the surrogate gradient approach adopted in
SNN training can outperform the naive straight-through estimator [25] in addressing the discontinuity in the activation function.
Moreover, the ofﬂine LTL rule performs better than the STBP rule, which can be explained by the fact that the gradient approximation errors of the surrogate gradient function will not compound across layers as in the STBP training [57].
Similar results have also been observed in the VGG-16 architecture (See Supplementary Materials Section E).
To shed light on the energy efﬁciency of LTL trained SNN models, we follow the common practice by computing the total synaptic operations (SynOps) ratio of SNN to ANN under different T w [56].
In general, the total SynOps (i.e., FLOPs) required by an ANN is a constant number that depends on the speciﬁc network architecture in use, while the total computations typically increase linearly with T w for rate-based SNNs.
As shown in Figure 2(b), to achieve comparable accuracy to the ANN counterpart, our SNN only requires 0.43 ⇥ total SynOps with T w = 8 .
It is worth noting that the multiply-accumulate (MAC) operation required by ANNs is 5 ⇥ more expensive than the accumulate [Figure] (AC) operations used in SNNs under 45 nm CMOS technology [23].
As such, our SNNs can yield an order of magnitude energy saving over the mainstream ANNs. Notably, this value can be further boosted when deploying our SNNs onto the emerging mixed-signal NC chips.
Moreover, when comparing with QCFS [5] which is a recently introduced ANN-to-SNN conversion method, the proposed LTL rule requires fewer SynOps probably due to a higher percentage of neurons remaining silent in our network.
This can be explained by the round-down operation that has been taken when calculating the target spike count in our method.
3.3 Rapid Learning Convergence
By decoupling the learning of each layer through local objectives, the proposed LTL rule cleverly circumvents the spatial credit assignment and reduces the credit assignment to only the temporal domain.
Moreover, the gradient approximation errors of the surrogate gradient learning method are alleviated as the effective gradient transportation distance is reduced.
Taking beneﬁts from these, the network convergence speed can be improved signiﬁcantly compared to those trained with the STBP and TET [13] rule.
As the learning curves are shown in Figure 3, both the ofﬂine and online LTL rules can converge rapidly within ﬁve epochs on the CIFAR-10 dataset.
In contrast, the accuracy is still increasing at Epoch 50 for both the STBP and TET learning rules.
Note that we follow the same settings for all these experiments, except for the learning objectives.
Similar results have also been observed in the VGG-16 architecture (see Supplementary Materials Section F).
3.4 Low Memory and Time Complexity
Memory Time
STBP [58] O ( MT w L ) [Equation] In this section, we study the memory and time complexity of the proposed LTL rule and compare it against other popular learning rules in Table 2.
The STBP [58] rule requires storing the intermediate state of each neuron for all time steps and layers so as to be used during backpropagation, resulting in O ( MT w L ) memory complexity, where M is the number of neurons in a layer, T w is the time window size, and L is the total number of layers.
The ofﬂine LTL rule has the same memory complexity as the STBP rule if all layers are trained in parallel.
However, as will be discussed soon, we can reduce the memory complexity by training all layers in sequence.
Notably, by updating the network parameters using only local information, the online LTL and DECOLLE [26] learning rules do not require storing any intermediate state, resulting in O (1) memory complexity.
In terms of time complexity, each parameter update of the STBP rule requires NMT w multiplications, where N is the number of input neurons to a layer.
Due to layer-wise interlocking, all layers need to be updated in sequence from top to bottom, requiring O ( NMT w L ) time complexity to update all network parameters once.
For ofﬂine LTL, by decoupling the dependency between layers and training all layers in parallel, the time complexity can be reduced to O ( NMT w ) .
As mentioned earlier, we can trade off the memory complexity with the time complexity by training the network layers progressively one after the other.
In this way, the memory usage of the ofﬂine LTL rule will be reduced to O ( MT w ) , while the time consumption will be increased to O ( NMT w L ) .
To validate the effectiveness of this training strategy, we perform an experiment on the CIFAR-10 dataset using
VGG-11 architecture and we obtain a comparable result (92.98%) to that obtained when all layers are trained concurrently (93.20%).
In this work, to achieve the shortest training time, we use the parallel training strategy as the default for all the other reported experiments.
Other users can ﬂexibly select the training strategy according to their available computing resources.
For online LTL, by leveraging the temporal local errors, the time complexity can be further reduced to O ( NM ) .
Note that the DECOLLE requires an additional readout layer to calculate local errors, thereby requiring more time than online LTL.
It is also worth mentioning that by gating the parameter updates with the proposed efﬁcient on-chip implementation, we can further reduce the time complexity for online LTL.
To provide a more concrete analysis of the memory and time complexity, we have measured the actual GPU memory usage and the training time on the CIFAR-10 dataset, which aligns well with the earlier theoretical analysis (see Supplementary Materials Section G).
In sum, by coupling the low computational complexity with the rapid learning convergence discussed in Section 3.3, the proposed LTL rule can dramatically improve the training efﬁciency of SNNs.
3.5 Robust to Hardware-related Noises
Device non-ideality issues of analog computing substrates remain a major hurdle for deploying
SNNs on ultra-low-power mixed-signal NC chips.
The proposed LTL rule can, however, address this challenge by enabling on-chip training in a noise-aware manner.
Here, we examine the effectiveness of the on-chip LTL rule in addressing hardware-related noises and we focus on four typical noise sources: device mismatch, quantization noise, thermal noise, and neuron silencing [6].
In particular, we use the noise models introduced by Julian et al.
[6], which were measured on DYNAP-SE [36] mixed-signal neuromorphic processor.
Therefore, it can faithfully reﬂect the actual hardware noises that one can expect in the real scenario.
To understand the effect of these noises, we ﬁrst take an
LTL pre-trained SNN and apply noise at different levels to it.
Then, we use the online LTL rule to calibrate the pre-trained SNN model under the given noise.
Device Mismatch Device mismatch causes neuronal and synaptic parameters to vary from the desired ones.
To simulate the parameter mismatch, we add Gaussian noise to both the initial network parameters and their parameter updates following ✓ 0 ⇠ N ( ✓ , ✓ ) .
In this model,  controls the level of mismatch, and we vary it from 0.05 to 0.4 to simulate different levels of noise.
As shown in
Figure 4(a), the accuracy of the pre-trained SNN degrades by more than 20% under the mismatch level  = 0 .
Notably, the proposed LTL rule can perform fast calibration and restore the original accuracy with only one training epoch.
This will be advantageous for those devices that have poor endurance.
Quantization Noise The non-volatile memory devices typically have a limited number of analog states, thereby limiting the usable number of bits per network parameter.
To simulate such a quantization noise, we perform post-training-quantization to the bit-width of 7 , 6 , 5 , 4 , 3 .
As shown in Figure 4(b), the SNN performs robustly to the quantization noise for bit-precision above 3 .
After which, the accuracy drops sharply by more than 30%.
Following the quantization aware training [25], the proposed LTL rule can alleviate the effect of this type of noise and successfully recover the accuracy by around 17% .
Nevertheless, due to the reduced representation power, the resulting 3-bit precision model is still lagging behind the full precision one by about 15%.
Thermal Noise Thermal noise is intrinsic to the neuromorphic devices, which can be modeled as
Gaussian noise on the input currents I {+
 ⇠+} N ( I,  I ) .
Similar to the device mismatch noise, we simulate different levels of thermal noise by varying the  from 0 .
01 to 0 .
As illustrated in Figure 4(c), the model is highly sensitive to thermal noise and the accuracy drops steadily with a growing level of noise.
We notice that the LTL rule can address such noise and quickly recover the accuracy in large part with only one training epoch.
Neuron Silencing Neuron silencing noise corresponds to the situation when a portion of spiking neurons fails to respond, and hence disturbs the network dynamics.
We simulate neuron silencing [Figure] noise by randomly masking the neuron outputs with a failure rate varying from 10% to 50% .
As shown in Figure 4(d), the accuracy drops steadily with a growing amount of failed neurons.
Similar to other noise types, the accuracies are quickly restored with the on-chip LTL training and maintained above 80% even under the challenging case when 50% of the neurons failed.
Comparison with the HIL Training Approach The proposed LTL rule exhibits better noise robustness and scalability than the HIL training approaches. To demonstrate this, we compare the online LTL rule with the HIL training approach [10] in addressing the aforementioned device-related noises (see Supplementary Material Section H for the details of experimental set-up and results). The LTL rule achieves a comparable accuracy with the HIL approach for small and moderate levels of noise (see Table 7), but it appears to be more robust when facing stronger noise levels (see Table 8).
Additionally, the LTL rule is more robust to the choice of the learning rate (see Table 9).
In summary, the proposed LTL learning rule is both effective and efﬁcient for addressing the device non-idealities of mixed-signal NC chips.
Conclusion
In this paper, we propose the Local Tandem Learning (LTL) rule for SNNs.
Taking inspiration from the teacher-student learning approach, we leverage the highly effective feature representations of easily trainable ANN to guide the training of SNNs.
This simpliﬁes the credit assignment problem and leads to rapid network convergence with signiﬁcantly reduced computational complexity.
We also demonstrate the required temporal credit assignment can be well approximated with locally available information, and we provide an efﬁcient on-chip implementation of the proposed LTL rule.
Under this on-chip setting, we demonstrate that the LTL rule can provide robustness against various device non-ideality issues.
It, therefore, opens up myriad opportunities for rapid and efﬁcient training and deployment of SNNs on ultra-low-power mix-signal NC platforms.
For future work, we will explore the proposed teacher-student learning approach on the recurrent neural networks and transformer architectures so as to handle those temporally rich signals, such as speech, video, and text.
Acknowledgments and Disclosure of [-your work [Yes] See Section 
-]Funding
This research work is supported by IAF, A*STAR, SOITEC, NXP, National University of Singapore under FD-fAbrICS: Joint Lab for FD-SOI Always-on Intelligent & Connected Systems (Award
I2001E0053), and National Research Foundation, Singapore under its Medium Sized Centre for
Advanced Robotics Technology Innovation (WBS: A-0009428-09-00).
This research is also supported by National Science Foundation of China (Grant Number: 62106038), Guangdong Provincial Key
Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen, China (Grant
No. B10120210117-KP02), Shenzhen Research Institute of Big Data, the National Key Research and
Development Program of China (Grant No. 2021ZD0200300), CCF-Hikvision Open Fund, and The
Hong Kong Polytechnic University under Grant No. P0043563.