S EQUENCE A PPROXIMATION USING F EEDFORWARD S PIKING N EURAL N ETWORK FOR S PATIOTEMPORAL L EARNING : T HEORY AND O PTIMIZATION M ETHODS
A dynamical system of spiking neurons with only feedforward connections can classify spatiotemporal patterns without recurrent connections.
However, the theoretical construct of a feedforward spiking neural network (SNN) for approximating a temporal sequence remains unclear, making it challenging to optimize SNN architectures for learning complex spatiotemporal patterns.
In this work, we establish a theoretical framework to understand and improve sequence approximation using a feedforward SNN.
Our framework shows that a feedforward SNN with one neuron per layer and skip-layer connections can approximate the mapping function between any arbitrary pairs of input and output spike train on a compact domain.
Moreover, we prove that heterogeneous neurons with varying dynamics and skip-layer connections improve sequence approximation using feedforward SNN.
Consequently, we propose SNN architectures incorporating the preceding constructs that are trained using supervised backpropagation-through-time (BPTT) and unsupervised spiking-timing-dependent plasticity (STDP) algorithms for classification of spatiotemporal data.
A dual-search-space Bayesian optimization method is developed to optimize architecture and parameters of the proposed SNN with heterogeneous neuron dynamics and skip-layer connections.
I NTRODUCTION
Spiking neural network (SNN) (Ponulak & Kasinski, 2011) uses biologically inspired neurons and synaptic connections trainable with either biological learning rules such as spike-timingdependent plasticity (STDP) (Gerstner & Kistler, 2002) or statistical training algorithms such as backpropagation-through-time (BPTT) (Werbos, 1990).
The SNNs with simple leaky integrate-andfire (LIF) neurons and supervised training have shown classification performance similar to deep neural networks (DNN) while being energy efficient (Kim et al., 2020b; Wu et al., 2019; Srinivasan & Roy, 2019).
One of SNN’s main difference from DNN is that the neurons are dynamical systems with internal states evolving over time, making it possible for SNN to learn temporal patterns without recurrent connections.
Empirical results on feedforward-only SNN models show good performance for spatiotemporal data classification, using either supervised training (Lee et al., 2016;
Kaiser et al., 2020; Khoei et al., 2020), or unsupervised learning (She et al., 2021).
However, while empirical results are promising, a lack of theoretical understanding of sequence approximation using
SNN makes it challenging to optimize performance on complex spatiotemporal datasets.
In this work, we develop a theoretical framework for analyzing and improving sequence approximation using feedforward SNN.
We view a feedforward connection of spiking neurons as a spike propagation path, hereafter referred to as a memory pathways (She et al., 2021), that maps an input spike train with an arbitrary frequency to an output spike train with a target frequency.
Consequently, we argue that an SNN with many memory pathways can approximate a temporal sequence of spike trains with time-varying unknown frequencies using a series of pre-defined output spike trains with known frequencies.
Our theoretical framework aims to first establish SNN’s ability to map frequencies of input/output spike trains within arbitrarily small error; and next, derive the basic principles for adapting neuron dynamics and SNN architecture to improve sequence approximation.
The theoretical derivations are then investigated with experimental studies on feedforward SNN for spatiotemporal classifications.
We adopt the basic design principles for improving sequence approximation to optimize SNN architectures and study whether these networks can be trained to improve performance for spatiotemporal classification tasks.
The key contributions of this work are:
• We prove that any spike-sequence-to-spike-sequence mapping functions on a compact domain can be approximated by feedforward SNN with one neuron per layer using skip-layer connections, which cannot be achieved if no skip-layer connection is used.
• We prove that using heterogeneous neurons having different dynamics and skip-layer connection increases the number of memory pathways a feedforward SNN can achieve and hence, improves SNN’s capability to represent arbitrary sequences.
• We develop complex SNN architectures using the preceding theoretical observations and experimentally demonstrate that they can be trained with supervised BPTT and unsupervised STDP for spatiotemporal data classification.
• We design a dual-search-space option for Bayesian optimization process to sequentially optimize network architectures and neuron dynamics of a feedforward SNN considering heterogeneity and skip-layer connection to improve learning and classification of spatiotemporal patterns.
We experimentally demonstrate that our network design principles coupled with the dual-searchspace Bayesian optimization improve classification performance on DVS Gesture
(Amir et al., 2017), N-caltech (Orchard et al., 2015), and sequential MNIST.
Results show that the design principles derived using our theoretical framework for sequence approximation can improve spatiotemporal classification performance of SNN.
R ELATED W ORK
Prior theoretical approaches to analyze SNN often focus on the storage and retrieval of precise spike patterns (Amit & Huang, 2010; Brea et al., 2013).
There are also works that consider SNN for solving optimization problems (Chou et al., 2018; Binas et al., 2016) and works that analyze the dynamics of SNN (Zhang et al., 2019; Barrett et al., 2013).
Those are different topics from the approximation of spike-sequence-to-spike-sequence mappings functions.
SNN that incorporates ex- and is theoretically capable of universal function approximation.
Feedforward SNN with specially designed spiking neuron models (Iannella & Back, 2001; Torikai et al., 2008) have been demonstrated for function approximation, while for networks using LIF neurons, function approximation has been shown with only empirical results (Farsa et al., 2015).
On the other hand, existing works that have developed efficient training process for SNN and demonstrated classification performance comparable to deep learning models, have mostly used simpler and generic LIF neuron models (Lee et al., 2016;
Kaiser et al., 2020; Kim et al., 2020b; Wu et al., 2019; Sengupta et al., 2019; Safa et al., 2021; Han et al., 2020).
Therefore, this paper develops the theoretical basis for function approximation using feedforward SNN with LIF neurons, and studies applications of the developed theoretical constructs in improving SNN-based spatiotemporal pattern classification.
The effectiveness of heterogeneous neurons (She et al., 2021) and skip-layer connections (Srinivasan & Roy, 2019; Sengupta et al., 2019) in SNN has been empirically studied in the past.
However, no theoretical approach has been presented to understand why such methods improve learning of spike sequences, and how to optimize SNN’s architecture and parameters to effectively exploit these design constructs.
It is possible to search for the optimal SNN configurations through optimization algorithms, but the large amount of hyper-parameters for spiking neurons and network structure creates a high-dimensional search space that is long and difficult to solve.
Bayesian optimization (Snoek et al., 2012) uses collected data points to make decision on the next test point that could provide improvement, thus accelerates the optimization process.
Prior works (Parsa et al., 2019; Kim et al., 2020a) have shown that SNN performance can be effectively improved with Bayesian optimization.
While those works consider a single or a few neuron parameters, the dual-search-space Bayesian optimization proposed in this work optimizes both network architecture and neuron parameters efficiently by separating the discrete search spaces from the continuous search spaces. t [Figure] spikes v n
[Figure]
=
[Figure] t n [Figure] n
[Figure]
n [Figure] n [Figure]
n
[Figure]
spikes n
[Figure]
spikes t nd … … … … … Neuron Dynamic d One neuron per layer per dynamic
Layer number: m (b)
Neuron Dynamic d n … …
A PPROXIMATION T HEORY OF F EEDFORWARD SNN 3. D EFINITIONS AND N OTATIONS
Definition 1 Neuron Response Rate γ
For a spiking neuron n with membrane potential at v reset and input spike sequence with period t in , γ is the number of input spike n needs to reach v th .
Definition 2 Memory Pathways For a feedforward SNN with m layers, a memory pathway is defined as a spike propagation path from input to the output layer.
Two memory pathways are considered distinct if the set of neurons contained in them is different.
Definition 3 Minimal Multi-neuron-dynamic (mMND)
Network
A densely connected network in which each layer has an arbitrary number of neurons that have different neuron parameters.
All synapses from one pre-synaptic neuron have the same synaptic conductance.
Notations Neuron Delay t nd is the time for a spike from pre-synaptic neuron to arrive at its postsynaptic neurons, as shown in Figure 1(a).
For a feedforward SNN with m layers, a skip-layer connection can be defined with source layer and target layer pair ( l s , l t ) .
The output feature map from source layer is concatenated to the original input feature map of the target layer.
For the analysis of spike sequence in temporal space, the notation of T max and T min are defined as positive real numbers such that T max > T min .
ϵ > 0 is the error of approximation.
Figure 1(a) shows two memory pathways receiving an input spike sequence with time-varying periods.
As the neurons have different dynamics, the two memory pathways have different response to the input spike sequence.
An example of mMND network with m layers and n neuron dynamics is shown in Figure 1(b).
SNN with multilayer perceptron (MLP) structure can be considered a scaled-up mMND network with multiple neurons for each dynamic.
A network with convolutional structure can be considered a scaled-up mMND network with duplicated connections in each layer.
We analyze the correlation of network capacity and structure based on mMND networks.
The design of neuron heterogeneity can also be implemented in MLP-SNN and Conv-SNN as described in
Section 4.
The analysis for network capacity can be extended to those networks according to their specific layer dimensions.
3. M ODELING OF S PIKING N EURON
SNN consists of spiking neurons connected by synapses.
The spiking neuron model studied in this work is leaky integrate-and-fire (LIF) as defined by the following equations:
[Equation]
R m is membrane resistance, τ m = R m C m is time constant and C m is membrane capacitance.
a is the resting potential.
I is the sum of current from all input synapses that connect to the neuron.
A spike is generated when membrane potential v cross threshold and the neuron enters refractory period r , during which the neuron maintains its membrane potential at v reset .
The time it takes for a pre-synaptic neuron to send a spike to its post-synaptic neurons is t nd .
Neuron response rate γ is a property of a spiking neuron’s response to certain input spike sequence.
We show how the value of γ can be evaluated below.
Remark For any input spike sequence, each individual spike can be described with Dirac delta function δ ( t − t i ) where t i is the time of the i -th input spike.
Consider membrane potential of a spiking neuron receiving the input before reaching spiking threshold, with initial state at t = 0 with v = v reset , solving the differential equation (1) leads to (Gerstner, 1995):
[Equation]
Here, G is the conductance of input synapse connected to the neuron, which is trainable.
From (2), there exists a value of u such that v m ( t u − 1 ) < v threshold and v m ( t u ) >
= v threshold .
By evaluating (2) for u given neuron parameters and input spike sequence, the neuron response rate γ can be found.
3. A PPROXIMATION T HEOREM OF F EEDFORWARD SNN
To develop the approximation theorem for feedforward SNN, we first aim to understand the range of neuron response rate that can be achieved.
We show with Lemma 1 that for any input spike sequence with periods in a closed interval, it is possible to set the neuron response rate γ to any positive integer.
Based on this property, we show with Theorem 1 that by connecting a list of spiking neurons with certain γ sequentially and inserting skip-layer connections, network with spike period mapping function P ( t ) can be achieved to approximate the target spike sequence.
To understand whether this capability of feedforward SNN relies on skip-layer connections, we develop Lemma 2 to prove that skip-layer connections are indeed necessary.
In subsection 3.4 we investigate the correlation between approximation capability and network structures by analyzing the cutoff property of spiking neurons, which can change the network’s connectivity.
In our analysis, we focus on two particular designs: heterogeneous network (Lemma 4) and skip-layer connection (Lemma 5), and show their impact on the number of distinct memory pathways in a network.
All lemmas are formally proved in the appendix.
Lemma
For any input spike sequence with period t in in range [ T min , T max ] , there exists a spiking neuron n with fixed parameters v th , v reset , a , R m and τ m , such that by changing synaptic conductance G , it is possible to set the neuron response rate γ n to be any positive integer.
Proof Sketch.
(Formal proof in Appendix A)
Given an input spike sequence, the highest possible membrane potential decay ∆ v for any input t in ∈ [ T min , T max ] can be derived as a function of neuron parameters.
We show it is possible to make ∆ v tend to zero by setting the neuron parameters.
Since the decay of v can be negligible, γ n can be set to any positive integer by changing G .
Theorem 1 For any input and target output spike sequence pair with periods ( t in , t out ) ∈
[ T min , T max ] ×
[ T min , T max ] , there exists a minimal-layer-size network with skip-layer connections that has memory pathway with output spike period function P ( t ) such that | P ( t in ) − t out | < ϵ .
Proof Sketch.
(Formal proof in Appendix B) With skip-layer connections, there can be multiple memory pathways in a minimal-layer-size network as neurons can be either included or skipped through.
Hence it is possible to create memory pathways with different delay times for each input spike in a network. By connecting the output of those memory pathways to a common neuron n ′ , spike sequence of any arbitrary period t int such that t int < =
t in can be generated within ϵ .
By setting γ n ′ > 1 , the output from n ′ receiving input spike sequence with t int is t n ′ out = γ · t int .
Hence it is possible to achieve a network with output spike period P ( t ) such that | P ( t in ) − t out | <
ϵ .
Lemma 2 With no skip-layer connection, there does not exist a minimal-layer-size network that has output spike period function P ( t ) such that for any input and target output spike sequence pair with periods ( t in , t out ) ∈
[ T min , T max ] ×
[ T min , T max ] , | P ( t in ) − t out | <
ϵ .
Proof Sketch.
(Formal proof in Appendix C)
A minimal-layer-size network without skip-layer connection has only one memory pathway.
For a particular input spike sequence with period t in , different output period P ( t in ) can be achieved by changing γ of neurons in the memory pathway.
We show that there exists two output spike periods P ( t in ) and P ′ ( t in ) , such that P ( t in )
− P ′ ( t in ) is a constant value independent of network or neuron configurations, and there can be no P ′′ ( t in ) such that P ( t in ) <
P ′′ ( t in ) <
P ′ ( t in ) .
Therefore, for any minimal-layer-size network, there exists t out within the range of ( P ( t in ) , P ′ ( t in ))
such that | P ( t in ) − t out | <
ϵ does not hold true.
3. N ETWORK S TRUCTURE AND M EMORY P ATHWAYS
Based on Theorem 1, it is possible to approximate an input/output spike sequence mapping function using a minimal-layer-size network with specific configuration, which can be considered as a memory pathway.
Since any bounded continuous function on a compact domain can be approximated to arbitrary accuracy using a piece-wise constant function, and it is possible to use a memory pathway to approximate each of the constant function, with increasing number of distinct memory pathways, a feedforward SNN can achieve approximation of continuous functions with less error.
In this subsection, we show that with two structural designs: heterogeneous network i.e. a network having neurons with different dynamics and adding skip-layer connections, a feedforward SNN has the capability to achieve more distinct memory pathways.
Cutoff Frequency of a Memory Path We first show the correlation of cutoff period and spiking neuron parameters with Lemma 3, which is proved in Appendix D.
Lemma
A spiking neuron has cutoff period ω c
= τ m ln(
[Equation]
) above which input spike sequence cannot cause the spiking neuron to spike.
Remark From Lemma 3, it can be observed that the cutoff period ω c of a neuron can be configured to any positive real number by changing the neuron parameters and synaptic conductance G .
Further, with fixed G , ω c can be configured to any positive real number by changing the neuron parameters.
Neurons that are in cutoff change the spike propagation path in a network as they send no output spikes.
This creates different memory pathways without changing the connections in a network.
Heterogeneous Network If an mMND network has the same parameters for all neurons in each layer, the majority of the neurons are included in the same memory pathway, leading to the upper bound of number of distinct memory pathways to be limited.
With Lemma 4, we show the relationship between the upper bound of the number of distinct memory pathways and the number of different neuron dynamics in an mMND network.
Lemma
For an mMND network with m layers and { λ 1 , λ 2 , ...
λ m } number of different neuron dynamics in each layer, the least upper bound of the number of distinct memory pathways is (cid:81) mi
=
λ i .
Proof Sketch.
(Formal proof in Appendix D) For an mMND network, it is possible to have neurons with different ω c in each layer, which creates λ i number of different neuron activation states for layer i .
Across all network layers, the highest possible number of different neuron activation states is therefore the product of λ of each layer.
Since neurons in cutoff do not propagate spikes, they can be removed from a memory pathway.
This leads to (cid:81) mi =
λ i as the least upper bound of the number of memory pathways.
Compared to a network with homogeneous neuron parameters, in which the upper bound of number of distinct memory pathways is λ m , Lemma 4 indicates that heterogeneous network increases the maximum achievable number of distinct memory pathways in a feedforward SNN.
Skip-layer Connection We show that adding skip-layer connection increases the upper bound of the number of memory pathways in a network with Lemma 5.
Lemma
For an mMND network with m layers and { λ 1 , λ 2 , ...λ m } different neuron dynamics in each layer and a skip-layer connection made between layer l a and l b , s.t. a, b ∈ { 1 , 2 , ...m } and ( b − a ) > 1 , the least upper bound of the number of memory pathways is (cid:81) mi
=
λ i + ( (cid:81) ai =
1 λ i · (cid:81) mi
= b λ i )
[Figure]
Proof Sketch.
(Formal proof in Appendix D)
Compared to the network considered in Lemma 4, by adding a skip-layer connection, there are additional possible neuron activation states in the network that result from the cutoff of neurons in layers between l a and l b .
Without layers between l a and l b in the spike propagation path, the least upper bound of the number of memory pathways is increased by the maximum number of distinct memory pathways in a network that has layers { l 1 , l 2 , ..., l a , l b , l b +1 , ..., l m } connected sequentially.
SNN A RCHITECTURE D ESIGN USING A PPROXIMATION T HEORY
In this section, we discuss design of SNN architectures as inspired by the developed approximation theory for feedforward SNN, with more details in Appendix G.
Network Template for BPTT Training For BPTT training, the network template is shown in Figure 2(a).
The heterogeneity in neuron dynamics is implemented by using the multi-neuron-dynamic layers, which can either be convolutional or fully connected.
The multi-neuron-dynamic layers use different neuron parameters for spiking neurons in each neuron dynamic module, which contains a certain number of feature maps for convolutional layers or a certain number of neurons for fullyconnected layers.
There are two types of synapses between layers: transferred synapses marked as black dashed arrows and learned synapses marked as red solid arrows.
The conductance of learned synapses is optimized by the BPTT algorithm during training, and the transferred synapses have the same conductance as the learned synapses from the same pre-synaptic neuron.
During forward pass, neurons in each layer receive the same input features and respond differently based on their neuron dynamics to generate different output features.
During back-propagation, only conductance of the learned synapses are updated.
Skip-layer connection is implemented with the output spike matrix from source layer concatenated to the original input spike matrix of the target layer.
The skip-layer connection has the same implementation as the regular connection between consecutive layers, with both learned and transferred synapses (Figure 2(a)).
The last layer of the network is a fully-connected layer with homogeneous dynamic to generate prediction labels.
Network Template for STDP Learning For networks trained with STDP, the template is shown in Figure 2(b).
Each layer contains a learner module and a memory module.
Learner modules use homogeneous neuron dynamic that is suitable for STDP learning, and memory modules consist of neurons with different dynamics.
There are also two types of synapses: transferred synapses and learned synapses.
Between two layers, memory modules are connected with transferred synapses and memory modules are connected to learner modules with learned synapses.
Leaner modules be-
tween layers are not directly connected.
STDP training proceeds as a layer-by-layer process.
During training of the first layer, conductance of synapses connecting neurons in memory module to neurons in learner module is learned with STDP using all training data without supervision.
Then, the learned conductance is transferred to the transferred synapses in the same layer.
During training of the second layer, layer 1 memory module perceives the same input features and generates different output features with the heterogeneous neurons.
This lay-by-layer process is repeated until the layer before the final layer finishes learning.
The final linear layer is then fine-tuned using stochastic gradient descent (SGD) based on spike frequency array from the last multi-neuron-dynamic layer generated based on the labeled data.
Skip-layer connection is implemented by connecting the memory module of the source layer to the target layer.
The connections are made with the two types of synapses and follow the same training process as the consecutive layers.
Dual-search-space Bayesian Optimization Bayesian optimization uses Gaussian process to model the distribution of an objective function, and an acquisition function to decide points to evaluate.
For data points in a target dataset x ∈ X and the corresponding label y ∈ Y , an SNN with network structure V and neuron parameters W acts as a function f V , W ( x ) that maps input data x to predicted label ˜
y .
The optimization problem in this work is defined as
[Equation]
V contains the number of layers N layers , the number of memory dynamics N dynmaic and skiplayer connection configuration variables N skip , L start and L end , each controlling the number of skip-layer connections, the first layer and last layer to implement skip-layer connections.
All of the values are discrete.
W contains the values for a , τ m and R m in (1), which are continuous.
We separate the discrete and continuous search spaces by implementing a dual-search-space optimization process, where V is first optimized with fixed, manually tuned neuron parameters.
After an optimal structure is found, W are optimized for the selected V .
Details on the configurations of the optimization process are listed in the appendix.
To achieve Bayesian optimization with constraints, we implement a modified expected improvement (EI) acquisition function similar to the one shown by Gardner (Gardner et al., 2014), which uses a Gaussian process to model the feasibility indicator due to its high evaluation cost.
In this work, since the constraint function can be explicitly defined, we use a feasibility indicator that is directly evaluated.
The modified EI function is defined as:
[Equation]
where W is the network configuration containing W and V .
W
+ is the test point that provided the best result.
∆( W ) is the explicitly defined indicator function that takes the value of 1 when all constraints are satisfied and 0 otherwise.
E XPERIMENTS 5. E XPERIMENT S ETTINGS
Datasets tested in the experiment include the DVS Gesture (Amir et al., 2017), which is a human gesture dataset captured by DVS cameras, and the N-Caltech101 (Orchard et al., 2015), which is an event-based version of the Caltech101 dataset.
The proposed method is also tested for MLP-style
SNN on the sequential MNIST dataset presented row-by-row.
We also vary the amount of labeled data used during training ranging from using 100% labeled data for training down to 10% labeled data (30% for N-Caltech101) during training.
Note, during STDP training networks always use the entire but un-labeled training dataset; however, only the fraction of the labeled data is used for supervised fine-tuning of the last layer.
Comparison is made for DVS Gesture and N-Caltech101 with prior works including ConvLSNN, which is a combination of convolutional SNN and recurrent SNN with long and short-term neurons trained with BPTT (Salaj et al., 2020), DECOLLE (Kaiser et al., 2020), which uses surrogate gradient to train a convolutional feedforward SNN, HATS (Sironi et al., 2018), which implements time surfaces and SVM for classification and H-SNN (She et al., 2021) which uses STDP to train a convolutional SNN with two neuron dynamics.
Additional function approximation experiments are discussed in Appendix E.
5. E FFECT OF D UAL - SEARCH - SPACE B AYESIAN O PTIMIZATION
We compare the proposed dual-search-space Bayesian optimization with regular Bayesian optimization using a single search space for network validation error over 5 runs.
The result from the NCaltech101 dataset is shown in Figure 3.
It can be observed that the two optimization approaches achieve similar minimum validation error after convergence.
By separating the search spaces, the proposed optimization process reaches convergence faster than regular single-search-space optimization.
It is also worth noting that, between the two stages in the optimization process for BPTT training, the first stage accounts for more reduction in validation error than the second stage.
This indicates that optimizing network structure causes more impact to BPTT training than optimizing neuron parameters, which is potentially due to the reason that network structure more heavily affects the number of memory pathways in the network than neuron parameters.
On the other hand, for
STDP training where learning behavior is sensitive to the dynamic of spiking neurons, the reduction of validation error is more equally shared between the two optimization stages.
Over the 5 runs, among all network configurations achieved after the dual-search-space optimization converges, we compare the configuration with the lowest number of trainable parameters against baseline models.
The specific configurations for the optimized networks are listed in Table 1.
It can be observed that for BPTT algorithm, the optimized networks have more layers than the STDP trained networks [Table] and the optimal values found for neuron parameters are highly distinct for the two training methods.
[Table]
5. A BLATION S TUDIES
To investigate the effect of using multiple neuron dynamics
[Table]
we apply the same dual-search-space
Bayesian optimization process for networks that have homogeneous neuron dynamic for the same
[Table] [Table]
number of evaluations as the proposed design.
Similarly, to study the contribution to performance gain from skip-layer connections, the Bayesian optimization process is used for network templates without skip-layer connections.
The optimization process runs for the same number of evaluations as the proposed design.
From the results shown in Table 2, it can be observed that compared to baselines, the proposed networks achieve the best accuracy for all datasets.
Specifically, when homogeneous network is used, the performance of STDP trained network is noticeably lower than the proposed method for DVS Gesture and N-Caltech101.
For BPTT training, using heterogeneous network and skip-layer connection show different level of benefit for each dataset.
For sequential MNIST which has less complexity, the improvement from using heterogeneous neurons and skiplayer connections is not as significant.
5. C OMPARISON WITH P RIOR W ORKS
DVS Gesture As shown in Table 3 (top), with 100% labels, the proposed network trained with BPTT demonstrates higher accuracy than all tested networks without using the most trainable parameters.
The proposed network trained with STDP has slightly lower accuracy than some baselines when 100% labels are used; for reduced-label training it outperforms all other networks.
N-Caltech101 As shown in Table 3 (bottom), the proposed network trained with BPTT outperforms all baselines with both 70% and 100% training labels and also has less trainable parameters than most baselines.
The un-supervised learning models i.e., H-SNN and the proposed network with STDP, show considerably lower performance (more than what was observed for DVS Gesture) than supervised ones when 100% labels are available; However, the proposed network with STDP shows better performance than H-SNN, and outperforms all networks when available labels are below 50%.
C ONCLUSION
We develop a theoretical basis to understand and optimize the ability of a feedforward SNN to approximate temporal sequence.
We analytically show how heterogeneity and skip-layer connections can improve sequence approximation with SNN, and empirically demonstrate their impact on spatiotemporal learning.
It is well-known in neuroscience that, heterogeneity (De Kloet & Reul, 1987) and irregular connectivity (Eickhoff et al., 2018) are intrinsic properties of human brains.
Our analysis shows that incorporating such concepts within artificial SNN is beneficial for designing high-performance SNN for classification of spatiotemporal data. A CKNOWLEDGMENTS
This material is based on work sponsored by the Army Research Office and was accomplished under Grant Number W911NF-19-1-0447.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government.
Authors would like to thank Drs.
Mark Mclean, Chris Purdy, Fernando Camacho and James Keiser from Laboratory for Physical Sciences for many helpful discussions on this work
A SNN D YNAMICS
Remark For a sequentially connected neuron list with m neurons all with γ = 1 and neuron delay t nd , an input spike at time t leads the neuron list to generate an output spike at time t + mt nd
Remark For any input sequence with period t in to a spiking neuron with response rate γ such that γ > 1 , if refractory period is set to r < t in , the neuron can exit refractory period before the next spike arrives.
Lemma
For any input spike sequence with period t in in range [ T min , T max ] , there exists a spiking neuron n with fixed parameters v th , v reset , a , R m and τ m , such that by changing synaptic conductance G , it is possible to set the neuron response rate γ n to be any positive integer.
Proof.
For a given input spike sequence period t in , consider the maximum possible membrane potential decay that can be reached within a period of t in .
From (1), when I = 0 , dvdt < 0 and | dvdt | increases with higher v .
Hence, the maximum decay of v is reached when initial membrane potential v ( t = 0) → v − th and the neuron decays for period t in = T max .
The decayed membrane potential v ( t = T max ) can be derived by solving the differential equation (1) for v ( t )
= v th at t = 0 :
[Equation]
[Equation]
It is possible to have a spiking neuron with R m , a and τ m such that ∆ v , defined as
[Equation]
, tends to zero.
With this configuration, since the the highest possible decay of membrane potential is negligible, for any target γ , it is possible to set G such that
[Equation]
The proof is complete.
Note, input sequence that has no spike has t in tending to infinity, thus violating the bounded constraint mentioned in Lemma 1.
B P ROOF OF T HEOREM
Theorem 1 For any input and target output spike sequence pair with periods ( t in , t out ) ∈
[ T min , T max ] ×
[ T min , T max ] , there exists a minimal-layer-size network with skip-layer connections that has memory pathway with output spike period function P ( t in ) such that | P ( t in ) − t out | < ϵ .
Proof.
For any given ( t in , t out ) , first consider the condition where t in > t out .
It is possible to construct a minimal-layer-size network N connecting m spiking neurons with neuron response rate γ = 1 sequentially, denoted as a m -tuple of neurons { n 1 , n 2 , ..., n m } .
Since any configuration of skip-layer connection with source layer and target layer pair ( l s , l t ) , such that l s ∈ [1 , m − 2] , and l t ∈ [ l s + 2 , m ] , can be added, it is possible to add a ( m − 2) -tuple of skip-layer connections
[Equation]
Denote the synaptic conductance for all the skip-layer connections as a ( m − 2) -tuple
[Equation]
For any t out < t in , it is possible to find a k -tuple of synaptic conductance
[Equation]
Set synaptic conductance in S G sl \ S ′ G sl to 0.
Then set the conductance of synapse connecting n m −and n m to 0.
In such way, The output spikes from network N have period
[Equation]
For given ϵ , it is possible to choose t nd such that t nd < 2 ϵ , therefore satisfying | P ( t in ) − t out | <
ϵ .
m can be chosen as
[Equation]
12) or equivalently:
[Equation]
[Equation]
Since T max − T min 2 ϵ is finite, m is finite.
For t in < t out , using N as described above, it is possible to achieve output spike with period within ϵ of any period in (0 , t in ] .
For a given t out , assume the configuration in neuron list N has output spike interval t ′ int such that kt ′ int
=
t out , where k is a positive integer.
From Lemma 1, it is possible to set G for a neuron n m +1 such that its neuron response delay satisfies γ n m +
= k for input spike period t ′ int .
A new network, denoted as N ′ , can be formed by connecting n m +1 to the output of
N .
N ′ has output spike with period P ( t in ) =
kt ′ int = t out .
Hence, to reach the given ϵ , it requires neuron list N to have output spike interval t int such that
[Equation]
Since k is finite, (14) can be achieved.
For t in > =
t out , it is possible to configure network N ′ such that t int satisfies | t int − t out | <
ϵ , and the value of γ n m +1 set to 1, hence | P ( t in ) − t out | < ϵ can be achieved.
The proof is complete.
C P ROOF OF L EMMA
Lemma 2 With no skip-layer connection, there does not exist a minimal-layer-size network that has output spike period function P ( t in ) such that for any input and target output spike sequence pair with periods ( t in , t out ) ∈
[ T min , T max ] ×
[ T min , T max ] , | P ( t in ) − t out | <
ϵ .
Proof.
A minimal-layer-size network N with m layers can be denoted as a m -tuple of neurons { n 1 , n 2 , ..., n m } connected sequentially.
Since no skip-layer connection exists, there is only one distinct memory pathway that contains all neurons { n 1 , n 2 , ..., n m } .
Denote the set of neuron response rate corresponding to each neuron in N as
[Equation]
[Equation]
For a given input sequence with t in , denote the timing of the first spike as t in 1 , consider the output spike sequence for network with
[Equation]
The first output spike from N has timing ˜ t out= t in 1 + mt nd , and the second output spike has timing ˜ t out= t in 1 + t in + mt nd .
It can be easily derived that the period of the output spike sequence is
[Equation]
Also consider the output spike sequence for network with γ j = 2 for any j ∈ { 1 , 2 , 3 , 4 , ..., m } and γ i = 1 ∀ i ∈ ( { 1 , 2 , 3 , 4 , ..., m } \ { j } ) (18)
Following the same process, the period of the output spike sequence is
[Equation]
Since the smallest increase to any γ i is by 1, there is no set of values for γ such that the network output spike sequence has period P ( t in ) satisfying t in < P ( t in ) <
2 t in .
Since within the range ( t in , 2 t in ) , there exists values of t out such that | P ( t in ) − t out | <
ϵ does not hold.
The proof is complete.
D M EMORY P ATHWAYS IN SNN
In this section we analyze the increase to the least upper bound of the number of memory pathways in a network by using heterogeneous networks and skip-layer connections.
Lemma
A spiking neuron has cutoff period ω c
= τ m ln(
[Equation]
) above which input spike sequence cannot cause the spiking neuron to spike.
Proof.
Consider (2), since membrane potential increases at time t i and decays otherwise, solving for t =
t i and the equation can be expanded:
[Equation]
For input with frequency f , t i +1 − t i =
∆ t = 1 f , subtracting membrane potential values at two consecutive t i provides:
[Equation]
setting time of the first input spike t 1 to zero leads to:
[Equation]
As e tiτm > 0 , and the term (( e ∆ t τm ) − 1)( v reset − a )+ R m τ m Ge ∆ t τm ) does not depend on t i , the polarity of ∆ v m does not change with time.
v m is either strictly increasing, staying the same or decreasing with higher t i .
This indicates that, when ∆ v m ≤ 0 the post-synaptic neuron can never spike regardless of how many pre-synaptic spikes it receives.
∆ v m ≤ 0 when input spike period t in satisfies
[Equation]
[Equation]
Therefore, the cutoff period of the neuron is
[Equation]
[Equation]
The proof is complete.
In the following proof, we consider cutoff frequency, f c = 1 ω c of spiking neurons.
From (24) it can be easily derived that f c can be configured to any positive real number by changing the neuron parameters.
Lemma
For an mMND network with m layers and { λ 1 , λ 2 , ...
λ m } number of different neuron dynamics in each layer, the least upper bound of the number of memory pathways is (cid:81) mi
=
λ i .
Proof.
Denote the set of neurons in layer l with distinct neuron dynamics as
[Equation]
Since the network is mMND, | S ln | = λ l .
Denote the set of cutoff frequency corresponding to each neuron in S ln as
[Equation]
Since neurons in S l n can have different neuron parameters, from Lemma 3, it is possible to set the parameters such that all entries of F lc are distinct.
Hence, there exists a permutation π such that
[Equation]
Denote the input spike frequency to layer l as f lin , neuron n li is a part of a valid memory pathway in the network if
[Equation]
Therefore, for input spike frequency f lin ≥ f n lπ ( λl ) c , all neurons in S ln can be part of a valid memory pathway of the network.
For input spike frequency such that f n lπ ( i ) c ≤ f lin < f n lπ ( i +1) c , i neurons can be part of a valid memory pathway of the network.
For any input to the network f in ∈ [ F min , F max ] , denote the number of ways different neurons in
S ln can be part of valid memory pathways as k l .
The total number of distinct memory pathways K in the network is the product of k l of all layers:
[Equation]
Since for all layers, 0 ≤ k l ≤ λ l ,
[Equation]
Consider that, for any layer l ∈ { 1 , 2 , 3 , ..., m } , the input f lin is bounded by [ F lmin , F lmax ] .
k l = λ l can be achieved by setting f n lπ (1) c and f n lπ ( λl ) c
such that:
[Equation]
Hence the bound is tight for (30).
The proof is complete.
Lemma
For an mMND network with m layers and { λ 1 , λ 2 , ...λ m } different neuron dynamics in each layer and a skip-layer connection made between layer l a and l b , s.t. a, b ∈ { 1 , 2 , ...m } and ( b − a ) > 1 , the least upper bound of the number of memory pathways is (cid:81) mi
=
λ i +( (cid:81) ai =
1 λ i · (cid:81) mi
= b λ i )
Proof.
Denote the mMND network with skip-layer connection between layer l a and layer l b as P .
Denote the set of all neurons in P as
[Equation]
where n 1 i is a neuron in layer l 1 , and n 2 i is a neuron in layer l 2 , etc.
The activation state o n ji of a neuron can be denoted with binary values 0 and 1 with o n ji = 1 representing that n ji receives input frequency that is above its cutoff frequency f n ji c .
The set of all possible neuron activation states O in P that generates non-zero network output feature vector can be partitioned into two subsets denoted as O A and O B .
Set O A contains all states where the input frequency f kin to any layer l k such that a < k < b satisfies
[Equation]
Set O B contains all the remaining neuron activation states in O , where all layers receive input frequency higher than cutoff frequency of at least one neuron in that layer.
For all the states in O A , no spike signal is sent from layer b − 1 to layer b , since at least one layer between l a and l b generates no output.
Hence, output from P is not affected if connections between layer l i and l i +1 , such that i ∈ { a, a + 1 , ..., b − 1 } , are removed.
Network P is therefore equivalent to network P ′ that has layers { l 1 , l 2 , ..., l a , l b , l b +1 , ..., l m } connected sequentially.
According to Lemma 4, it can be derived that the least upper bound of the number of distinct memory pathways in P ′ is (cid:81) ai =1 λ i · (cid:81) mi
= b λ i .
Hence, for all states in set O A , the least upper bound of the number of distinct memory pathways in
P is also (cid:81) ai =
1 λ i · (cid:81) mi
= b λ i .
For all states in set O B , since the activation of neurons in the source layer of the skip-layer connection is already accounted for when considering layer l a , the least upper bound of the number of distinct memory pathways is the same as network P that has no skip-layer connection, which is (cid:81) mi
=1 λ i according to Lemma 4.
For the set of memory pathways from states in O A , denoted as M A , and the set of memory pathways from states in O B , denoted as M B , the number of memory pathways of network P is | M A ∪ M B | .
From Lemma 5, the bound is tight for | M A | ≤ (cid:81) ai =1 λ i · (cid:81) mi
= b λ i and for | M B | ≤ (cid:81) mi =
λ i .
It also satisfies that M A ∩ M B = ∅ since all elements in M A have ( m − ( b − a − 1)) layers and all elements in M B have m layers.
Hence the bound is tight for [Equation]
The proof is complete.
E T IME - VARYING F UNCTION A PPROXIMATION
A time-varying function f ( t ) can be approximated using piece-wise constant function, such as illustrated in Figure 4.
It is therefore possible to approximate the time-varying function with a feedforward SNN by approximating each of the constant function with a memory pathway.
In this section, we test the approximation capability of feedforward SNN for time-varying functions using this principle.
The target functions to approximate have the form of:
[Equation]
[Equation]
Here, x is variable; n and m are function parameters.
For discrete-time simulation of the network, we approximate the target function with [Equation]
[Figure] where t in is the input spike period and t out is the output spike period.
We test approximation performance of a small-scale feedforward SNN with 6 fully-connected layers, skip-layer connections { (2 , 5) , (3 , 5) } and 4 neuron dynamics.
The network is trained with BPTT to minimize mean squared error (MSE) loss between the spike period of network output t ′ out and target spike period t out .
To evaluate the network’s approximation performance, we construct 6 networks with the same structure as discussed above, and change each to have different trainable parameter numbers by scaling the layer size.
For baseline comparison, networks with 6 layers, no skip-layer connection and using homogeneous neurons, configured to have the same trainable parameter numbers as the proposed networks, are tested.
All networks are trained with BPTT.
The loss function measures the difference between the network output spike trains and the target spike trains with MSE.
Two sets of function parameters: ( m = 1 , n = 3 . 3) and ( m = 2 , n = 2 . 1) are tested for the target functions f ( x ) on domain [3 , 10] .
The resulting MSE losses for different network scales are shown at the bottom of Figure 5.
It can be observed that the proposed networks can approximate target functions with less error than the baseline networks at all network scales.
The smallest tested networks have relatively high losses while performance increases quickly with more trainable parameters.
The rate of performance improvement decreases when trainable parameter number is above 4000.
To understand the impact of the target function parameters to approximation performance of SNN, we test baseline and the proposed network with 4167 trainable parameters for different pairs of n 2. m -2. 1.8e0.7eBaseline Network n 2. m -2.
3.4e1.8eProposed Network (a) (b) n m
L o ss ( l o g ) -6. function parameters m and n . The resulting MSE loss is shown with a heat map in Figure 6.
It can be observed that for all m and n value pairs, the proposed network achieves lower loss than the baseline network.
Another observation is that, there is no clear correlation between the value of m and approximation error.
On the other hand, for higher values of n , the approximation error generally increases for both baseline and the proposed network.
F N ETWORK S PIKING A CTIVITY
To investigate neuron spiking activity in the proposed network, for the function approximation task as described in Appendix E, with parameters ( m = 1 , n = 2 . 3) and a randomly selected approximation point f ( x = 19) , we plot the timing of neuron spikes at the last layer over training epochs in
Figure 7 (a).
It can be observed that the network initially generates spikes at widely distributed timings.
After the first 50 training epochs spike timing starts to converge and remains stable at around 200 epochs.
The final output spike period is t ′ out = 48 , which matches the target output period.
Next, we investigate spiking activity of the BPTT trained network as described in Section 5.4 for
N-Caltech101 classification task.
Three different test data points are presented to the network, and the spikes from neurons in the first depth of layer 8 and neurons in the final layer, are recorded and shown in Figure 7 (b).
Here, (i), (iii) and (v) are from layer 8; (ii), (iv) and (vi) are from the final layer.
(i) and (ii) are from the observation of a data point with class label “5”; (iii) and (iv) are from the observation of another data point also with class label “5”; (iii) and (iv) are from the observation of a data point with class label “1”.
It can be observed that neurons in layer 8 exhibit similar activity in (i) and (iii) as the network is presented with data points from the same class.
(ii) and (iv) show that most spiking activities are [Figure] from the neuron with index 5, indicating correct prediction for the two data points.
When a data point from a different class is presented, spiking activity in (v) shows different patterns than those in (i) and (iii).
In the final layer, neuron with index 1 generates the most spikes, leading to a correct prediction.
However, for this data point, neuron with index 58 is also generating a considerable number of spikes, indicating that the network is more likely to mis-classify this data point, compared to the other two tested data points.
G I MPLEMENTATION OF H ETEROGENEOUS C ONV -SNN
Multi-neuron-dynamic (MND) networks with convolutional layers can be considered a scaled-up version of the mMND network, where each neuron dynamic now contains a matrix of neurons that receive input features from different spatial locations.
To implement heterogeneous Conv-SNN, first consider a regular Conv-SNN layer with no heterogeneity.
The spiking neuron matrix has dimension { W, H, D } , where D is the depth of the layer.
The convolution filter has dimension { C, w, h, D } where C is the number of input channels and D is the number of output channels.
Based on this, a heterogeneous layer l can be constructed, by concatenating heterogeneous neuron matrices with the same W and H along layer depth.
The resulting spiking neuron matrix has dimension { W, H, λD } , where λ is the number of neuron dynamics.
Hence, layer depth i ∈ { k + 1 , k + 2 , k + 3 , ..., k + D } have the same neuron parameters, where 0 ≤ k ≤ λ − 1 is an integer representing the index of neuron dynamics.
The convolution filter has the same dimension as the regular Conv-SNN layer: { C, w, h, D } , which is shared by layer depth with different k values.
During forward pass of this layer, a convolution operation is applied to generate an input signal matrix with dimension { W, H, D } .
For each value of k , neurons in depth { k +1 , k +2 , k +3 , ..., k + d } are simulated based on the neuron parameters for such neuron dynamic index k using the signal matrix as input.
For the next convolutional layer l ′ receiving input from layer l , the filter dimension is { λD, w ′ , h ′ , D ′ } with λD input channels.
H D ETAILS ON THE B AYESIAN O PTIMIZATION P ROCESS
During the first stage of the dual-search-space optimization process, the parameters to optimize include: N layer , L start , L end , N skip , N dynamic , all of which are positive integers.
Specifically,
N layer is the number of convolutional layers.
For skip-layer connection, there are three configuration parameters to optimize: starting layer L start , which is the source layer of the first skip-layer connection; ending layer L end , which is the target layer of the last skip-layer connection; skip-layer connection number N skip , which defines how many connections to implement.
The source layer of the N skip skip-layer connections are placed evenly between L start and L end , each with skip length equal to
[Equation]
In the case of
[Equation]
, the value of L end is reduced to the maximum value that satisfies
[Equation]
For heterogeneity, the number of different dynamic N dynamic in all layers is optimized jointly.
The constraints for the parameters are defined as:
[Equation] [Equation]
and
[Equation]
The manually configured neuron parameters are, τ m = 100 and R m = 300 for all neuron dynamics, and a ∈ [ − 30 , − 5] is distributed evenly for each neuron dynamic.
Figure 8 shows visualization of the learned distribution from the Bayesian optimization process.
Here, mapping from the search space of skip-layer connection configurations to the validation accuracy of N-Caltech101 is shown.
The highest validation accuracy is reached at L start = 2 and L end = 11 .
Due to the exponential increase of search space with the number of neuron dynamics in each layer, it is highly inefficient to search for every neuron parameters of each dynamic.
In the second stage of the optimization process, we choose to apply Bayesian optimization for the parameter a of each neuron dynamic separately, while τ m and R m are optimized jointly with the same values shared by all neuron dynamics.
a values are taken to the precision of 10 0 , and τ m and R m values are taken to the precision of 10 1 .
The constraints are a ∈ [ − 30 , − 5] , τ m ∈ [50 , 200] and R m ∈ [200 , 400] .
The value of t nd for all networks is set to 1.
The parameters of each optimized network are shown in
Table 1.
Note, the skip-layer connections are listed as source and target layer pairs.