EGRU: Event-based GRU for activity-sparse inference and learning
Anonymous Author(s)
AffiliationAddress email
Abstract
The scalability of recurrent neural networks (RNNs) is hindered by the sequentialdependence of each time step’s computation on the previous time step’s output.
Therefore, one way to speed up and scale RNNs is to reduce the computationrequired at each time step independent of model size and task.
In this paper, wepropose a model that reformulates Gated Recurrent Units (GRU) as an event-basedactivity-sparse model that we call the Event-based GRU (EGRU), where unitscompute updates only on receipt of input events (event-based) from other units.
When combined with having only a small fraction of the units active at a time(activity-sparse), this model has the potential to be vastly more compute efficientthan current RNNs.
Notably, activity-sparsity in our model also translates into sparseparameter updates during gradient descent, extending this compute efficiency tothe training phase.
We show that the EGRU demonstrates competitive performancecompared to state-of-the-art recurrent network models in real-world tasks, includinglanguage modeling while maintaining high activity sparsity naturally duringinference and training.
This sets the stage for the next generation of recurrentnetworks that are scalable and more suitable for novel neuromorphic hardware.
Introduction
Large scale models such as GPT-3 [8], switch transformers [17] and DALL-E [53] have demonstratedthat scaling up deep learning models to billions of parameters cannot just improve the performanceof these models but lead to entirely new forms of generalisation.
For example, GPT-3 can dobasic translation and addition even though it was trained only on next word prediction.
While it isunknown if scaling up recurrent neural networks can lead to similar forms of generalisation, thelimitations on scaling them up preclude studying this possibility.
The dependence of each time step’scomputation on the previous time step’s output is the source of a significant computational bottleneck,preventing RNNs from scaling well.
Therefore, in recent years, RNNs, despite their many desirabletheoretical properties [15] such as the ability to process much longer context and their computationalpower [58, 61], have been supplanted by feedforward network architectures.
By reducing the computation required at each time step, independent of model size and task, we canspeed up and better scale RNNs.
We propose to do this by designing a general-purpose event-basedrecurrent network architecture that is naturally activity-sparse.
Dubbed the Event-based Gated
Recurrent Unit (EGRU), our model is an extension of the Gated Recurrent Unit (GRU) [12].
Withevent-based communication, units in the model can decide when to send updates to other units, whichthen trigger the update of receiving units.
Therefore, network updates are only performed at specific,dynamically determined event times.
With activity-sparsity, most units do not send updates to otherunits most of the time, leading to substantial computational savings during training and inference.
We formulate the gradient updates of the network to be sparse using a novel method, extending thebenefit of the computational savings to training time.
The biological brain, which relies heavily on recurrent architectures and is at the same time extremelyenergy efficient [44], is a major source of inspiration for the EGRU.
One of the brain’s strategiesto reach these high levels of efficiency is activity-sparsity.
In the brain, (asynchronous) event-basedcommunication is just the result of the properties of the specific physical and biological substrate onwhich the brain is built.
Biologically realistic spiking neural networks and neuromorphic hardware alsoaim to use these principles to build energy-efficient software and hardware models [54, 59].
However,despite progress in recent years, their task performance has been relatively limited for real-world taskscompared to state-of-the-art recurrent architectures based on LSTM and GRU.
We view the EGRU asa generalisation of spiking neural networks, moving away from modeling biological dynamics towarda more general-purpose recurrent model for deep learning.
In this paper, we first introduce a version of EGRU based on a principled mathematical approach that formulates the dynamics of the internal states of the network in continuous time.
The units of the networkcommunicate solely through message events triggered when the internal state of a unit reaches a threshold value.
This allows us to derive exact gradient descent update equations for the network analogousto backpropagation-through-time (BPTT) that mirrors the activity-sparsity of the forward pass.
We then introduce a discrete simplification of this continuous-time model that is also event-basedand activity-sparse while being easier to implement on today’s prevailing machine learning librariesand thus directly comparable to existing implementations of GRU and LSTM.
The backwards passhere uses an approximate version of BPTT, and these updates are also sparse.
The sparsity of the backward-pass overcomes one of the major roadblocks in using large recurrentmodels, which is having enough computational resources to train them.
We demonstrate the taskperformance and activity sparsity of the model implemented in PyTorch, but this formulation willalso allow the model to run efficiently on off-the-shelf hardware, including CPU-based nodeswhen implemented using appropriate software paradigms.
Moreover, an implementation on novelneuromorphic hardware like [13, 28], that is geared towards event-based computation, can make themodel orders of magnitude more energy efficient [49].
In summary, the main contributions of this paper are the following:
We introduce the EGRU, an event-based continuous-time variant of the GRU model.2.
We derive an event-based form of the error-back-propagation algorithm for EGRU.
We introduce a discrete-time version of EGRU that can be directly compared to currentLSTM/GRU implementations.
We demonstrate that the EGRU exhibits task-performance competitive with state-of-the-art
recurrent network architectures (based on LSTM, GRU) on real-world machine learningbenchmarks.
We show that EGRU exhibits high levels of activity-sparsity during both inference andlearning.
Related work
Activity sparsity in RNNs has been proposed previously in various forms [29, 47, 48], but onlyfocusing on achieving it during inference.
Conditional computation is a form of activity sparsityused in [17] to scale to 1 trillion parameters.
This architecture is based on the feedforward transformerarchitecture, with a separate network making the decision of which sub-networks should be active [60].
An asynchronous event-based architecture was recently proposed specifically targeted towards graphneural networks [57].
QRNNs [7], SRUs[39] and IndRNNs [40] target increasing the parallelism in arecurrent network without directly using activity-sparsity.
Unlike [17], our architecture uses a unit-localdecision making process for the dynamic activity-sparsity, specifically for recurrent architecture.
Thecost of computation is lower in an EGRU compared to [48], and can be implemented to have parallelcomputation of intermediate updates between events, while also being activity sparse in its output.
Models based on sparse communication [65] for scalability have been proposed recently forfeedforward networks, using locality sensitivity hashing to dynamically choose downstream unitsfor communicating activations.
This is a dynamic form of parameter-sparsity [25].
Parameter/modelsparsity [26] is, in general, orthogonal to and complementary with our method for activity-sparsity,and can easily be combined for additional gains.
Biologically realistic spiking networks [42] are often implemented using event-based updates and havebeen scaled to huge sizes [34], albeit without any task-related performance evaluation.
Models fordeep learning with recurrent spiking networks [3, 56] mostly focus on modeling biologically realisticmemory and learning mechanisms.
Moreover, units in a spiking neural network implement dynamicsbased on biology and communicate solely through unitary events, while units in an EGRU sendreal-valued signals to other units, and have more general dynamics.
A sparse learning rule was recentlyproposed [4] that is a local approximation of backpropagation through time, but not event-based.
The event-based learning rule for the continuous time EGRU is inspired by, and a generalization of,the event-prop learning rule for spiking neurons [64].
As in that paper, we use the adjoint methodfor ordinary differential equations (ODEs) to train the continuous time EGRU [10, 51] combined withsensitivity analysis for hybrid discrete/continuous systems [11, 19].
Using pseudo-derivatives for backpropagating through the non-differential threshold function, as we use for our discrete-time EGRU, wasoriginally proposed for feedforward spiking networks in neuromorphic hardware in [16] and developedfurther in [3, 66].
The sparsity of learning with BPTT when using appropriate pseudo-derivativesin a discrete-time feed-forward spiking neural network was recently described in [50].
A continuous time version of sigmoidal RNNs was first proposed in [2] and for GRUs in [14].
Thelatter used a Bayesian update for network states when input events were received, but the network itselfwas not event-based.
As in [38, 47], the focus there was on modeling irregularly spaced input data,and not on event-based network simulation or activity-sparse inference and training.
[9] also recentlyproposed a continuous time recurrent network for more stable learning, without event-based mechanics.
GRUs were formulated in continuous time in [33], but purely for analyzing its autonomous dynamics.
Event-based GRU
3.1 Time-
[Figure]
We base our model on the GRU [12], illustrated for convenience in Fig.
1A. It consists of internalgating variables for updates ( u ) and a reset ( r ), that determine the behavior of the internal state y .
The state variable z determines the interaction between external input x and the internal state.
Thedynamics of a layer of GRU units, at time step t , is given by the set of vector-valued update equations:
[Equation]
where W u/r/z , b u/r/z denote network weights and biases, ⊙ denotes the element-wise (Hadamard)product, and σ ( · ) is the vectorized sigmoid function.
The notation (cid:2) x ⟨ t ⟩ , y ⟨ t − 1 ⟩ (cid:3) denotes vector concatenation.
The function g ( · ) is an element-wise nonlinearity (typically the hyperbolic tangent function).
We introduce an event generating mechanisms by augmenting the GRU with a rectifier and a clearingmechanism (see Fig.
1B for an illustration).
This introduces an event-based variant of the internalstate variable y ⟨ t ⟩ i , that is nonzero when the internal dynamics reach a threshold
ϑ i and is clearedimmediately afterwards.
Formally, this can be included in the model by adding an auxiliary internalstate c ⟨ t ⟩ i , and replacing y ⟨ t ⟩ =( y ⟨ t ⟩ 1 ,y ⟨ t ⟩ 2 ,... ) with the event-based form
[Equation]
where H ( · ) is the Heaviside step function and ϑ i > 0 is a trainable threshold parameter.
This formis well suited for time sparsity, since H ( · ) acts here as a gating mechanism, by generating a singlenon-zero output when c ⟨ t ⟩ i crosses the threshold ϑ
i .
That is, at all time steps t with c ⟨ t ⟩ i <
ϑ i , ∀ i , wehave y ⟨ t ⟩ i =0 .
The − y ⟨ t − 1 ⟩ i term in Eq.
2 makes emission of multiple consecutive events by the sameunit unlikely, hence favoring overall sparse activity.
With this formulation, each unit only needs tobe updated when an input is received either externally or from another unit in the network.
This isbecause, if both x ⟨ t ⟩ i =
y ⟨ t ⟩ i =0 for the i -th unit, then u ⟨ t ⟩ i ,r ⟨ t ⟩ i ,z ⟨ t ⟩ i are essentially constants, and hencethe update for y ⟨ t ⟩ i can be retroactively calculated efficiently on the next incoming event.
3.2 Limit to continuous time
The discrete time model Eq.
(1) considers the GRU dynamics only at integer time points,t 0 =0 ,t 1 =1 ,t 2 =2 ,... .
However, in general it is possible to express the GRU dynamics for an arbitrarytime step ∆ t , with t n = t n − 1 +∆ t .
The discrete time GRU dynamics can be intuitively interpretedas an Euler discretization of an ordinary differential equation (ODE) [33] (see Supplement), whichwe extend further to formulate the EGRU.
This is equivalent to taking the continuous time limit ∆ t → 0to get dynamics for the internal state c ( t ) starting from the discrete time EGRU model outlined above.
In the resulting dynamical system equations inputs cause changes to the states only at the event times,whereas the dynamics between events can be expressed through ODEs.
To arrive at the continuoustime formulation we introduce the neuronal activations a u ( t ) , a r ( t ) and a z ( t ) , with
[Equation]
with dynamics
[Equation]
and
[Equation]
(4) where τ s and τ m are time constants, c ( t ) , u ( t ) and z ( t ) are the continuous time analogues to c ⟨ t ⟩ ,u ⟨ t ⟩ and z ⟨ t ⟩ , and ˙ a X denotes the time derivative of a X .
The boundary conditions are defined for t =0as a X (0)= c (0)= 0 .
The function F in Eq.
(4) determines the behavior of the EGRU between eventtimes, i.e. when x ( t )=
and y ( t )= 0 .
Nonzero external inputs and internal events cause jumps inc ( t ) and a X ( t ) .
Furthermore, the formulation of the event generating mechanisms Eq.(2) introduced above can beexpressed in a straightforward manner in continuous time.
Note that in continuous time the exact times at which the internal variable c i ( s ) reaches the threshold ( c i ( s )
=
ϑ i ) can be determined with veryhigh precision.
Therefore, the value of c i ( s ) and the instantaneous amplitude of y i ( s ) simultaneouslyapproach ϑ i at time point s , so that the − y i term in Eq.
2 effectively resets c i ( s ) to zero, right afteran event was triggered.
To describe these dynamics we introduce the set of internal events e , e k ∈ e ,e k = ( s k ,n k ) , where s k are the continuous (real-valued) event times, and n k denotes which unit gotactivated.
An event e k is triggered whenever c n k ( t ) reaches ϑ .
More precisely:
[Equation]
[Equation]
where the superscript .
− ( . + ) denotes the quantity just before (after) the event.
Immediately after anevent has been generated the internal state is cleared: c + n k ( s k )=0 .
At the time of this event, the activations of all the units m ̸ = n k connected to unit n k experiences a jumpin its state value.
The jump for a X ,m is given by:
[Equation]
6) where X ∈ { u, r, z } , r X = {+
+} when X ∈ { u,z } and r X = r when
X = { r } .
This is equivalent to y i =
c − n kbeing the output of each network unit.
A similar jump is experienced on arrival of an external input,using the appropriate input weights instead (see Supplement for specifics).
The continuous time event-based state update is illustrated in Fig.
1C and D for the delay-copy taskdescribed in Section 4.1.
Two EGRU units are used here and states c i ( t ) and event times s k are shown.
At the beginning of the trial an input pattern ( x 1 = 1 , x 2 = 0 , and x 1 = 0 , x 2 = 1 in Fig.
1C and D,respectively) has to be memorized in the network and retrieved again after the recall cue ( x 3 = 1 )was given.
The parameters are trained with the event-based updates described in Section 3.3.
Therequired memory is stored in the internal events and state dynamics.
State updates can be performedin an event-based fashion, i.e. by jumping from one event time s k to the next s k +1 .
In-between statevalues follow the state dynamics Eq.(4) and their values are not needed to perform the updates (butare shown here for the sake of illustration).
By construction, the state updates for external and internalevents only happen on receipt of event.
Since Eqs.
3, 4 are linear ODEs, the intermediate updates dueto autonomous state dynamics can also be performed cumulatively and efficiently just at event times,hence avoiding any computation in the absence of incoming events.
3.3 Event-based gradient-descent using adjoint method
To derive the event-based gradient updates for the EGRU we define the loss over duration T as 177 (cid:82) T 0 ℓ c ( c ( t ) ,t ) dt , where ℓ c ( c ( t ) ,t ) is the instantaneous loss at time t .
T is a task-specific time durationwithin which the training samples are given to the network as events, and the outputs are read out.
Ingeneral ℓ c ( c ( t ) ,t ) may depend arbitrarily on c ( t ) , however in practice we choose the instantaneous lossto depend on the EGRU states only at specific output times to adhere to our fully event-based algorithm.
The loss is augmented with the terms containing the Lagrange multipliers λ c , λ a X to add constraintsdefining the dynamics of the system from Eqs.
The total loss L thus reads
[Equation]
The Lagrange multipliers are referred to as the adjoint variables in this context, and may be chosenfreely since both τ m ˙ c ( t ) − F ( t, a u , a r , a x , c ) and τ s ˙ a X + a X are everywhere zero by construction.
We can choose dynamics and jumps at events for the adjoint variables in such a way that they canbe used to calculate the gradient d L dw ji .
Calculating the partial derivatives taking into account thediscontinuous jumps at event times depends on the local application of the implicit function theorem,which requires event times to be a differentiable function of the parameters.
See [10, 19, 64] for adescription of applying the adjoint method for hybrid discrete/continuous time systems with furthertheoretical background, and the Supplement for a derivation specific to the EGRU.
The time dynamics of the adjoint variables is given by the following equations with a boundarycondition of λ c ( T )
[Equation]
[Equation]
for X ∈{ u,r,z } , and M T denoting the transpose of the matrix M .
The event updates for the adjointsare described in the Supplement.
In practice, the integration of λ is done backwards in time.
For the recurrent weights w X ,ij from the different parameter matrices W X for X ∈ u,r,z , we can writethe weight updates using only quantities calculated at events e k as:
[Equation]
The corresponding value of ξ X ,ijk =( ξ X ,k ) ij is given by the following formula, written in vector formfor succinctness:
[Equation]
where ⊗ is the outer product, c − refers to the value of c ( t ) just before event e k , r − X =0 for X ∈{ u,z }and equal to the value of r ( t ) just before event e k for X = { r } , λ + a X refers to the value of the adjointvariable λ a X ( t ) just after the event e k .
Thus, the values of r ( t ) , c ( t ) needs to be stored only at eventtimes, and λ a X ( t ) needs to be calculated only at these times, making the gradient updates event-based.
See the Supplement for the update rules for the input weights and biases.
3.4 Sparse approximate BPTT in discrete time
[Figure]
Indiscretetime, thenetworkusesathresholdactivationfunction H ( c ) todecidewhethertoemitaneventas described in Eq. 2.
Since H ( c ) is not differentiable at the threshold ϑ i , we define a pseudo-derivativeat that point for calculating the backpropagated gradients.
The pseudo-derivative is defined as a piecewise linear function that is non-zero for values of state c i between ϑ i + ε
and ϑ i − ε as shown in the insetin Fig.
2B. Since the pseudo-derivative is zero whenever the internal state of the unit is below ϑ i − ε , thebackpropagated gradients are also 0 for all such units, making the backward-pass sparse (see Fig.2 foran illustration).
Note that the case where the internal unit state is above ϑ
i + ε tends to occur less often,since the unit will emit an event and the internal state will be cleared (Eq. 2) at the next simulation step.
3.5 Computation and memory reduction due to sparsity
For the forward pass of the discrete time EGRU, an activity sparsity of α (i.e. an average of α eventsper simulation step) leads to the reduction of multiply-accumulate operations (MAC), by factor α .
We focus on MAC operations, since they are by far the most expensive compute operation in thesemodels.
If optimally implemented an activity sparsity of 80% will require 80% fewer MAC operationscompared to a GRU that is not activity-sparse.
Computation related to external input is only performedat input times, and hence is as sparse as the input, both in time and space.
During the backward pass, asimilar factor of computational reduction is observed, based on the backward-pass sparsity β which is,in general, less than α .
This is because, when the internal state value is not within ± ε of the thresholdϑ , the backward pass is skipped, as described in section 3.4.
Since our backward pass is also sparse, weexpect to need to store only β fraction of the activations for later use, hence also reducing the memoryusage.
In all our experiments, we report activity-sparsity values calculated through simulations.
Results4.1 Delay-copy task
To illustrate the behavior of the continuous-time EGRU model (Fig.
1C,D) we used a simple delay-copytask (also called the copy memory task [24]).
A binary vector was presented to the network at the inputtime.
This was followed by a delay period, after which the network was given a cue input indicatingthat it should recall the input seen before.
A small network with only two EGRU units was used here,trained with the event-based learning rules described in Section 3.3.
Right after the cue input, thenetwork had to report the memorized input pattern.
EGRU outputs y i emitted at network event times were convolved with an exponential kernel to retrieve output traces, which were then used to retrievethe stored binary patterns based on their relative magnitudes.
The kernel time constant was chosento be significantly lower than the delay time such that the network had to retain the memory in theevent dynamics.
The binary cross-entropy loss was used to train this model until it reached perfect(100%) bitwise accuracy on this task.
Fig.
1C,D shows the dynamics of the continuous-time modelafter training, as well as the output trace and events.
The network has learned to generate events suchthat output traces reliably encode the stored input patterns.
[Figure]
We next evaluate our model on gesture prediction, which is a popular real-world benchmark for RNNs.
Here and in the remainder of the experiments we used the discrete time version of EGRU, since it iseasier to implement and use while retaining most of the advantages of the continuous time model.
Weuse the DVS128 Gesture Dataset [1], where the inputs are defined as events.
This dataset is widely usedin neuromorphic research and enables us to demonstrate our model’s performance and computationalefficiency on event-based data.
The dataset contains 11 gestures from 29 subjects recorded with a
DVS128 event camera [41].
Each event encodes a relative change of illumination and is given asspatio-temporal coordinates of X/Y position on the 128 × 128 -pixel sensor and time stamp.
Raw eventtimes were combined into ‘frames’ by binning them over time windows of 25 ms. Frames were thendownscaled to 32 × 32 pixels using a maxpool layer reference architecture paraeffective accuactivity backward (# units) meters
MAC racy sparsity sparsity
He et al.
LSTM (512)
[Table]
Innocenti et al.
AlexNet
[Table]
DA
[Table]
ours
[Table] [Table]
ours
[Table] [Table]
ours
[Table]
ours
[Table]
Unlikepreviousapproachesthatfocusedonafeedforward/RNNhybridapproach[1,20,31], wefocusedon pure RNN based architectures following the work of [23].
A binary cross-entropy loss was appliedwith an additional regularization loss on the output gate to produce 5% activity and the state variable c tobe slightly below the threshold.
The models were trained using the Adam optimizer for 1000 epochs to verify their stability but typically reached a plateau performance after 200 epochs (see Supplement forfurther details).
Due to this, backward sparsity as described in Section 3.5 was calculated at epoch 100.
Comparison of model performance on gesture prediction is presented in Table 1.
The model hadinherent activity-sparsity of 70% which the regularization increased to 90% without significantperformance decrease.
EGRU consistently outperformed GRU networks of the same size on this taskby a small margin.
Adding data augmentation (DA) by applying random crop, translation, and rotation,as previously done in [31], increased the performance to over 97% of this pure RNN architecture,coming close to state-of-the-art architectures that even include costly AlexNet pre-processing.
4.3 Sequential MNIST
Next, we tested the EGRU on the sequential MNIST task [37], which is a widely used benchmarkfor recurrent networks.
In this task, the MNIST handwritten digits were given as input one pixel ata time, and at the end of the input sequence, the network output was used to classify the digit.
Wetrained a 1-layer EGRU with 512 units using Adam optimizer for 200 epochs.
In Table 2, we reportthe results of discrete-time EGRU along with other state-of-the-art architectures.
EGRU achieveda task performance comparable to previous architectures while using much fewer operations (morethan an 8-fold reduction in effective MAC operations compared to GRU).
[Table]
4.4 Language Modeling
Natural language processing is a popular domain for benchmarking recurrent neural networks.
Weevaluated our model on a language modeling task based on the PennTreebank [43] dataset to validatethe functionality of our model.
While techniques such as neural cache models [21] or dynamicevaluation [36] have been shown to improve language models, we focused on the RNN model itself inthis work, taking [46] as our baseline.
Following [46], our models consisted of a dense 400-dimensionalembedding layer, and three stacked RNN cells with DropConnect applied to the hidden-to-hiddenweights [62].
Dimensions of the internal states were chosen to match the number of trainable parametersof [46].
The weights of the final softmax layer were tied to the embedding layer [30, 52].
All our modelsare optimized with Adam for 800 epochs, and parameters were tuned for each model individually.
Details on training and model parameters can be found in the Supplement.
Results are shown in Table 3.
In our experiments, GRUs did not reach the performance of LSTM variants on this task, which, tothe best of our knowledge, is consistent with recent RNN language modeling literature [45, 46].
Atthe same time, EGRU slightly outperformed GRU, while maintaining high levels of activity sparsity.reference architecture paraeffective validation test activity (# units) meters MAC sparsity Gal et al. Variational LSTM 24M77.
Melis et al. [45] 1 layer LSTM 24M61. Merity et al. AWD-LSTM 24M 24M 60.
- ours
GRU (1350)
24M 24M 75.
- ours
EGRU (1350)
24M 5.4M 73.
85.7% ours
EGRU (2700) 76M
[Table]
Discussion
We have introduced EGRU, a new form of a recurrent neural network that is competitive with currentdeep recurrent models yet can efficiently perform both inference and learning.
To achieve this, wefirst formulated the GRU in continuous time and converted it to an event-based form that achievedactivity-sparsity naturally.
Furthermore, the gradient-descent updates on this time-continuous EGRUmirrored the activity sparsity of the inference.
We then demonstrated a discrete-time simplificationof this model that also exhibited event-based activity-sparse inference and learning while being easierto implement with popular ML frameworks such as PyTorch or Tensorflow.
The EGRU achieved competitive task performance on various real-world tasks such as gesturerecognition and language modeling while achieving a sparsity of up to 80% for the gesture recognitiontask and 90% on the sMNIST and language modeling task.
Scaling up networks for language modelinghas shown some of the most promising results in the last few years [8, 17] Hence our choice of task,albeit on a smaller scale, was to validate the functionality of the model.
Considering the need forextensive hyperparameter search [45] for language modeling, our model achieved promising resultswhile maintaining a high degree of activity-sparsity.
For example, our EGRU with 1350 hidden unitsreached perplexities comparable with LSTM and GRU, while maintaining an activity-sparsity of86 % ( 14% of the units active on average).
The amount of computation used by an EGRU also scalessub-linearly with an increase in the size of the network and number of parameters, making it a scalablealternative to LSTM/
GRU based architectures (see Supplement).
While we use the GRU as the basis for our model due to its simplicity, this formulation can easily be extended to any arbitrary network dynamics, including the LSTM, allowing specialized architectures fordifferentdomains.
Theadjointmethodforhybridsystemsthatweusehereisapowerfulgeneral-purposetool for training event-based activity-sparse forms of various recurrent neural network architectures.
Another novel outcome of this paper is that this theory can handle inputs in continuous time as events,which is very intuitive, hence providing an alternative to the more complex controlled differentialequations [35].
The EGRU can also be used for irregularly spaced sequential data quite naturally.
The compute efficiency of this model can directly translate into gains in energy efficiency whenimplemented using event-based software primitives.
These same properties would also allow themodel to work well on heterogenous compute resources, including pure CPU nodes, and neuromorphicdevices such as Intel’s Loihi [13] and SpiNNaker 2 [28], that can achieve orders of magnitude higherenergy efficiency.
The EGRU model will also perform well in more mainstream deep learninghardware that is enabled for dynamic sparsity, such as the Graphcore system [32].
On neuromorphicdevices with on-chip memory in the form of a crossbar array, the activity sparsity directly translatesinto energy efficiency.
For larger models that need off-chip memory, activity-sparsity needs to becombined with parameter-sparsity to reduce energy-intensive memory access operations.
In summary, starting with the motivation of building scalable, energy-efficient deep recurrent models,we demonstrated the EGRU, which reduces the required compute for both inference and learning byenhancing sparsity in the network.
This approach lays the foundation for exploring novel capabilitiesthat can emerge from scaling up RNNs similar to what has been seen for feed-forward architecturesin recent years.
Potential negative societal impact: The proposed model is a new variant of the previously published
GRU and would therefore essentially inherit all potential negative societal impacts from that model,including the potential risks that come with automated surveillance systems, vulnerability to fraud andadversarial attacks, etc.
(see [6] and [63] for critical reviews).
However, the model also provides thepotential societal benefit of making these models more energy-efficient and thus reducing the energyand carbon footprint of machine learning.
Scaling this model to larger sizes, especially for languagemodeling, can lead to the same problems as current large language models [5].
The effect of activitysparsity on prediction bias needs to be studied further in the same way as for parameter sparsity [27].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See end of Section 1 for a summary of main claims.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] The authors have read and approved the ethics guideline
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
and the Supplement
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3 and the
Supplement
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimen-
tal results (either in the supplemental material or as a URL)?
[Yes]
In the Supplement
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes] Some details in the main text and further details in Supplement
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes]
In the Supplement
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
In the Supplement
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review Board [Table]
(IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]