B RAIN INSIGHTS IMPROVE RNN S ’ ACCURACY AND ROBUSTNESS FOR HIERARCHICAL CONTROL OF CONTINUALLY LEARNED AUTONOMOUS MOTOR MOTIFS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the problem of learning dynamics that can produce hierarchically organized continuous outputs consisting of the ﬂexible chaining of re-usable motor motifs from which complex behavior is generated.
Can a motif library be efﬁciently and extendably learned without interference between motifs, and can these motifs be chained in arbitrary orders without ﬁrst learning the corresponding motif transitions during training?
This requires (i) parameter updates while learning a new motif that do not interfere with the parameters used for the previously acquired ones; and (ii) successful motif generation when starting from the network states reached at the end of any of the other motifs, even if these states were not present during training (a case of out-of-distribution generalization).
We meet the ﬁrst requirement by designing recurrent neural networks (RNNs) with speciﬁc architectures that segregate motif-dependent parameters (as customary in continual learning works), and try a standard method to address the second by training with random initial states.
We ﬁnd that these standard RNNs are very unreliable during zero-shot transfer to motif chaining.
We then use insights from the motor thalamocortical circuit, featuring a speciﬁc module that shapes motif transitions.
We develop a method to constrain the RNNs to function similarly to the thalamocortical circuit during motif transitions, while preserving the large expressivity afforded by gradient-based training of non-analytically tractable RNNs.
We then show that this thalamocortical inductive bias not only acts in synergy with gradient-descent
RNN training to improve accuracy during in-training-distribution motif production, but also leads to zero-shot transfer to new motif chains with no performance cost.
Besides proposing an efﬁcient, robust and ﬂexible RNN architecture, our results shed new light on the function of motor preparation in the brain.
I NTRODUCTION AND RELATION TO OTHER WORKS
Animals have the remarkable ability to efﬁciently learn and compose elaborate continuous behaviors, which often relies on the ﬂexible chaining of ‘motifs’ - reproducible bouts of behavior - in response to hierarchical commands (Zimnik & Churchland, 2021; Geddes et al., 2018; Merel et al., 2019b).
The mechanisms behind this are however not fully understood, and engineering controllers for dynamical systems that could perform such complex and structured continuous behaviors in the real world has been a long-standing challenge of robotics (Brooks, 1986;
Prescott et al., 1999; Merel et al., 2019b).
Such tasks involve two different computational operations: ﬁrst, at coarse temporal intervals, the ﬂexible selection of discrete and abstract action commands by a ‘high-level controller’; and, second, the transmission of each abstract command to a ‘lower-level controller’ that continuously produces a corresponding command - a ‘motif’ - for the motor effector.
Many lines of work have used artiﬁcial neural networks (ANNs) to solve the ﬁrst - discrete - computational operation (notably using tools from deep reinforcement learning, e.g. (Merel et al., 2019a;
Frans et al., 2018; Dennis et al., 2020)) and have leveraged the ability of ANNs to perform well when using a rich training set that includes many motif sequences (Frans et al., 2018; Dennis et al., 2020; OpenAI et al., 2021; Xu et al., 2020).
However, to the best of our knowledge, in the context of the second computational operation, there is a gap in the literature about how to design ANN mechanisms that enable zero-shot transfer to performing new continuous motif sequences from a library of independently learned motifs.
Here we will focus on this latter task, which notably highlights a remarkable human skill, as we can - for instance - learn to pronounce a new word and then immediately include it in arbitrary sentences.
While this biological relevance has led the neuroscience community to start investigating related questions, this literature has focused on a hand-designed ‘bottom-up’ approach.
This approach studies how networks can function in spite of strong constraints introduced e.g. to mimic the brain activity patterns or connectivity, and/or to ensure analytical tractability (Kao et al., 2020; Logiaco et al., 2019; Sussillo et al., 2015; Zimnik & Churchland, 2021; Ijspeert et al., 2013; Kulvicius et al., 2012).
Importantly, this approach cannot determine whether speciﬁc network features are more generally advantageous for solving a task - instead, with this bottom-up approach, these chosen features could reﬂect constraints that arose through random evolutionary idiosyncracies, or could become irrelevant for networks that cannot be designed through analytical insights but that can still be successfully trained with modern machine learning algorithms.
Overcoming these limitations has implications for engineering and can provide new neuroscientiﬁc insight.
To address this knowledge gap, here, we use Recurrent Neural Networks (RNNs) whose computational power is not arbitrarily constrained.
An RNN is a type of ANN which is a generic dynamical system, and which therefore naturally ﬁts the desired characteristics of motor outputs.
Consequently,
ANNs are indeed regularly used for tasks requiring the production of continuous outputs, including in the context of robotics (Wyffels & Schrauwen, 2009; Sussillo & Abbott, 2009; Tani, 2003; Liu et al., 2019; Merel et al., 2019a; Maheswaranathan et al., 2019).
We will examine the ability of
RNNs to (i) independently learn motor motifs in order to to build a continuously expandable motif library; and (ii) ﬂexibly chain motifs in arbitrary orders (Fig. 1a and see Appendix A.1 for a formal deﬁnition).
In order to better dissect the mechanisms by which RNNs can fail or succeed at this task, we focus on the purest form of motor control through dynamics: the production of trajectories without needing the external anchor of a time-dependent sensory input.
We refer to this as autonomous control, which is especially relevant when sensory input is too unreliable (Yeo et al., 2016; Shenoy et al., 2013; Brembs, 2021), but also in other cases such as the above-mentioned speech production.
We will show that while it is possible to engineer RNNs to independently, extendably and efﬁciently learn to produce single motifs in response to discrete input commands, these RNNs are limited in their generalization ability during improvisation of motif sequences.
We will then use insights from the mammalian thalamocortical motor system - notably the presence of a motor preparation phase before each motif (Zimnik & Churchland, 2021; Nashef et al., 2021; Logiaco et al., 2019).
We will show that weaving in these insights into performance-optimized RNNs leads to both improved motif production accuracy (through a positive synergy with single motif gradient-descent training) and excellent robustness during generalization to motif sequencing.
T ASK AND ARCHITECTURE DESIGN
Here, we study the ability of RNNs to fulﬁll the ﬁrst requirement of our task: the ability to acquire an extendable library of autonomous motifs (Fig. 1a, left column).
However, before describing these analyses, we want to clarify how we chose the motifs that the RNNs have to learn.
First, we chose motifs of long durations – on the order of a thousand timesteps – so that they strongly leverage the above-mentioned autonomous capabilities of RNNs.
Second, because we are interested in assessing the relative expressive power of the RNNs we study, we have designed two different types of motor motifs so that they constitute two ‘difﬁculty’ levels for the RNNs.
Following recent analytical approaches for studying RNN dynamics (Schuessler et al., 2020b;a; Logiaco et al., 2019), we characterize the difﬁculty of motifs as the number of certain basis functions - complex exponentials that act similarly as different frequencies of a Fourier transform - needed to approximate a motif well through linear combination.
Therefore, we deﬁne a set of oscillatory motifs that are relatively easy to produce, and a set of more difﬁcult ‘step’ motifs (Fig.
1d,f-g, see Appendix A.2 for the full list of motifs used in this paper).
We train motifs using gradient descent – speciﬁcally, ADAM (Kingma &
Ba, 2015)).
Our objective function is the mean square error between desired and actual output.
We will now study how RNNs can meet the requirement to learn a new motif without ‘catastrophic interference’ with the memory of previously learned motifs - which relates to the literature on continual learning (Kirkpatrick et al., 2017; Parisi et al., 2019; Yoon et al., 2020; Farajtabar et al., 2019). This line of work emphasizes avoiding interference between gradient updates used to train a network on many sub-tasks, while promoting the re-use of neural resources across sub-tasks (so that the network makes efﬁcient use of its parameters).
We will use an ‘architectural’ approach to
[Figure]
b c g ad / g 0cn g
mu / g 0cn g 0ptb g
o ptb
=1.
M i n i m u m r oo t m e a n s q u a r e e rr o r
A dd i t i v e a r c h
i t e c t u r e
M u l t
i p l i c a t i v e a r c h
i t e c t u r e
M i n i m u m r oo t m e a n s
[Figure]
a
[Figure]
e
[Figure]
o r
[Figure]
continual learning that consists in segregating tuned parameters across the different motifs, because it will later facilitate the weaving of biological insights in our networks. With this approach, interference is fully prevented during sequential learning; but the proposed architectures may not be efﬁcient, which is what we investigate below.
To do so, we will compare these architectures to a ‘standard’ RNN with no segregation of tuned parameters.
This RNN (‘control architecture’) (i) is fully-tuned, (ii) receives a static input b µ to instruct the motif µ (Maheswaranathan et al., 2019; Sussillo & Abbott, 2009; Tani, 2003), and (iii) is trained in a plausible noise-robust regime imposed by a random initialization of its state (see A.6.3).
As expected given that this RNN has no protection against catastrophic interference (Kirkpatrick et al., 2017), we indeed ﬁnd that when learning motifs sequentially, this RNN immediately forgets previously learned motifs when learning a new one (not shown) – even though it is possible to learn motifs by simultaneously including all of them in each training batch (Fig. 1c, and see below).
This ‘control architecture’ obeys the following dynamics:
[Equation]
The output y = w (cid:124) tanh
( x ) = w (cid:124) r is produced through the vector w that is initialized from a centered Gaussian distribution with std 1 / √ N cn – where here N cn = 50 is the number of recurrent units of this control network, so that its outputs have appropriate maximal positive or negative magnitude scaling as √ N cn .
Also, the recurrent interactions weights J are initialized with iid elements taken from a standard Gaussian as previous work has shown that when choosing a gain g cn 0 > 1 this leads to rich dynamical regimes appropriate for complex computations (Sompolinsky et al., 1988; Sussillo & Abbott, 2009; Sussillo et al., 2015; Schuessler et al., 2020b).
We performed hyperparameter tuning on g 0 .
The dynamics are discretized using Euler’s integration method where dt = 0 .
1 τ .
[Table]
We now consider modiﬁcations of this control RNN that address this issue of catastrophic forgetting by segregating the parameters involved in learning different motifs while sharing common computational resources across motifs (Parisi et al., 2019; Merel et al., 2019a;b).
Here, we will share (i) ﬁxed readout weights (with the above-mentioned centered Gaussian distribution with std 1 / √ N as this is sufﬁcient to ensure the successful production of our motifs); and (ii) the recurrent weights as they can set rich ‘baseline dynamics’ that can be modulated by some motif-speciﬁc weights.
We globally adjust the shared recurrent weights through tuning their above-mentioned gain hyperparameter g (a similar alternative could be to pre-tune these recurrent weights to an original set of motifs and to then freeze them, but - as we will see - our chosen approach leads to good results).
First, we consider an ‘additive’ architecture (Fig.
1b, top).
Here, each motif µ is produced in response to learning the input vector b µ , leading to the following dynamics for the network activities:
[Equation]
where the input acts as a motif-speciﬁc controller of the dynamics.
Note that this occurs because the gradient of the loss with respect to the input weights propagates through the recurrent connections - whereas if different outputs weights would be learned for different motifs (Jaeger, 2007), the dynamics would not be affected by learning.
Second, we consider a ‘multiplicative’ architecture (Fig.
1b, bottom), that is inspired by both previous machine learning literature (Sutskever et al., 2011; Schuessler et al., 2020b) and the anatomy of the brain’s motor system (Guo et al., 2017; Logiaco et al., 2019).
Here, each motif µ is produced in response to both a learned input vector b µ and a learned rank-one perturbation of the connectivity u µ v (cid:124) µ .
The latter is equivalent to a loop through an instantaneous ‘unit’ receiving input from the recurrent network through the weights v µ and feeding back through the weights u µ .
Here, this motif-speciﬁc loop participates to modulating the dynamics (Logiaco et al., 2019), in a way that yields more computational ﬂexibility compared to networks with random feedback weights (Susman et al., 2021).
Interestingly, learning the full recurrent weights in randomly initialized RNNs can yield a low-rank weight update (Schuessler et al., 2020b).
Therefore, by imposing that the motif-speciﬁc learning is restricted to a low-rank weight perturbation, we expect to get close to full-weight learning while enabling segregation of the learned weights per motif.
The dynamics are:
[Equation]
where g mu and J are deﬁned as for the additive network, and u µ and v µ are each learned and initialized iid from a centered Gaussian with std g ptb o / √ N (i.e. expected norm g ptb o ).
To test the baseline relative accuracies of the additive, multiplicative and control networks, we trained them on all the possible chains of motifs of length two (excluding repetitions of the same motif) for the ten step motifs (Fig. 1c left).
Consistent with the continual learning aspect of our task – which values limiting the number of parameters that need to be tuned and stored per motif – we equalized the number of tunable parameters across architectures (Collins et al., 2017).
Interestingly, after optimizing over hyperparameters (i.e., g ad , g mu , g ptb o , and g cn 0 ), we found that all networks had similar accuracy (Fig. 1c, reminiscent of (Collins et al., 2017)).
This suggests that our strategy of segregating tunable parameters in the additive and multiplicative networks do not lead to drastic decrease in per-tuned-parameter expressivity, while by construction preventing interference when learning motifs sequentially (while the control network suffers from forgetting of previously learned motifs when learning new ones as expected, not shown).
On the other hand, we note that the multiplicative architecture, which is closer to models constrained to mimic brain dynamics and architecture (Logiaco et al., 2019), appears to have similar accuracy as the additive network while requiring fewer neurons.
This echoes recent results suggesting that more biologically-plausible object-recognition ANNs tend to have architectures that require fewer neurons (Nayebi et al., 2021).
We now turn to investigate the robustness of these different architectures when improvising motif sequences after being trained on single motifs – a training strategy which, by construction, enables learning new motifs without interference for the additive and multiplicative architectures.
B RITTLENESS OF STANDARD RNN S DURING GENERALIZATION
Here, we ask whether RNNs trained on single motifs can produce arbitrary chains of motifs – a form of zero-shot transfer where the RNN’s internal state differs between training and testing due to the
RNN’s memory over several timesteps (a similar network mechanism as in (Lake & Baroni, 2017)).
We evaluate a standard technique used to promote both generalization and noise-robustness in
ANNs, that relies on leveraging randomness (e.g. (Vezhnevets et al., 2017; Liu & Hodgins, 2017;
Merel et al., 2019a))
during training.
Each motif is trained in isolation but with initial network activities selected randomly according to a distribution that approximates the activities at the ends of all other motifs (Fig. 1e).
In this scenario, if we can approximate the end-of-motifs distribution well enough and if training succeeds, then all transitions should work with no transition-speciﬁc training.
Unfortunately, the end-of-motifs distribution is unknown, is training-dependent, and could be heterogeneous across motifs.
However, since we are using networks with large N , Gaussian weights, and a tanh nonlinearity, it is known that an uncorrelated Gaussian vector can in some cases wellapproximate this unknown distribution (Landau & Sompolinsky, 2018).
Indeed, we observed that a standard normal is a good choice for the marginal statistics (Fig.
2a and b, panels (ii)).
Hence, we tried training the RNNs to generate the step motifs one at a time when starting from random initial network activities x init drawn from a standard normal distribution.
We set our hyperparameters to the optima from Fig.
1c ( g ad = 1 . 4 , g mu = 1 .
and g ptb o = 1 . 5 ) and otherwise trained as above.
For all architectures, training was successful (Fig.
2a and b panel (i), left; Fig.
2c left).
Additive
[Figure]
a t i
[Figure]
i
[Figure] Step motifs are always used except in c Additive architecture multiplicative architec , compared to the desired motif shown in color) when initial conditions x init are randomly drawn from a standard Gaussian as during training vs. transitioning
We then asked our networks to chain two motifs (the ﬁrst one starting from random initial activities as in training), and found that the performance of the second motif was substantially degraded compared to the same motif produced when starting from random (Fig.
2a,b panels (i) and (iii); see also Fig.
1g right).
Each transition from a given ﬁrst motif leads to a relatively reproducible output for the second motif because the network variability is low at the end of the ﬁrst motif.
However the second motif’s output can vary substantially from the target when following certain ﬁrst motifs.
Moreover, the inaccuracies are not restricted to the start of the second motif; rather, the whole output is either clearly shifted or even more grossly wrong.
Note that these issues are also present in the control architectures and are therefore not caused by our strategy of segregating the tuned parameters per motif to tackle catastrophic interference (Fig.
2c, Appendix A.3.5).
Interestingly, the shape of the marginal distribution of the elements of x at the end of a given motif µ ( x µ end ) was not a good predictor of whether transitions from this motif would lead to worse performance.
For instance, in the case of the additive network, the transition from motif 1 to motif 2 leads to poor performance for motif 2 (Fig.
2a (i), top right), even though the marginal distribution of x 1 end at the end of motif 1 appears extremely similar to a standard Gaussian (Fig.
2a (ii), red curve).
Conversely the distribution of x 5 end at the end of motif 5 is wider than the standard Gaussian (Fig.
2a (ii), purple curve), but the transition from motif 5 to motif 2 leads to a more accurate output (Fig.
2a (i), second chain from top right).
This strongly suggests that between-unit correlations impact transition success, which complicates sampling the various x µ end when initializing motifs during training.
Figs.
2a,b panels (ii) show that the standard deviation of the marginal distribution of the values of x µ end slightly exceeds 1 for some motifs.
To ensure that the transition failures we observed when initializing with the standard normal were not due to this mismatch in scale, we retrained our additive network with initial x values sampled from a Gaussian with std of 1.2 (Fig.
2a panels (iv) and (v)).
We observe that the end-of-motif distributions still have standard deviations around 1 for all motifs (Fig.
2a panel (iv)).
Despite training with a wider distribution, performance impairment after transitioning still occurred with even slightly greater errors compared to training with a standard normal (Fig.
2a panel (iv) Fig.
2c dashed cyan line).
In addition, we veriﬁed that the decreased performance during zero-shot transfer was not a consequence of using our relatively difﬁcult step motifs for which accuracy is limited even when starting from random.
Using our oscillatory motifs instead (as in Fig.
1g), we see that performance when starting from random improves signiﬁcantly, but zero-shot transfer is again unsuccessful with relatively large second motif errors during motif chaining (the light blue line in Fig.
2c, notice that the increase of error during zero-shot transfer is of similar amount as for the step motifs).
Finally, we also investigated whether larger networks would somehow be more regularized in a way that would favor zero-shot transfer performance.
However, increasing the network size from
N = 300 to N = 1000 did not improve the performance during zero-shot transfer; instead, as expected, it only improved accuracy when the network was initialized from random as in training (Fig. 2c, cyan line vs. dark blue line).
The failure to robustly transition between motifs when training on single motifs that are randomly initialized is a particular instance of an out-of-distribution generalization limitation, a general feature of ANNs (Russin et al., 2019; D’Amour et al., 2020).
Not only is the shape of the true marginal distribution of the values of x µ end at the end of a given motif not exactly matched to the statistics of the Gaussian distribution used during training, but we also neglect the correlations and other higher order interactions between the values of x µ end . As these complex x µ end statistics are shaped during training, there is a self-consistency issue between initialization during training and resulting x
Note that trying to avoid this issue by including all transitions between motifs in the training set (Fig. 1c, left) would scale quadratically with the number of motifs and is thus ultimately prohibitive. Further, even including a few motifs transitions in the training set would disrupt the sequential learning strategy that prevents the training set to grow over time in a continual learning setting.
Indeed, on the one hand, sequential training would require motif parameters to only be trained for transitioning from previously learned motifs - which would compromise the robustness of the motifs that were learned earlier, especially if the motifs’ properties change over the ’life’ of the agent. On the other hand, re-training earlier-learned motifs to improve their accuracy when they are initialized from later-learned motifs causes the training set to grow with the number of learned motifs, and runs the risk of modifying the network’s end-of-motif activity for these re-trained motifs, which could in turn affect the transitioning from these motifs to any others.
Another naive attempt to solve the transition issue would be to train all motifs from a ﬁxed arbitrary network state, and to implement a hard reset to this state before each motif.
However, this strategy would introduce detours at transitions in case of discrepancy between two motif’s end and start points – and would more generally prevent the RNN’s dynamics to use the implicit information about the previous motif’s shape encoded into its internal state in order to implement a smooth transition to the next motif. As a side note, training single motifs initialized with a reset from a noiseless ﬁxed state would not be noise robust (Fig. 16a), while using a noisy reset actually corresponds to testing and training our networks on random initial conditions – and we will show below that, besides introducing detours at transitions, this naive strategy leads to impaired learning and poorer performance compared to our biologically-inspired solution.
To address these issues, we will now leverage insights from the brain’s motor network.
This is promising because human behavior demonstrates that the brain has acquired inductive biases that enable zero-shot motor sequence improvisation with agile transitions between motifs.
B RAIN - INSPIRED INDUCTIVE BIASES IMPROVE ACCURACY & ROBUSTNESS
Recordings from the brain’s motor cortex during ﬂexible sequencing indicate that a period of motor preparation precedes every motif in a sequence (Zimnik & Churchland, 2021).
This preparation leads the activity to quickly converge towards a motif-speciﬁc pattern (Lara et al., 2018).
This motor preparation process involves interactions between the recurrent motor cortex and thalamus (Nashef et al., 2021)
– a brain region whose neurons quickly respond to cortical input and then projects back to cortex, forming feedback loops that effectively act as perturbations of the effective cortical connectivity (Kao et al., 2020; Logiaco et al., 2019).
In a computationally limited linear regime, previous work has shown that this preparatory perturbation can be linked to the optimal control solution to bring the cortical activity towards the unique motif-speciﬁc stable state instructed by an external input provided to cortex (Kao et al., 2020; Logiaco et al., 2019).
However, the dynamics of the continuously nonlinear RNNs that we and modern ML practitioners consider (Maheswaranathan et al., 2019; Wyffels & Schrauwen, 2009; Sussillo & Abbott, 2009; Tani, 2003; Liu et al., 2019;
Merel et al., 2019a) are richer and qualitatively different from this linear regime, and do not allow setting the parameters through analytics as done in the former neuroscience works (Kao et al., 2020;
Logiaco et al., 2019).
Indeed, in this limited linear regime, (i) single motifs could not be produced in response to a simple static input (Logiaco et al., 2019) – whereas the RNNs can (Fig. 2); (ii) spontaneous activity decays to zero, whereas units are spontaneously active in our networks (Fig.
3b, ‘prior learning’ curves, (Sompolinsky et al., 1988)); and (iii) a single stable ﬁxed point exists, while continuously nonlinear RNNs can exhibit multistability (Maheswaranathan et al., 2019).
Therefore, it is unclear whether such a preparatory connectivity perturbation can be successfully trained with gradient descent to shape a convergence phase in the more powerful performance-optimized continuously nonlinear RNNs.
Also, to draw conclusions about the functionality and applicability of the preparatory connectivity perturbation, it is critical to determine whether it can act in synergy with gradient descent training of the motif-speciﬁc parameters in the continuously nonlinear RNNs.
To answer these questions, we ﬁrst design a training protocol for a preparatory connectivity perturbation (that we call ‘preparatory module’, Fig. 3a left).
This training is done once and for all before learning any motif; details are in A.5.1 and we summarize the procedure below.
We want to train the weights U prep V (cid:124) prep that can shape fast convergence to zero activity in the absence of input in nonlinear RNNs.
To do so, we initialized the network activities with standard random initial x values and trained using the cost function (cid:80) t (cid:107) r ( t ) (cid:107)
(Fig. 3a), while the dynamics of our networks obey:
[Equation]
where g x is g ad or g mu for the additive and multiplicative networks respectively.
We initialized the weights U prep and V prep with centered Gaussian with std (cid:113) 0 .
05 / √ P ∗ N , with P = 50 (though smaller values of P give similar results as long as P/N is more than a few percent).
Then, with the thalamic preparatory modules in hand (whose weights are then ﬁxed), we trained additive and multiplicative networks on individual motifs with the network activities x randomly initialized with a standard Gaussian, but starting each motif with a motif preparation period now also involving a motif-speciﬁc input (Fig. 3b).
For the additive network, our only modiﬁcation from before is that we included the thalamic preparatory module in the network dynamics for the ﬁrst 5 τ :
[Equation]
[Figure]
A dd i t i ve
M u l t .
d c a ... min ∫ ||tanh(x)||
W x x tanh(x) ( g J + + − = τ
U prep V T prep ) { 2 dt prior learning
[Figure]
prep
[Figure]
prep
[Figure]
learning
U prep & V prep
Time (number of τ s )
[Figure]
| t anh ( x ) ||
[Figure]
l t
[Figure]
τ
[Figure] [Figure]
e
[Figure]
RNNs thalamocortical insights a ) Training a preparatory module consisting of P loops weights forming a perturbation U prep V (cid:124) prep of the connectivity, for a strongly nonlinear network.
b ) Training the motif-speciﬁc parameters in the additive, multiplicative and two-input architectures.
c ) Average root mean square error over motifs during training conditions vs. during transitioning, in the additive and multiplicative architectures; and with or without preparatory module.
Ten step motifs are used.
d ) Example network performance when initializing the network state with the same random distribution as during training, as opposed to when transitioning from another motif.
e )
Same as c but comparing additive and two-inputs architectures using more motif transitions.
Stars indicate a signiﬁcant error increase (signed rank test).
For the multiplicative network, to make a more direct comparison with the thalamocortical model (Appendix A.4), we set the dynamics such that the input b µ and loop u µ v (cid:124) µ were only active during the preparatory and post-preparatory periods respectively:
[Equation]
First, the RNNs trained with a preparatory module are much more accurate even during the production of single motifs initialized with the same distribution as used during training, especially for the more difﬁcult step motifs ( ≈ 50% , Fig.
3c and d, left).
This strongly suggests that, when training with random initialization, the imposition of fast network dynamics by the preparatory module allows the tuned preparatory input to more efﬁciently steer the dynamics to explore and ﬁnd a motif-speciﬁc network state that leads to more accurate motif production (Kemeth et al., 2021). Accordingly, we ﬁnd that the preparatory module speeds up learning (see A.6.4).
After training, the preparatory module also helps converging quickly to the correct motif-speciﬁc network state.
Conversely, without the preparatory module, the motif-speciﬁc parameters appear to struggle to support accurate motif production on their own (create fast dynamics at motif start, shape the pattern towards which activity must converge, and modulate the shared RNN dynamics during motif production).
Second, the RNNs trained with a preparatory module are now able to tackle zero-shot transfer to chains of motifs at no performance cost, while driving smooth interpolating transitions (Fig.3c-e,
Fig.14 top right).
This strongly argues that the preparatory module shapes a wide attractive landscape at motif start that includes the complex correlated network states reached at the end of motifs.
Third, through nonlinear interactions between the preparatory input and the rich dynamics of the
RNNs, our networks can match the motifs’ shapes during preparation (Fig.
3d, ﬁrst two rows).
This is another advantage of our RNNs over the above-mentioned dynamically limited models (which can only implement a motif-independent interpolation during preparation, besides having more limited expressivity (Kao et al.
(2020); Logiaco et al.
(2019) and Appendix A.4).
This enables robust, accurate and plausible motif transitions without any tuning of the duration of the preparatory period.
Finally, we want to stress the advantages of the preparatory connectivity perturbation U prep V (cid:124) prep compared to possible alternative approaches that would only try to match the apparent timecourse of the neural activity during preparation in the brain (Sussillo et al., 2015; Zimnik & Churchland, 2021).
First, the preparatory loops can modulate the dynamics in a unit-speciﬁc way to create efﬁcient convergence with a subtle modulation of the RNN connectivity (Fig. 12a,d).
This is the reason why the preparatory input can still interact with the rich RNN’s recurrent connectivity, which allows this input to shape the output into the desired motif even during preparation.
Second, because these loops are not tuned for any particular motif, they can be optimized once and for all, and do not impose any additional cost when learning new motifs.
Instead, when considering naive ways of setting speciﬁc dynamics at the beginnings of motifs such as using a second motif-speciﬁc input at the start of each motif (‘two-inputs’ approach, Fig. 3c), any parameter used for preparation scales with the number of motifs.
We made this ‘two-inputs’ approach comparable to our preparatory module approach by applying the ﬁrst input the network for the ﬁrst 5 τ of the motifs and then a second input during the remainder of the motif, and by equalizing the number of parameters involved (as we had done between the additive and multiplicative network) by setting N = 150 .
We found that this strategy led to both decreased accuracy and robustness compared to using our preparatory module, with both effects combining to a large ( ≈ 40% ) increase in RMSE during transitioning.
We could precisely assess the transitions robustness in these networks this by using an extended motif transition data set (Fig. 3e).
More precisely, we considered transitions between pairs of step motifs where the transition occurs at fractions of 0 . 5 , 0 .
9 , 1 . 0 of the total duration of the ﬁrst motif (e.g. Fig.
3e right, bottom).
This is a more challenging task – that involves a larger variety of network states from which to transition than before (then, we only considered the 100% scenario where all motifs ended at the same value: zero).
For these transitions, the ‘two-inputs’ network showed a signiﬁcant increase in error – including occasional very large errors for some transitions, see Appendix A.5.2 – compared to initializing motifs with the randomness used during training.
In contrast, the additive network had equally robust performance in both conditions (Fig. 3f).
Thus, our results argue that using brain insights to constrain the architectures of networks has advantages, even beyond taking inspiration from the biological activity patterns (Li et al., 2019; Sinz et al., 2019; Zador, 2019).
D ISCUSSION
We found that when trained on many randomly initialized single motifs, gradient-trained nonlinear
RNNs struggle during zero-shot transfer to new sequence orders.
This is relatively surprising given that such RNNs are known to be able to learn attractive trajectories (Laje & Buonomano, 2013;
Sussillo & Abbott, 2009; Pollock & Jazayeri, 2020).
This suggests that training mostly constrains the readout dimension in our setting.
This is both a blessing - as it is the ultimate reason why different motifs can efﬁciently share parameters and thus beneﬁt from a common dynamical baseline that is only partially modulated, without being hurt by irrelevant aspects of the dynamics (Logiaco et al., 2019; Russo et al., 2018) - and a curse as it makes ﬂexible motif chaining difﬁcult.
In this context, we show that using a preparatory module whose architecture is brain-inspired and who imposes a dynamics that mimics motor cortical activity during hierarchical motor sequencing (Zimnik & Churchland, 2021) can efﬁciently enable zero-shot transfer to new sequence orders.
Even more surprisingly, introducing the preparatory module in the nonlinear RNNs actually sizably improves single motif training such that the motifs are also more accurate during training conditions.
These results reveal a general function for motor preparation speciﬁcally for ﬂexible chaining of motor motifs, and show that it is not a byproduct of arbitrary biological hardware constraints or of the restricting assumptions made by previous models to allow for analytical tractability (Logiaco et al., 2019; Kao et al., 2020).
Therefore, ultimately, for the application to robotics control - at least in a context where intrinsic dynamics matter (Liu et al., 2019; Yeo et al., 2016) - this preparatory module could be very useful (i) to increase accuracy, (ii) to enable smooth transitions governed by dynamics that can be tuned to avoid passing through undesirable states, and (iii) to offer a mechanism for learning appropriate transition times by gradient descent - which are all current challenges for stateof-the-art networks (Merel et al., 2019a;b).
In addition, our approach is extendable to combining recurrent dynamics - useful in contexts where sensory feedback is not fully reliable (Yeo et al., 2016;
Liu et al., 2019) - with corrections driven by sensory inputs (Guo et al., 2019).
Our work therefore joins several recent calls outlining the need to infuse ANNs with more expert domain knowledge in order to achieve good generalization in real-world scenarios involving out-of-distribution generalization and zero-shot transfer (Russin et al., 2019; D’Amour et al., 2020; Zador, 2019).
A A PPENDIX A. F ORMAL DEFINITION OF OUR PROBLEM SETTING
Here is how we formally deﬁne the task solved in our paper: -Given a dynamical system D with (i) internal variables h (i e., h ( t ) is deﬁned through its derivative with respect to the continuous time variable t ) and (ii) the property that D can be in one of M different states indexed by µ (with µ ∈ 1 . . .
M , and M is large and a priori unknown), each lasting a ﬁxed duration t µ , such that whenever D is in state µ , a function of the internal states F ( h ) exactly follows a ﬁxed continuous function g µ (of length t µ ) that we term a ‘motif’.
(Concretely, if a particular trajectory of states [ s 2 , s 1 , s 3 ] is visited, F ( h ( t )) is formed of the concatenation of three continuous ’motifs’ indexed by [ s 2 , s 1 , s 3 ] .)
- Given a sequence of indices I =
[ p 1 , ...
p K ] that label a sequence of states undergone by D , where ∀ i, p i ∈ { 1 , . . .
, M } - Given labeled training examples that only contain a ﬁnite number T µ of discrete samples from each continuous function g µ , and given that the learning from different g µ can occur sequentially and extendably without interference - Find a function f which, for each function g k , is parameterized by a vector V µ of size N V < T µ (where N V is as small as possible) such that f ( I ; , V 1 , . .
. , V M , dt ) produces a time series corresponding to F ( h ) when D undergoes the sequence of states ordered by I , with an adjustable sampling interval given by dt (such that a smaller dt leads to a smoother time series).
Finally, we note that is also desirable that f be noise-robust. In our work, we solve this problem using an RNN to implement the function f – a natural choice given that it models a dynamical system D . In addition, the tunable motif-speciﬁc parameters V µ are either the input b µ for the additive network, or the concatenation of the input b µ and the loop parameters u µ and v µ in the multiplicative network. As we mention in the text, given that we work in a continual learning setting where we would like to learn an extendable library of many motifs, we are trying to limit the number of tuned parameters per motif N V (table 1).
This is important for our networks to constitute viable solutions that can be built on for designing motor controllers, because it is then desirable to limit both the memory requirements and the time needed to update the state of the network. Note that we ensured noise-robustness in our work by injecting noise during training in the hidden state of our RNN (which, by a simple transformation of variable, is equivalent to injecting a transformed noise into the input b µ received by the network). The larger component of the noise we used consists in a random initialization of the network state, and this noise then propagates to later timesteps through the recurrent dynamics (see section A.6.3). A. M OTIF GENERATION METHOD AND FULL LIST OF MOTIFS USED
For this study, we use two different types of motifs that challenge the dynamics of continuous time
RNNs: oscillatory motifs, that are easier to produce; and step motifs that are harder.
A.2. O SCILLATORY MOTIFS
To generate oscillatory motifs that would be relatively easy to produce by RNNs, we sampled the output from a random Gaussian network with tanh nonlinearity set in the chaotic regime (Sompolinsky et al., 1988; Sussillo & Abbott, 2009;
Sussillo et al., 2015).
To make these motifs very easy and possible to connect in ‘seamless’ chains, we also low-pass ﬁltered the resulting trajectories.
Fig.
4 lists the oscillation motifs used in this paper.
[Figure]
[Figure]
S TEP MOTIFS
These motifs consist of a series of positive and negative discrete jumps with intervening constant periods.
To generate a particular motif, we follow the following steps.
First, we generate a centered
Ornstein–Uhlenbeck process z by running the dynamics dz/dt = − z
+ (cid:16) γ/ √ dt (cid:17) χ
( t ) , where χ ( t ) is taken from a standard Gaussian independently at each timestep, dt=0.1 and γ = 3 √ 2 such that the steady-state standard deviation of this process is 3.
After a ‘warmup’ period of 200 dt that we discard, we produce an additional 1000 dt of z-values that will be the basis for generating the motif.
Second, we draw random time intervals T k with a uniform probability between 50 dt and 500 dt and select the ﬁrst few in a list [ T 1 , T 2 , ..., T j ] such that (cid:80) jk
=1 T k < 1000 dt .
Third, we set the value of the motif between the start and time T 1 to the average of the z-values over the same time interval; and similarly for the subsequent intervals, the value of the motif between the times [ T i − 1 + dt, T i ] i ∈ [2 ,...,k ] is set to the average of the z-values over the corresponding interval.
Finally, we pad the end of the motif with zeros for 50 dt and also reset the ﬁrst value of the motif to 0.
Fig.
5 lists the step motifs used in this paper.
A. P ERFORMANCE OF ‘ VANILLA ’ RNN S
Here we show examples for the motif-by-motif performance of different ‘vanilla’ RNNs (i.e., without a preparatory module) that relate to section 3 of the main text, where network training involves producing single motifs from a random initialization of the network state.
The networks are tested both on new random initialization (in-distribution generalization) and during motif transitioning (out-of-distribution generalization).
A.3. A DDITIVE NETWORK , N=300, STEP MOTIFS
In Fig.
6, we show an additive network of size 300 trained on the step motifs.
This is a different network than the one we show in the main text Fig.
2a (i)-(iii) but the conclusions are identical – despite the presence of a small variability between different networks trained from different initial seeds (see e.g. main text Fig. 1c).
A.3. M ULTIPLICATIVE NETWORK , N=100, STEP MOTIFS
In Fig.
we show a multiplicative network of size 100 trained on the step motifs.
This is a different network than the one we show in the main text Fig.
2b but the conclusions are identical – despite the presence of a small variability between different networks trained from different initial seeds (see e.g. main text Fig. 1c).
Note that the number of parameters tuned for motifs is adjusted relative to the additive network of size 300 shown previously (main text table 1).
A.3. A DDITIVE NETWORK , N=1000, STEP MOTIFS
In Fig. 8, we show an additive network of size 1000 trained on the step motifs.
Notice that while increasing the network size improved the performance when starting from the random initialization used during training, there are still large errors when transitioning from other motifs.
In Fig.
9, we show an additive network of size 300 trained on the oscillatory motifs.
Notice that while, with these easier motifs, the performance is really good when starting from the random initialization used during training (as compared to the performance for the step motifs), there are still large errors when transitioning from other motifs.
Also, we want to stress that, during sequencing, even when focusing on a particular ﬁrst motif, the network performance can be very different depending on the identity of the second motif.
For instance, let’s focus on the light green motif in the panel ‘When starting from random’, bottom row on the left - the sequences where this motif is the ﬁrst motif are shown as green traces in the panel ‘When starting from other motifs’.
When this green light motif is the ﬁrst motif in a sequence, the performance is almost ﬂawless for following the motif in the panel ‘When starting from other motifs’, second line on the left.
In contrast, the performance is very poor for another following motif situated in the panel ‘When starting from other motifs’, fourth line on the right.
A.3. C ONTROL NETWORK , N=50, STEP MOTIFS
In Fig.
10, we show a control network of size 50 trained on the step motifs.
Like the additive and multiplicative networks, this control network struggles when generalizing from single motif training to motif sequencing.
A. A NALYTICALLY TRACTABLE THALAMOCORTICAL MODEL
Here, we summarize the insights that can be gained from an analytically tractable thalamocortical model whose switching linear dynamics are constrained to stay in the linear regime during individual motif preparation and execution ((Logiaco et al., 2019), see also (Kao et al., 2020)).
Importantly, the constraints on the dynamics of this model largely limit its computational capacity, and no online learning method has been suggested for this model so that the approach is strongly limited to simple scenarios where analytical tractability is possible.
The work set forth in the main text is therefore critical to test and expand the ideas of this model in a setting that is amenable to solving useful tasks in an engineering sense, and for determining whether the biological features included in
[Figure]
[Figure]
The model consists of a recurrent cortical module with connectivity J cc and activities x whose projection through the readout weights w constitutes the output.
This cortical module interacts with a non-recurrent thalamic module through instantaneous loops consisting of corticothalamic and thalamocortical weights.
The basal ganglia, which provides inhibitory input into thalamus, are modeled as selectively disinhibiting speciﬁc thalamic loops in order to cause execution of the associated motif.
Motif execution: During motif µ , a single thalamic loop is disinhibited leading to the dynamics:
[Equation]
where
[Equation]
with motif-speciﬁc loop vectors u µ and v (cid:124) µ .
We now consider how these dynamics can approximate a desired output y µ , knowing that in general a good approximation for y µ can be reached through a (preliminarily numerically identiﬁed) linear combination of a small number K of complex exponentials: y µ ( t )
≈ ˆ y µ ( t ) =
(cid:80)
Kk =1 [ ˆ α µ ] k e [ˆ λ µ ] k t (see main text Fig.
1d; (Logiaco et al., [Figure] Gaussian iid initialization Transitioning from other motifs (test)
[Figure] [Figure] The cortical readout can exactly match ˆ y µ if the eigenvalues of ˜ J µ contain the entries of the vector ˆ λ µ and if the initial network activities x init µ are set correctly.
We accomplish the former (Fig. 11a) by setting v µ =
L (cid:124) diag( Lu µ ) − 1 Q + 1 , where L is the left eigenvector matrix of J cc and
Q kj = 1 / ([ˆ λ µ ] k − λ j ) where λ j is an eigenvalue of g ( J cc − I ) .
Next, we set the initial activities at the beginning of motif µ to x init µ =
˜ R µ diag( ˜ R (cid:124) µ w ) − 1 α µ where ˜ R µ contains right eigenvectors of ˜ J µ (with the ﬁrst K columns corresponding to the eigenvalues in ˆ λ µ ) and [ α µ ] k ≤ K =
[ ˆ α µ ] k and [ α µ ] k>K = 0 .
The preceding two steps do not specify u µ , and with random u µ the readout will be highly sensitive to noise in x init µ (pink trace in Fig.
11a right; see (Logiaco et al., 2019)).
However, if u µ is set to minimize the analytically-computed expected readout deviation due to noise in x init µ by modify- then
Motif transitions: To successfully transition to motif µ , it is sufﬁcient to implement a mechanism by which x approaches x init µ , which will be the case if the dynamics during a so-called “preparatory period” has x init µ as its steady-state.
Additionally, it is desirable that the transition dynamics are fast and that they do not cause large transients values on the readout while relaxing to steady-state
(Kaufman et al., 2014).
To achieve this, it is possible (Logiaco et al., 2019) to employ a speciﬁc thalamic subpopulation of size P which is disinhibited during all motif transitions, as well as a constant input c µ speciﬁc to the upcoming motif µ , leading to the dynamics: [Equation] where J prep ≡ [Equation]
J cc −
[Equation]
U [Equation] with N × P loop weights U prep and V prep .
With these dynamics, the activity at steady-state will match x init µ if c µ =
− J prep x init µ (Fig. 11b).
Note that the difference δ x = x − x init µ between the cortical activities and their steady state decays at a rate that is independent of c µ and therefore of the upcoming motif: τ ˙ δ x =
J prep δ x for all µ .
This allows us to optimize the same weights U prep and V prep to favor rapid and smooth transitions between all pairs of motifs – even though this approach is limited: this optimization has to be motif-independent, so that the shape of the readout cannot be adjusted to the speciﬁc motif being prepared (while the continuously nonlinear RNNs can do this, see Fig. 3d).
Following ref.
(Logiaco et al., 2019), we achieve fast transitions by minimizing the time-integral of the expected square norm of δ x , with rates δ x 0 at the beginning of the transition period sampled iid.
Here, we also augment our cost function with the time-integral of the expected squared derivative of the readout to favor smoother transitions on average.
Our total cost function is therefore:
[Equation]
R [Equation] where R prep and L prep are the right and left eigenvectors of J prep , and its eigenvalues λ prep are used to compute Λ ij = − 1 / ( λ prep i +
λ prep [Equation] and Γ ij = λ prep i λ prep j Λ ij . Finally, N is the number of cortical units and β is a hyperparameter which trades off the relative importance of transition speed and readout smoothness.
Note that the model parameters can be adjusted through analytical and semi-analytical techniques which do not require stochastic gradient descent on the simulated dynamics.
This is an advantage when strictly imposing linear dynamical regimes during each motor preparation and motor execution for the ﬁxed type of autonomous computation that it was designed for, but these restrictions can be limiting to optimize the network for more complex objectives (for instance, including correcting responses to sensory feedback).
We simulated the model on the same task as the gradient-trained RNNs, using our step motifs.
The motif-speciﬁc parameters scale as in the multiplicative architecture (main text table 1), but the thalamocortical model does have a few more hyperparameters.
To make sure that these did not induce an inability to compare between approaches, we reduced the cortical size to N = 99 .
Further, after exploring a few values, we set g/τ = 0 .
5 , K = 10 , P = 50 , β = 1 / 20 , and the motif transition duration to 5 τ .
The readout weights w and recurrent weights J cc were sampled from a centered Gaussian distribution with std 1 / √ N .
The approximations ˆ y µ were ﬁt to the target motifs under the constraints that ˆ y µ (0)
= 0 , that the elements of ˆ λ µ had negative real part and were at least (cid:15) = 0 .
apart from each other, and that the magnitudes of the elements of ˆ α µ were not exceedingly large (no larger than 3).
The resulting ˆ λ µ and ˆ α µ were then used to optimize u µ , v µ , and c µ as described above.
We generated sequences from the thalamocortical model after initializing with iid standard Gaussian samples for the elements of x (this choice of a unit standard deviation indeed leads to readout values within the same range as the target motifs, Fig. 11c).
Interestingly, the motifs were produced with the same reliability when preceded by another motif in a sequence or when starting from random initial conditions (Fig. 11d,e).
Also, choosing K = 10 pushed the network close to its limit in terms of noise robustness, without allowing the constrained thalamocortical network to reach the performance of nonlinear RNNs with the preparatory module (Fig. 11f).
In conclusion, the constrained thalamocortical model is therefore limited in its expressivity, in its capacity to shape the readout during motif transitions, as well as by the rigid and unpractical procedure by which its weights are adjusted.
We will now present additional results demonstrating that insights from this constrained thalamocortical model can be weaved into RNNs to combine zero-shot transfer ability with large expressivity and the ﬂexibility afforded by gradient-based training.
A. W EAVING IN INSIGHTS FROM THE THALAMOCORTICAL MODEL IN GRADIENT - TRAINED ANN S
In Section 4 of the main text, we presented the results of augmenting each of the additive and multiplicative networks with a 50-unit thalamic transition module that is active for the ﬁrst 5 τ of every motif.
Here we present more detail about these results.
A.5. D YNAMICS OF THE RNN S INTERACTING WITH A THALAMIC TRANSITION MODULE
The dynamics of our nonlinear recurrent networks with the transition module but without any input are:
[Equation]
where g x is g ad or g mu for the additive and multiplicative networks respectively.
The weights in U prep and V prep were initialized with centered Gaussian with std (cid:113) 0 .
05 / √ P ∗ N .
We trained the weights of the transition modules with ADAM under the cost function (cid:80) t | r ( t ) | 2 where r ( t ) =
tanh( x ( t )) .
Minibatches consisted of 64 trials of length 20 τ , each starting with random x values sampled iid from the standard normal distribution.
After 1,000 minibatches, both N = 300 and N =networks were seen to have converged.
Figs. 12a,d show the eigenvalue distributions of g x J + U prep V (cid:124) prep after training.
Importantly, though the thalamic module is low rank ( P < N ), all eigenvalues have real part signiﬁcantly less thanwhich causes decay of the network rates towards a 0 ﬁxed-point in the vicinity of this ﬁxed-point.
For comparison, we show the eigenspectrum of g x J rnd
+ U prep V (cid:124) prep for random matrix J rnd which has the same statistics as J .
In this case, there continues to be large eigenvalues that will prevent fast rate decay.
These results demonstrate that the solutions U prep and V prep needed to negate the amplifying dynamics of J are speciﬁc to that particular J .
In Figs.
12b,e, we show the time evolution of the norm of the rate vector r during three sample trajectories prior to learning and one post-learning (all post-learning samples are nearly identical).
On the scale plotted, the norms after learning are indistinguishable from zero after approximately 7 τ .
With the thalamic transition modules in hand, we retrained the additive and multiplicative networks as described in the main text Sections 2 and 3.
For the additive network, our only modiﬁcation from before is that we included the thalamic transition module in the network dynamics for the ﬁrst 5 τ :
[Equation]
For the multiplicative network, to make a more direct comparison with the thalamocortical model presented in the previous section of the Appendix, we set the dynamics such that the input b µ and loop u µ v (cid:124) µ were only active during the transition and post-transition periods respectively:
[Equation]
Figs. 12c,f show that both networks learn to perform the task and show no degradation in their performance when tested on sequence generation (i.e., having initial x values given by the ends of other motifs) rather than when starting them with standard Gaussian x values (which was their training regime).
Similarly, Fig.
13 shows that training on smooth motifs using a preparatory module leads to robust performance during zero-shot transfer to motif chaining in an example additive network (compare to Fig. 9).
Notice how the shape of the motif is also matched during the preparatory period, which is enabled by the interaction between the motif-speciﬁc input b µ and the nonlinear tanh dynamics.
[Figure] [Figure]
⌧ s ) <latexit sha1_base64="yw143PH/rBxkb+ZwfSHGKY2K/30=">AAACC3icbVA9SwNBEN3zM8avqKXNkkTQJtyJoHaijWWEfAhJCHubOV3c3Tt258RwpLfxr9hYKGLrH7Dz37j5KDTxwcDjvRlm5oWJFBZ9/9ubm19YXFrOreRX19Y3Ngtb2w0bp4ZDnccyNtchsyCFhjoKlHCdGGAqlNAM7y6GfvMejBWxrmE/gY5iN1pEgjN0UrdQbCM8oI2ymlBA93WqQjA0jmi5jSzt2vLBoFso+RV/BDpLggkpkQmq3cJXuxfzVIFGLpm1rcBPsJMxg4JLGOTbqYWE8Tt2Ay1HNVNgO9nolwHdc0qPRrFxpZGO1N8TGVPW9lXoOhXDWzvtDcX/vFaK0UknEzpJETQfL4pSSTGmw2BoTxjgKPuOMG6Eu5XyW2YYRxdf3oUQTL88SxqHleCocnp1WDo7n8SRI7ukSPZJQI7JGbkkVVInnDySZ/JK3rwn78V79z7GrXPeZGaH/IH3+QPzLppZ</latexit>
[Figure]
<latexit sha1_base64="yw143PH/rBxkb+ZwfSHGKY2K/30=">AAACC3icbVA9SwNBEN3zM8avqKXNkkTQJtyJoHaijWWEfAhJCHubOV3c3Tt258RwpLfxr9hYKGLrH7Dz37j5KDTxwcDjvRlm5oWJFBZ9/9ubm19YXFrOreRX19Y3Ngtb2w0bp4ZDnccyNtchsyCFhjoKlHCdGGAqlNAM7y6GfvMejBWxrmE/gY5iN1pEgjN0UrdQbCM8oI2ymlBA93WqQjA0jmi5jSzt2vLBoFso+RV/BDpLggkpkQmq3cJXuxfzVIFGLpm1rcBPsJMxg4JLGOTbqYWE8Tt2Ay1HNVNgO9nolwHdc0qPRrFxpZGO1N8TGVPW9lXoOhXDWzvtDcX/vFaK0UknEzpJETQfL4pSSTGmw2BoTxjgKPuOMG6Eu5XyW2YYRxdf3oUQTL88SxqHleCocnp1WDo7n8SRI7ukSPZJQI7JGbkkVVInnDySZ/JK3rwn78V79z7GrXPeZGaH/IH3+QPzLppZ</latexit>
[Figure]
e
M u l t i p li
[Figure]
t i v e
[Figure] [Figure]
A.5. C OMPARING THE THALAMIC TRANSITION NETWORK TO OTHER WAYS OF IMPLEMENTING A DIFFERENT DYNAMICAL REGIME AT TRANSITIONS
In the main text, at the end of section 4, we gave statistics about a comparison between using transition module described above and an alternative approach for setting the RNN in a different dynamical state at the beginning of a motif.
More speciﬁcally, we considered a ‘two-inputs’ approach where we use a ﬁrst motif-speciﬁc input during the ﬁrst 5 τ of each motif to try to facilitate transitions, and a second motif-speciﬁc input during the rest of the motif to facilitate the production of the motif by the network.
Following the general approach in our article which adjusts the number of parameters tuned per motif (main text table 1), we set the size of the recurrent network to N = 150 so that the two inputs together lead to 300 parameters tuned per motif (and therefore 3000 parameters tuned in total when learning the ten motifs).
We used an extended data set consisting of transitions between pairs of step motifs where the transition occurs at 50%, 60%, 70%, 80%, 90%, or 100% of the total duration of the ﬁrst motif.
This is a more challenging task than before when we only considered the 100% scenario because all motifs end at the same value (zero).
As stated in the main text, we found that the ‘two-input’ network did not perform as well as the matched additive network with a transition module (as described in the previous subsection of the Appendix, this module is not tuned to particular motifs, so the size of the recurrent network is here simply N = 300 to match the per-motif parameter count with the ‘two-input’ network).
In Fig.
14, we show examples for the single-motif performance of the ‘two-input’ network and the matched additive network, when transitioning from 70% vs. 100% of the duration of the previous motif (the latter one being the same situation as considered elsewhere in this work). Examining Fig.
14 reveals that the ‘two-inputs’ network behaves differently from the additive network in two major ways.
First, its overall motif accuracy is lower; and second, when transitioning from 70% of the duration of the previous motif, their accuracy consistently decreased compared to when initializing the motif at random as during training.
This decrease in accuracy was seen consistently when transitioning in all scenarios when the previous motifs interrupted before their ending (which is zero-padded).
This is probably because the larger variability of network outputs when interrupting motifs is reﬂected in a larger variability of network states.
Importantly, the additive network with a transition module could handle this variability to allow the network to produce a more accurate output when chaining an interrupted motif with a subsequent one. These results strongly suggest that ‘engineering’ a ﬁxed point in the dynamics at the start of motifs (at a position that is adjusted with a preparatory input) is a particularly potent way of ensuring accurate and robust motif performance. In addition, close to a ﬁxed point (which is presumably a good description of the dynamical regime close to the end of the preparatory period), the dynamics is quasi-linear, in which case classical results from control theory indicate that feedback loops such as those of the transition module are the optimal strategy to steer the network towards its target. Therefore, these results further strengthen our conclusion that the transition module we propose is a particularly potent strategy for zero-shot transfer during motif chaining.
We tested various parameters of ADAM. We identiﬁed the following as yielding successful training in our setting: learning rate = 10 − 3 − 10 − 4 , β 1 = β 2 = 0 . 5 , and (cid:15) =
10 − 8 .
Default parameters for ADAM most often worked very well (especially for the additive networks), but the parameters above could help training for the small control networks, or occasionally for the multiplicative networks – whose training was often non-monotonous (see Fig.
15 below), which could be especially limiting when training easier motifs for which the errors are small and/or the network activity varies little between different runs.
[Figure]
T r a n s i t i o n s f r o m 70 %
o f p r e v .
m
o t
i f
T r a n s i t i o n s f r o m 100 %
o f p r e v .
m
o t
i f
A.6. D IFFERENCES IN LEARNING CURVES BETWEEN THE ADDITIVE AND THE MULTIPLICATIVE NETWORKS
Our investigation through hyperparameter search of the performance of additive and mutliplicative networks matched in number of tuned parameters (by ﬁxing the size of the additive network to
N = 300 and the size of the multiplicative network to N = 100 ) suggests that they lead to similar errors on average when trained to produce the step motifs (main text Fig.1c).
Interestingly, we observed that these similar performances were typically reached through different types of learning curves (Fig. 15): the multiplicative networks tend to show discontinuous steps in the learning curve, whereas the additive networks typically have smooth learning curves.
This suggests that the performance of the multiplicative networks may be limited by the ability of gradient descent to fully optimize these networks to their best possible performance.
In the future, it would be interesting to investigate ways of improving the optimization of the multiplicative networks - for instance using motif-speciﬁc learning rate schedules, or by increasing its size to N = 300 while only training onethird of the weights of the loop and input vectors (so that it has as many untuned parameters as the additive network). A dd i t i v e a r c h i t e c t u r e M u l t
i p l i c a t i v e a r c h
i t e c t u r e # of training minibatches R oo t m e a n s q u a r e e rr o r # of training minibatches R oo t m e a n s q u a r e e rr o r
The RNNs we propose need to be noise robust so that they can constitute viable solutions that can be built upon for motor control in the real world. First, a transition to a new motor motif can be triggered in order to correct a corrupted motor command that corresponds to a noisy network state. Second, for more general applications of the current framework to motor control in the real world, the dynamics often need to be modulated by sensory input – which is never perfectly reliable.
In particular, the triggering of motif transitions is likely to often depend on sensory input.
For these reasons, we imposed noise robustness through a relatively large noise at motif start, implemented through a random initialization of the state of the network: at each run, the initial activities x are sampled iid from a Gaussian distribution.
Importantly, because of the recurrent dynamics, the initial noise is also propagating at later times through the recurrent dynamics.
Additionally, in most of our simulations - including all the simulations with our preparatory module - we added a small (Gaussian with std = 0 .
001 ) amount of noise to the dynamics at each timestep.
Here, we show that the inclusion of noise is indeed necessary for getting noise-robust solutions (Fig. 16), illustrating this for additive networks.
When training networks on single motifs initialized from the origin without noise - as opposed to adding Gaussian noise to the origin as we did in the main text - testing the networks with small amounts of noise leads to unreliable network outputs (Fig.
16 a, second column).
In contrast, a network trained on motifs initialized from a standard
Gaussian as a more reliable output across a range of standard deviation of the initialization (Fig.b, ﬁrst three columns).
Finally, we note that while training the vanilla additive network without noise yields a higher accuracy in the noiseless conditions compared to a vanilla RNN handicapped by the requirement of noise robustness during training (Fig.
a left vs. Fig.
{+16 b left), even in these noiseless initialization conditions the vanilla additive network does not perform better than an additive network with a preparatory period and a noisy initialization (Fig. 
+}
a left vs Fig. 12c).
Therefore, not only do our biologically-inspired networks with a preparatory period display robust generalization to both in-distribution and out-of-distribution perturbations, they also do it at no accuracy cost compared to networks that are much less noise robust, and at no additional cost in terms of number of tuned parameters per motif.
In addition, we will see in the following section that our preparatory module also speeds up learning.
A.6. T HE PREPARATORY MODULE COMBINED WITH RANDOM INITIALIZATION SPEEDS UP LEARNING
In the last section of the main text results, we argued that the preparatory module - which introduces fast convergence timescales at the beginning of motifs - helps single motif training.
Indeed, when initializing motifs from a standard Gaussian both during training and testing, we observed that the networks with a preparatory module reach much lower error than the networks without such a module (Fig.
3c left, dashed lines vs. full lines).
[Figure]
Here, we give more evidence of this by examining the learning dynamics of the networks with or without a preparatory module.
We ﬁnd that networks trained with a preparatory module and standard Gaussian initialization reach a performance plateau faster (Fig. 17).
{+Interestingly, not only is this true compared to training without a preparatory module with the same a standard Gaussian initialization (Fig. 
+}
a middle and right; and Fig.
17 b), but is is also true when compared to training without a preparatory module from a noiseless origin (Fig.
17 a left).
Taken together, our results strongly argue that our preparatory module helps network performance in two ways.
First, when training single motifs from a ﬁxed noisy initialization, the fast dynamics timescales introduced by the preparatory module appear to help the training-induced changes in the input to drive the exploration of different possible activity states until ﬁnding a motif-speciﬁc one that leads to particularly accurate motif production (Mohajerin & Waslander, 2019; Kemeth et al., 2021).
Second, it helps ﬁltering out noise and robustly converging to this motif-speciﬁc activity pattern to start motif production.
Therefore, the preparatory module helps the network implement computations that are shared between motifs, and then acts as an inductive bias such that the motifspeciﬁc parameters can be fully devoted to motif-speciﬁc functions (shape the motif-speciﬁc readout and ﬁnal activity pattern during the preparatory period, and modulate the network dynamics during the remainder of motif production for accurate readout).
This is why the preparatory module can be trained once and for all before training any motif speciﬁc parameters, which leads to improvements in accuracy and robustness at no per-motif cost in terms of tuned parameters and without introducing any learning interference in a continual learning scenario.
[Figure]