Do What Nature Did To Us: Evolving Plastic Recurrent Neural Networks For Task Generalization
Abstract
While artiﬁcial neural networks (ANNs) have been widely adopted in machine learning, the gaps between ANNs and biological neural networks (BNNs) attracts attention increasingly.
In this paper, we propose a framework named as Evolutionary Plastic Recurrent Neural Networks ( EPRNN ).
Inspired by BNN,
EPRNN composes Evolution Strategies, Plasticity Rules, and Recursion-based
Learning all in one meta learning framework for generalization to different tasks.
More speciﬁcally, EPRNN incorporates with nested loops for meta learning — an outer loop searches for optimal initial parameters of the neural network and learning rules; an inner loop adapts to speciﬁc tasks.
In the inner loop of EPRNN , we effectively attain both long term memory and short term memory by forging plasticity with recursion-based learning mechanisms, both of which are believed to be responsible for memristance in BNNs.
The inner-loop setting closely simulate that of BNNs, which neither rely on gradient based optimization methods nor require the exact forms of learning objectives.
To evaluate the performance of
EPRNN , we carry out extensive experiments in two groups of tasks:
Sequence
Predicting , and Wheeled Robot Navigating .
The experiment results demonstrate the unique advantage of EPRNN compared to state-of-the-arts based on plasticity and recursion while yielding comparably good performance against deep learning based approaches in the tasks.
The experiment results suggest the potential of
EPRNN to generalize to variety of tasks and encourage more efforts in plasticity and recursion based learning mechanisms.
Introduction
Artiﬁcial Neural Networks (ANNs) achieve huge success in machine learning. Despite being initially inspired by Biological Neural Networks (BNNs), the optimization paradigm for ANNs has diverged from that of BNNs.
BNNs are believed to learn through plasticity rules (Gerstner et al., 1993), among which Hebb’s rule (Hebb, 1949) is most well known. In contrast, the optimization of ANNs is dominated by gradient descent methods, the evidences of which are less revealed in BNNs .
Though up to now gradient descent methods are the most efﬁcient optimizers for ANNs, their limitations are beginning to attract attentions, including the problems of catastrophic forgetting, over consumption of data, and the requirement for manual efforts in designing target functions.
Those challenges are becoming the main impedance for further development of machine intelligence.
Recent studies show that under proper meta-parameter optimization, both recursion (Santoro et al., 2016; Mishra et al., 2018) and plasticity (Soltoggio et al., 2008; Najarro & Risi, 2020) can be valid alternatives to train ANNs for task generalization.
Unlike gradient descending, both of the two learning mechanisms work in an unsupervised manner, without requiring exact forms of learning objectives, which are believed to more closely simulate BNNs.
Recursion based learning typically employs recurrent neural networks ( RNN ), LSTM (Hochreiter &
Schmidhuber, 1997), and self-attention (Mishra et al., 2018; Chen et al., 2021) layers as learners.
Learning takes place within the feed forward pass.
The information are updated in the hidden states instead of the parameters.
Recursion are found to be extremely sample efﬁcient in generalized supervised tasks (Santoro et al., 2016), zero-shot generalization in language (Brown et al., 2020), and reinforcement learning (Mishra et al., 2018;
Chen et al., 2021) when compared with various type of gradient descent methods.
So far among those learners, though self-attention based learners such as Transformers have state-of-the-art performance, the computational costs for inference per step grow linearly with the sequence length, which makes them only available to relatively short sequences.
On the other hand, recurrent learners such as RNN and LSTM have constant per-step inference costs, but suffer form poor asymptotic performances.
That is, when sequences are getting longer, performances no longer improve or even deteriorate.
This is partly due to the limitation of the memory spaces.
For instance, an recurrent hidden layer of size n has a memory of scale O ( n ) .
In contrast, its parameters scale with O ( n 2 ) , which makes parameter-updating more powerful as learning mechanisms than recursion-only.
[Figure]
Prior to our work, evolving plasticity (Soltoggio et al., 2008, 2018; Lindsey & Litwin-Kumar, 2020;
Yaman et al., 2021) has been proposed to reproduce the natural evolution and plasticity in simulation, as shown in Figure 1.
Implementing plasticity is not straight-forward, unlike gradient-descent, plastic rules are not universal but have to be optimized beforehand, which is not possible without a further outer-loop optimizer over the inner-loop learning.
Evolutionary algorithms (Zhang et al., 2011) are typically applied in the outer loop to search for meta-parameters shaping the learning rules, which can be regarded as information carried by genomes during evolution.
Those optimized plasticity rules are then applied in the inner loop to further tune the parameter of NN for better adaptions to the environment.
Due to the complexity in training, plasticity is insufﬁciently investigated and its potential is not well understood. Existing works on evolving plasticity are mainly focused on feed-forward layers.
However, as recursion widely exists in BNNs, it is desirable to also investigate plasticity in recurrent connections.
Another motivation of combining recursion with plasticity is to simultaneously possess the merits of updating information in hidden states and in connection weights .
Both recursion and plasticity mechanisms are believed to be relevant to memristance.
While recursion that updates hidden states only is likely to be related to short-term memories, synaptic plasticity that updates connection weights is more related to long-term memories (Bliss & Lømo, 1973; Mel, 2002; Linares-Barranco & SerranoGotarredona, 2009).
Based on this motivation, we propose a new learning framework named as
[Table]
Evolutionary Plastic Recurrent Neural Networks ( EPRNN ).
Speciﬁcally, we apply plastic rules in recurrent connections by utilizing the information of time-dependent hidden states.
Then, following the natural evolution of intelligence (Fig. 1), in the outer loop we employ evolution strategies to search for proper initial parameters and plastic rules.
To validate our proposal, we carry out experiments on two groups of tasks: Sequence Predicting and Wheeled Robot Navigating .
We compared the proposed framework with the other recursion and gradient based learners.
The experiment results show that EPRNN can surpass decent models such as LSTM using the most naive recurrent structures.
Although the validated model structure is relatively simple, we believe that validating such prototypes is an important step toward more general cases.
Related Works 2.1 Meta Learning
Meta learning aims at building learning machines that gain experience using task-speciﬁc data over a distribution of tasks.
Inspired by human and animal brains that are born with both embedded skills and capability of acquiring new skills, meta learning implements two nested learning loops: The outer learning loops optimize the meta that typically involves initial parameters (Finn et al., 2017; Song et al., 2019), learning rules (Zoph & Le, 2017; Najarro & Risi, 2020; Pedersen & Risi, 2021), and model structures (Soltoggio et al., 2008; Li & Malik, 2016) over a distribution of tasks; The inner learning loops adapt the model to speciﬁc task by utilizing the meta.
According to different inner-loop optimizers, we roughly classify the methods into model-based and parameter-updating methods.
The model-based methods do not update the parameters in the inner-loop, where learning is accomplished by updating the hidden states (e.g., recursion); The parameter-updating methods update the parameter in the inner-loop (e.g., gradient descent (MAML), plasticity).
From this point of view, our method can be classiﬁed into both groups.
A brief review of the typical meta learning paradigms is presented in Table 1.
2.2 Evolving Plasticity
The proposal of the learning mechanism of BNNs is initially raised by Hebb’s rule (Hebb, 1949), the most well-known part of which is “neurons ﬁre together wire together”.
It is further polished by Spike Time-Dependent Plasticity (STDP) (Gerstner et al., 1993) indicating that the signal of learning is dependent on the temporal patterns of the pre-synaptic spike and post-synaptic spike.
Learning could also appear in inhibitory connections, also known as anti-hebbian (Barlow, 1989).
Also, relationships between STDP and memristance are also investigated (Linares-Barranco & Serrano-Gotarredona, 2009).
Since many of those rules are related to spiking neural networks (Ghosh-Dastidar & Adeli, 2009), to apply them to ANNs, simplifed rules are proposed (Soltoggio et al., 2008) instead: given the pre-synaptic neuron state X and post-synaptic neuron state Y , the connections between X and Y are updated by
[Equation]
m is the output from neuron modulators that adjust the learning rates of plasticity.
Most of the existing rules are sub-classes of Equation 1.
For instance, some works neglect the neural modulator m (Najarro & Risi, 2020;
Miconi et al., 2018), others have set B , C , and D to
(Miconi et al., 2018, 2019).
The learned rule A, B, C, D will inevitably be dependent on initial parameter of W , however, learning plastic rules that is not dependent on the initial parameters was also investigated (Najarro &
Risi, 2020; Yaman et al., 2021).
Algorithms
Problem Settings .
We consider an agent (learner) that is dependent on meta-parameter θ .
It has the capability of adapting itself to a distribution of tasks T j ∈ T by interacting with the environment
T j through observation i t and action a t .
In K-shot learning, the agent is allowed to ﬁrst observe samples of length K (this stage can be referred as meta-training-training , see Beaulieu et al.
, then its ﬁtness is calculated in meta-training-testing rollouts.
In Generalized Supervised Learning tasks ( GSL ), the observations typically include features ( x t ) and labels ( y t ) in meta-training-training stage ( i t = { x t , y t } ), and the labels are left out for predicting in the meta-training-testing stage (Santoro et al., 2016; Garnelo et al., 2018).
In Generalized Reinforcement Learning tasks ( GRL ), the observations typically include states ( s t ), actions ( a t − 1 ), and feedbacks ( r t − 1 )
( i t = { s t , a t − 1 , r t − 1 } , sometimes r t − 1 can not be observed) (Mishra et al., 2018).
The goal of meta-training is to optimize θ such that the agent achieves higher ﬁtness in meta-training-testing .
In meta-testing , similarly, the learned parameters are given meta-testing-training and meta-testing-testing in order, the performances in meta-testing-testing are evaluated.
Plastic Recurrent Neural Networks (PRNN) .
Given a sequence of observations i 1 , ..., i t , ... , we ﬁrst consider an recurrent neural network ( RNN ) that propagates forward and yields sequence of outputs a t following:
[Equation]
where h t is the hidden states at step t .
In PRNN, we kept W i stationary, but we set W t to be plastic, so that we add a subscript t to mark different W t at different steps.
Regarding h t as pre-synaptic neuron states, and h t +1 as post-synaptic neuron states, by applying Equation 1, we update W t with:
[Equation]
where we use (cid:12) and · to represent “element-wise multiplication” and “matrix multiplication” respectively.
h and 1 are column vectors.
W A , W B , W C , W D are collection of plastic rules of A, B, C, D from Equation 1, which has the same shape as W t .
m t is the neural modulators that adjusts the learning rates of plasticity.
We calculate m t by applying a neuron modulating layer denoted with:
[Equation]
[Figure]
[Equation]
A sketch of PRNN is presented in Figure 2.
The main difference between PRNN and naive RNN is that PRNN updates both the hidden states and the connection weights during the forward pass.
Evolving PRNN .
Given task T j ∈ T , by continuously applying Equation 2 to 7 over meta-trainingtraining and meta-training-testing , the ﬁtness is eventually dependent on the initial parameters, learning rules, and the sampled task T , which is denoted as:
[Equation]
Following Evolution Strategies (ES) (Salimans et al., 2017), in k th outer-loop iteration, we sample different tasks from T , and meta-parameters θ k,i ( i ∈ [1 , n ] ) from the neighbourhoods of θ k .
We evaluate the ﬁtness of sampled meta-parameters, and update the meta-parameters by applying:
[Equation]
Why Recurrent Neural Networks?
As stated in Equation 1, plasticity in feed-forward-only NNs allows NNs to gain experiences from single-frame observation only.
In cases of non-sequential GSL, the plasticity has chances to tune the connection weights to the speciﬁc task by relying on observing one single frame of data ( i t = { x t , y t } ), since its information of the feature and the supervision is complete.
However, in general cases, learning can not happen without summarizing sequences of observations.
For instances, a human driver getting used to a new car through continuously interacting and observing.
Moreover, in GRL, there are time lag between the observation of states and feed-backs.
Recursion helps to summarize historical observations to give the correct update for the connection weights.
Although, compared with naive RNN , there are obviously bunches of more sophisticated neural structure such as GRU and LSTM , we believe it is more desirable to start from simplest recurrent structure to study the potential of combining recursion and plasticity.
Experiments 4.1 Tasks for Evaluation
In generalized tasks, we have each task T j dependent on some conﬁguration parameters that are hidden from the agent.
Below we introduce two groups of generalized tasks that we experiment on.
Sequence Predicting (Generalized Supervised Learning tasks).
We randomly generate sequences of vectors y ( t ) =
( y 1 ( t ) , ..., y l ( t )) , where y i ( t ) =
A i sin ( 2 πtn
i
+ φ i ) , t = 1 , 2 , 3 , ...
and A i ∼ U (1 , 3) , n i ∼ U (10 , 100) , φ i ∼ U (0 , 2 π ) , and U ( a, b ) represents the uniform distribution between a and b .
A , n and φ are hidden from the agent.
The front part of the sequence y (1) , y (2) , ..., y ( K ) is exposed to the agent, and the left part y
( K + 1) , y ( K + 2) , ..., y ( N ) is to be predicted.
The ﬁtness is the opposite of mean square error (MSE) between the predicted sequence and the ground truth.
We test the methods for comparison in four groups of tasks including ( l = 1 , K = 10 , N = 20) , ( l = 1 , K = 25 , N = 50) , ( l = 3 , K = 10 , N = 20) , and ( l = 3 , K = 25 , N = 50) (see
Figure 3(a)(b)).
Wheeled Robot Navigating (Generalized Reinforcement Learning tasks).
The agent is to navigate a two-wheeled robot to a randomly generated goal in 2-D space g =
( g x , g y ) .
We assume that there is a signal transmitter on the goal and a receiver on the robot.
The robot observes the signal intensity decided by A t =
A 0 − k · log ( d t /d 0 )
+ (cid:15) (inspired by the attenuation for electromagnetic wave, see
Friis (1946)), where d t is the current distance between the robot and the goal, (cid:15) ∼ N (0 , σ ) is the white noise in the observation.
g x , g y , A 0 , k are environment related conﬁgurations that is hidden from the agent.
For each task, we sample conﬁgurations by g x , g y ∼ U ( − 0 .
5 , 0 . 5) , A 0 ∼ U (0 . 5 , 2 . 0) , and k ∼ U (0 . 1 , 0 . 5) .
The action is the rotation speed of its two wheels that controls the orientation and velocity of the robot.
The reward at each step is r t = − d t , an episode terminates when the robot approaches the goal or steps reaches the maximum of 100.
We also hide its own position and orientation from the agent, such that the agent relies on recording its own action the signal strength
A t to locate itself.
We investigate three types of navigating circumstances with different level of noises in the observed signal: Low Noise ( σ = 0 . 01 ), Median Noise ( σ = 0 . 05 ), and High Noise ( σ = 0 . 2 ) (see Figure 3(c)). [Figure] [Figure]
4.2 Experiment Settings
We add the following methods into comparison, the methods share exactly the same outer loop and differ in the inner loop.
• ES-MAML (Song et al., 2019) :
We use four gradient descent steps in the inner loop, the learning rate of each step is treated as meta-parameters which is to be optimized by the outer loop.
As MAML can not utilize instant observation in zero-shot case, we show results of both ES-MAML (zero-shot) and ES-MAML (one-rollout) in Wheeled Robot Navigating .
Except for ES-MAML , the other methods are measured with zero-shot meta-testing score only in Wheeled Robot Navigating .
• ES-RNN : Vanilla RNN as the inner loop learner.
• ES-LSTM :
Long Short Term Memory (Hochreiter & Schmidhuber, 1997) as the inner loop learner.
• EPMLP (Soltoggio et al., 2018): Multi-Layer Perceptrons (MLP) with plasticity rules implemented.
• EPMLP (Random) (Najarro & Risi, 2020):
The main difference with EPMLP is randomly setting the parameters of the plastic layers at the beginning of each inner loop instead of using ﬁxed trainable initial parameters.
• EPRNN (w/o m) :
Removing neuron modulator ( m t ) from the plasticity rule in PRNN.
We add additional non-plastic fully connected layers before and after the plastic layers to increase the representation capability of the models.
For fairness, we kept those layers identical for all compared methods.
For every 100 outer loops in meta training , we add an extra meta testing epoch, evaluating the average ﬁtness of current meta-parameters on testing tasks.
Each run includes 15000 outer loops and 150 meta-testing epochs.
Each result is concluded from independent 3 runs.
Our coderelies on PARL 2 for parallelization.
We leave the detailed illustration of the model structures and hyper-parameters in the Appendices.
4.3 Results
We present the experiment results in Figure 4 (In Table 2, and Table 3, we also list the summarized performance by averaging the Top-3 meta-testing scores in the latest 10 meta-testing epochs of each run over 3 independent runs).
Generally, we can conclude that PRNN performs substantially better when compared with naive RNN .
In some cases, it even produces better results compared with LSTM , despite the simpler model architecture.
It is also interesting to notice that the gap between RNN and
PRNN are smaller in shorter sequences or low-noise environments, but larger in more challenging tasks with longer sequence or higher noise (In Wheeled Robot Navigating tasks, higher noise pushes the agent to maintain a longer memory in order to ﬁlter the noise and ﬁgure out the way to goal).
This phenomenon reafﬁrms the lack of long-term memories in RNN , and shows that PRNN signiﬁcantly improves this drawback.
Comparing EPRNN (w/o m) with RNN and EPRNN clearly demonstrates that simple ABCD rule (without the neural modulator) may also work to some extent, but introducing the neuron modulator m t can further beneﬁt the learner.
Among plasticity based methods, we show that recursion is more advantageous than MLP in evaluated tasks.
It is also worth noticing that EPMLP and EPMLP (Random) perform steadily beyond the gradient-based learner ( ES-MAML ).
In ES-MAML , the gradient can only be calculated after an episode is completed, while EPMLP is able to perform sequential learning even though no feedback is available during its life time.
This demonstrates the possibility of surpassing human-designed gradient-based learning rules with automatically learned unsupervised rules.
Moreover, comparison between EPMLP and EPMLP (Random) validate the proposal of Najarro & Risi (2020), implying the possibility of discovering global plastic learning rules instead of rules coupled with the initial parameters.
Yet we see optimization of the initial parameters is still advantageous, which can also be validated by evidences in nature that the newborn lives already have certain embedded skills (e.g.,
Newborn human babies have reﬂexes of suckling and grasping; Foals can stand shortly after being born).
[Figure]
(l=1,K=10,N=20) [Figure]
l
[Figure]
[Table]
[Table] 4.4 Analysis To investigate whether plasticity rules update the connection weights as expected, we test the trained model with different tasks and record the updating trajectories of plastic connection weights W t and hidden states h t .
We run t-SNE visualization to map those tensors ( W t s and h t s) to 2-D space and show their trajectories in Figure 5.
The connection weights W t typically start from the same position and gradually move in different directions depending on the task conﬁgurations.
The ﬁnal weights effectively capture environmental conﬁguration that was hidden from the agent.
Particularly, in Figure 5(d) for Wheeled Robot Navigating tasks, we can see that W t captures only the signal transmission patterns ( A 0 , k ), but neglects the position of the goal ( g x , g y ).
We guess that
A 0 , k are important stationary patterns that helps the agent to interpret the observed signal strength ( A t ), while the absolute position of goal is less important as its relative position to the robot is changing continuously.
This is a clear demonstration showing that plasticity has perform meaningful update on the connections weights depending on the tasks.
On the other hand, the hidden states (Figure ??
(c)(d)(e)(h)) are noisier and less distinguishable among different tasks.
The small repeated circles in the trajectory of hidden states might correspond to short-term patterns such as the periodicity of Sequence Predicting task and the circular movements in Wheeled Robot Navigation tasks.
(see
Appendices ?? )
Conclusions
In this paper we present a nature-inspired learning framework composed of Evolution Strategies,
Plastic rules, and Recursion.
Experiment results show that plasticity can be effectively forged with recursion to enhance the learning capability.
The proposed framework can achieve equivalent or even better performances compared with more sophisticated neural structures, by applying the simplest recurrent neural structures.
Moreover, we also show that under proper meta parameters, plasticity has a chance to surpass gradient descent methods.
We believe the learning framework of Figure 1 can be extended to more sophisticated plastic rules and model structures, uncovering better learners in the future.
Also, it would be more interesting if such framework can be validated in more complex environments such as natural language processing [Figure] (a [Figure]
for
[Figure] [Figure] [Figure] [Figure]
[Figure]
A=
[Figure]
n
[Figure]
c [Figure] [Figure] tasks and vision-related tasks.
Finally, we are looking forward that this work can shed light to new paradigm of building intelligent machines and inspire more efforts in this line of research,
A Appendices A.1 Model Architecture and Training Details
To maintain fairness for comparison, the model architectures of the compared methods are similar, where the differences lie in only one of their layers (Figure 6).
We use 3 hidden layers for Sequence
Predicting tasks and 4 hidden layers for Wheeled Robot Navigating tasks.
For ES-MAML , we replace the PRNN layer with fully connected layer; For ES-RNN and ES-LSTM , we replace it with
RNN and LSTM respectively; For EPMLP , we replace it with plastic fully connected layer, where the plasticity rule is stated by Equation 1, and the neural modulator m is calculated by an additional dense layer with sigmoid activation.
The hidden sizes of all the hidden layers are 64 (for LSTM it is 64 hidden states and 64 cell states).
[Figure]
For Sequence Predicting tasks, the input observation and the output action has the dimension of l .
In the meta-training-training and meta-testing-train stages, we use the ground truth y t − 1 as input observation;
In the the meta-training-test and meta-testing-test stages, its previous action ( a t − 1 ) are taken as inputs.
For Wheeled Robot Navigating tasks, the output action is the control command of its two wheels (a length 2 vector), the input observation is the concatenation of its previous action and the current observed signal intensity ( A t ).
[Figure]
𝜃 # +𝛼 ! ’
∑ (𝜃 #
[Figure]
[Figure]
# ) ( 𝐹𝑖𝑡 𝜃 #,( {𝑇 !
[Figure]
[Figure]
𝑇 "
[Figure]
𝜃 # [Figure] [Figure] The training process is accelerated by utilizing the paralleling mechanism of PARL.
We employ 400 workers (400 Intel(R)
Xeon(R) CPU E5-2650) running inner-loops for 400 off-springs in each generation of the evolution, and additional 1 CPU to perform evolutionary update (shown in
Figure 7(b)).
It takes 3 to 12 hours for each run depending on the length of the inner loop and model architectures.
Following the previous work (Salimans et al., 2017), we rank normalize the ﬁtness among 400 workers.
The learning rate α is set to 0 .
The mutation is performed by adding independent Gaussian noises to each parameter.
During the meta-training , we sample B different tasks and 400 meta-parameters.
Each worker run B meta-training-training and B meta-trainingtesting given the assigned meta-parameter, then the ﬁtness is averaged.
We set B = 16 for Sequence
Predicting tasks, and B = 4 for Wheeled Robot Navigating tasks.
For each meta-testing epoch, we evaluated the current meta parameters in newly sampled 1600 tasks.
A.2 Supplementary Experimental Results and Analyses
We show more t-SNE visualization of the connection weights and hidden states in Figure 8 which can be compared with Figure 5.
The transformation of hidden states in ES-RNN (Figure 8(a)(b)(c) and (e)) are even less distinguishable among different tasks compared with that of EPRNN (Figure 5(d)(e)(f) and (h)), showing naive RNNs fail to capture task conﬁgurations.
The weights of EPMLP can also capture some hidden conﬁgurations (Figure 8(d)) but are less accurate compared with that of EPRNN (Figure 5(g)).
For instance, the gold and grey line share the same hidden conﬁguration of A = 0 .and
k = 0 .
The connection weights effectively captures this information and the two lines overlap with each other in EPRNN , but it does not happen in EPMLP .
Similar phenomenon can be observed for the red and the green lines ( A = 0 .
5 and k = 0 . 1 ).
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
, -0.3), A=0.5, k=0.1 g=(-0.3, -0.3), A=2.0, k=0.1 g=(-0.3, -0.3), A=0.5, k=0.5 g=(-0.3, -0.3), A=2.0, k=0.
[Figure]
We also plot the trajectories of the robot in Wheeled Robot Navigation tasks.
All the agents struggle to ﬁnd the goal by continuously shifting their directions, which is more frequent in cases of higher observation noise.
A reasonable guess is that in higher noise cases, the agents are less conﬁdent in their estimations.
They have to more frequently adjust their directions to acquire diverse observations for more effective denoising.
[Figure]