D O W HAT N ATURE D ID T O U S : E VOLVING P LASTIC R ECURRENT N EURAL N ETWORKS F OR T ASK G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
While artiﬁcial neural networks (ANNs) have been widely adopted in machine learning, researchers are increasingly obsessed by the gaps between ANNs and biological neural networks (BNNs).
In this paper, we propose a framework named as Evolutionary Plastic Recurrent Neural Networks ( EPRNN ).
Inspired by BNN,
EPRNN composes Evolution Strategies, Plasticity Rules, and Recursion-based
Learning all in one meta learning framework for generalization to different tasks.
More speciﬁcally, EPRNN incorporates with nested loops for meta learning — an outer loop searches for optimal initial parameters of the neural network and learning rules; an inner loop adapts to speciﬁc tasks.
In the inner loop of EPRNN , we effectively attain both long term memory and short term memory by forging plasticity with recursion-based learning mechanisms, both of which are believed to be responsible for memristance in BNNs.
The inner-loop setting closely simulates that of BNNs, which neither query from any gradient oracle for optimization nor require the exact forms of learning objectives.
To evaluate the performance of
EPRNN , we carry out extensive experiments in two groups of tasks:
Sequence
Predicting , and Wheeled Robot Navigating .
The experiment results demonstrate the unique advantage of EPRNN compared to state-of-the-arts based on plasticity and recursion while yielding comparably good performance against deep learning based approaches in the tasks.
The experiment results suggest the potential of
EPRNN to generalize to variety of tasks and encourage more efforts in plasticity and recursion based learning mechanisms.
I NTRODUCTION
While Artiﬁcial Neural Networks (ANNs) were initially inspired from Biological Neural Networks (BNNs), ANNs have achieved great success in handling machine learning tasks due to the strong capacity of handling large datasets.
To learn from data, ANNs proposed to use gradient-based optimizers to minimize various loss functions as differentiable learning objectives. Compared to ANNs, BNNs are evidenced to learn through plastic rules (Gerstner et al., 1993), aka Hebb’s rule (Hebb, 1949), where the differentiable objectives are not required.
In addition to the use of gradient oracles, ANNs usually need overconsumption of datasets as training samples, acquire manual efforts in designing loss functions as the objective, and also suffer from catastrophic forgetting.
All these technical issues suggest us rethinking BNNs for further development of machine intelligence.
Recent studies show the learning mechanisms of BNNs, such as plasticity (Soltoggio et al., 2008; Najarro & Risi, 2020) and recursion (Santoro et al., 2016; Mishra et al., 2018), under appropriate metaparameter optimization, can facilitate meta-learning of ANNs for task generalization.
Unlike gradientbased methods, these mechanisms simulate the learning behaviors of BNNs in a mimic manner and don’t require any explicit-form objectives for task generalization.
More recently, authors in (Miconi et al., 2019) proposed a recurrent plastic framework for lifelong learning of ANNs, where a socalled “neuromodulated plasticity” has been used to implement Hebbian plasticity with differentiable objectives and gradient-based optimization.
Though above studies have investigated meta-learning of ANNs using the two mechanisms derived from BNNs with gradient-based methods (Miconi et al., 2019) optionally, in this work, we aim at further improving task generalization using both plasticity and recursion mechanisms, in a gradient-free fashion that ﬁts BNNs more (Lillicrap et al., 2020). Backgrounds. Though learning in BNNs has not been fully understood, some of the learning mechanisms and rules, such as plasticity (Gerstner et al., 1993) and recursion (Pollen, 2003), have been observed in brains and adopted by ANNs.
Typically, Recursion-based learning employs recurrent neural networks ( RNN ), LSTM (Hochreiter & Schmidhuber, 1997), and self-attention (Mishra et al., 2018; Chen et al., 2021) layers as learners.
Learning takes place within the feed forward pass.
The information are updated in the hidden states instead of the parameters.
Recursion are found to be extremely sample efﬁcient in generalized supervised tasks (Santoro et al., 2016), zero-shot generalization in language (Brown et al., 2020), and reinforcement learning (Mishra et al., 2018;
Chen et al., 2021) when compared with various type of gradient descent methods.
So far among those learners, though self-attention based learners such as Transformers have state-of-the-art performance, the computational costs for inference per step grow linearly with the sequence length, which makes them only available to relatively short sequences.
On the other hand, recurrent learners such as RNN and LSTM have constant per-step inference costs, but suffer form poor asymptotic performances.
That is, when sequences are getting longer, performances no longer improve or even deteriorate.
This is partly due to the limitation of the memory spaces.
For instance, an recurrent hidden layer of size n has a memory of scale O ( n ) .
In contrast, its parameters scale with O ( n 2 ) , which makes parameter-updating more powerful as learning mechanisms than recursion-only.
[Figure]
conﬁgurations of the brain in the new born life.
As the neural networks interact with the environment through actions and observations, its connections and hidden neuron states are further updated to better adapt to the environment.
Plasticity are believed to be important part of the learning mechanisms.
The ﬁtness depends on the learning and adapting capability of each individual.
In addition to recursion-based learning, evolving plasticity (Soltoggio et al., 2008; 2018;
Lindsey &
Litwin-Kumar, 2020; Yaman et al., 2021) has been proposed to reproduce the natural evolution and plasticity in simulation, as shown in Figure 1.
Implementing plasticity is not straight-forward, unlike gradient-based learning methods, plastic rules are not universal but have to be optimized beforehand, which is not possible without a further outer-loop optimizer over the inner-loop learning.
Evolutionary algorithms (Zhang et al., 2011) are typically applied in the outer loop to search for meta-parameters shaping the learning rules, which can be regarded as information carried by genomes during evolution.
Those optimized plasticity rules are then applied in the inner loop to further tune the parameter of NN for better adaptions to the environment.
Another line of works tries to bring gradient-based learning algorithms to plasticity rule optimization (Miconi et al., 2018; 2019). Although gradient-based learning approaches are regarded to be more sample efﬁcient compared with Evolutionary methods in most supervised learning cases, it is found that evolution can be more efﬁcient in cases of very long horizon in reinforcement learning (Salimans et al., 2017b). Our Works.
In this work, inspired by the previous works (Cabessa & Siegelmann, 2014; Miconi et al., 2018; 2019) that improve recursive neural networks using plastic rules for capacity of learning, [Table] Finn [Table]
Nichol et [Table] [Table] Garnelo et
[Table] [Table] Mishra et al., 2018) [Table]
Song et al., [Table] Najarro & Risi, 2020) [Table]
[Table] [Table] et al.,
[Table] [Table] we propose a novel meta-learning framework namely Evolutionary Plastic Recurrent Neural Networks ( EPRNN ) for task generalization.
Speciﬁcally, this work makes contributions as follows
• We study the problem of task generalization through simulating BNNs without the need of gradient oracles.
To the best of our knowledge, this work is the ﬁrst to address plasticity, recursion and natural evolution for meta-learning in an entirely gradient-free setting.
• The proposed EPRNN consists of nested loops for meta-learning under natural evolutions.
The inner loop constructs RNNs applying plastic rules in recurrent connections by utilizing the information of time-dependent hidden states.
The outer loop employs evolution strategies that search for proper initial parameters and plastic rules (Fig. 1) to tune the RNN.
• We carry out experiments on two groups of tasks:
Sequence Predicting and Wheeled Robot Navigating , and compare EPRNN with other BNN-inspired and gradient-based learners.
The results show EPRNN surpasses decent models under the same gradient-free settings. The most relevant works to our study are (Miconi et al., 2018; 2019; Lindsey & Litwin-Kumar, 2020; Yaman et al., 2021). Compared to (Miconi et al., 2018) that leverage gradient oracles to efﬁciently search plastic rules, the proposed EPRNN could work well in gradient-free settings (i.e., when the overall goal of meta-learning is non-differentiable).
Compared to (Lindsey & Litwin-Kumar, 2020; Yaman et al., 2021) that use evolutionary strategies to learn plastic rules, EPRNN also incorporates a RNN-based inner loop for recursion-based learning. The work (Miconi et al., 2019) also uses recursion (i.e., RNN) and differentiable plasticity in nested loops to train self-modifying neural networks, EPRNN replaces the outer loop with evolutionary strategies to generalize tasks with nondifferentiable objectives. Though EPRNN is not as competitive as gradient-based methods, which is capable of optimizing advanced neural networks with large datasets, our work still demonstrates the potential of using plasticity and recursion for meta-learning through natural evolution.
R ELATED W ORKS
Meta Learning.
Meta learning aims at building learning machines that gain experience using taskspeciﬁc data over a distribution of tasks.
Inspired by human and animal brains that are born with both embedded skills and capability of acquiring new skills, meta learning implements two nested learning loops: The outer learning loops optimize the meta that typically involves initial parameters (Finn et al., 2017; Song et al., 2019), learning rules (Zoph & Le, 2017; Najarro & Risi, 2020; Pedersen & Risi, 2021), and model structures (Soltoggio et al., 2008; Li & Malik, 2016) over a distribution of tasks; The inner learning loops adapt the model to speciﬁc task by utilizing the meta.
According to different inner-loop optimizers, we roughly classify the methods into model-based and parameter-updating methods.
The model-based methods do not update the parameters in the inner-loop, where learning is accomplished by updating the hidden states (e.g., recursion); The parameter-updating methods update the parameter in the inner-loop (e.g., gradient descent (MAML), plasticity).
From this point of view, our method can be classiﬁed into both groups.
A brief review of the typical meta learning paradigms is presented in Table 1.
Plasticity-based Learning.
The proposal of the learning mechanism of BNNs is initially raised by Hebb’s rule (Hebb, 1949), the most well-known part of which is “neurons ﬁre together wire together”.
It is further polished by Spike Time-Dependent Plasticity (STDP) (Gerstner et al., 1993) indicating that the signal of learning is dependent on the temporal patterns of the pre-synaptic spike and postsynaptic spike.
Learning could also appear in inhibitory connections, also known as anti-hebbian (Barlow, 1989).
Also, relationships between STDP and memristance are also investigated (LinaresBarranco & Serrano-Gotarredona, 2009).
Since many of those rules are related to spiking neural networks (Ghosh-Dastidar & Adeli, 2009), to apply them to ANNs, simplifed rules are proposed (Soltoggio et al., 2008) instead: given the pre-synaptic neuron state X and post-synaptic neuron state Y , the connections between X and Y are updated by
[Equation]
m is the output from neuron modulators that adjust the learning rates of plasticity. Most of the existing rules are sub-classes of Equation 1.
For instance, some works neglect the neural modulator m (Najarro & Risi, 2020; Miconi et al., 2018), {+others have set B , C , and D to 
 (Miconi+} et al., 2018; 2019).
The learned rule A, B, C, D will inevitably be dependent on initial parameter of W , however, learning plastic rules that is not dependent on the initial parameters was also investigated (Najarro &
Risi, 2020; Yaman et al., 2021).
A LGORITHMS
Problem Settings .
We consider an agent (learner) that is dependent on meta-parameter θ .
It has the capability of adapting itself to a distribution of tasks T j ∈ T by interacting with the environment
T j through observation i t and action a t .
In K-shot learning, the agent is allowed to ﬁrst observe samples of length K (this stage can be referred as meta-training-training , see Beaulieu et al.
, then its ﬁtness is calculated in meta-training-testing rollouts.
In Generalized Supervised Learning tasks ( GSL ), the observations typically include features ( x t ) and labels ( y t ) in meta-training-training stage ( i t = { x t , y t } ), and the labels are left out for predicting in the meta-training-testing stage (Santoro et al., 2016; Garnelo et al., 2018).
In Generalized Reinforcement Learning tasks ( GRL ), the observations typically include states ( s t ), actions ( a t − 1 ), and feedbacks ( r t − 1 )
( i t = { s t , a t − 1 , r t − 1 } , sometimes r t − 1 can not be observed) (Mishra et al., 2018).
The goal of meta-training is to optimize θ such that the agent achieves higher ﬁtness in meta-training-testing .
In meta-testing , similarly, the learned parameters are given meta-testing-training and meta-testing-testing in order, the performances in meta-testing-testing are evaluated.
Plastic Recurrent Neural Networks (PRNN) .
Given a sequence of observations i 1 , ..., i t , ... , we ﬁrst consider an recurrent neural network ( RNN ) that propagates forward and yields sequence of outputs a t following:
[Equation]
where h t is the hidden states at step t .
In PRNN, we kept W i stationary, but we set W t to be plastic, so that we add a subscript t to mark different W t at different steps.
Regarding h t as pre-synaptic neuron states, and h t +1 as post-synaptic neuron states, by applying Equation 1, we update W t with:
[Equation]
where we use (cid:12) and · to represent “element-wise multiplication” and “matrix multiplication” respectively.
h and 1 are column vectors.
W A , W B , W C , W D are collection of plastic rules of A, B, C, D from Equation 1, which has the same shape as W t .
m t is the neural modulators that adjusts the learning rates of plasticity.
We calculate m t by applying a neuron modulating layer denoted with:
[Equation]
A sketch of PRNN is presented in Figure 2.
The main difference between PRNN and naive RNN is that PRNN updates both the hidden states and the connection weights during the forward pass. [Figure] [Equation] [Figure]
[Equation]
Evolving PRNN .
Given task T j ∈ T , by continuously applying Equation 2 to 7 over meta-trainingtraining and meta-training-testing , the ﬁtness is eventually dependent on the initial parameters, learning rules, and the sampled task T , which is denoted as:
[Equation]
Following Evolution Strategies (ES) (Salimans et al., 2017a), in k th outer-loop iteration, we sample different tasks from T , and meta-parameters θ k,i ( i ∈ [1 , n ] ) from the neighbourhoods of θ k .
We evaluate the ﬁtness of sampled meta-parameters, and update the meta-parameters by applying:
[Equation]
Why Recurrent Neural Networks?
As stated in Equation 1, plasticity in feed-forward-only NNs allows NNs to gain experiences from single-frame observation only.
In cases of non-sequential GSL, the plasticity has chances to tune the connection weights to the speciﬁc task by relying on observing one single frame of data ( i t = { x t , y t } ), since its information of the feature and the supervision is complete.
However, in general cases, learning can not happen without summarizing sequences of observations.
For instances, a human driver getting used to a new car through continuously interacting and observing.
Moreover, in GRL, there are time lag between the observation of states and feed-backs.
Recursion helps to summarize historical observations to give the correct update for the connection weights.
Although, compared with naive RNN , there are obviously bunches of more sophisticated neural structure such as GRU and LSTM , we believe it is more desirable to start from simplest recurrent structure to study the potential of combining recursion and plasticity.
E XPERIMENTS 4. T ASKS FOR E VALUATION
In generalized tasks, we have each task T j dependent on some conﬁguration parameters that are hidden from the agent.
Below we introduce two groups of generalized tasks that we experiment on.
Sequence Predicting (Generalized Supervised Learning tasks).
We randomly generate sequences of vectors y ( t ) =
( y 1 ( t ) , ..., y l ( t )) , where y i ( t ) =
A i sin ( 2 πtn
i
+ φ i ) , t = 1 , 2 , 3 , ...
and A i ∼ U (1 , 3) , n i ∼ U (10 , 100) , φ i ∼ U (0 , 2 π ) , and U ( a, b ) represents the uniform distribution between a and b .
A , n and φ are hidden from the agent.
The front part of the sequence y (1) , y (2) , ..., y ( K ) is exposed to the agent, and the left part y ( K + 1) , y ( K + 2) , ..., y ( N ) is to be predicted. The ﬁtness
[Figure] ) Sequence Predicting (l=1,K=25,N=25)     1.51.00.50.00. [Figure] ) Sequence Predicting (
[Figure] K=25,N=25) [Figure] is the opposite of mean square error (MSE) between the predicted sequence and the ground truth.
We test the methods for comparison in four groups of tasks including ( l = 1 , K = 10 , N = 20) , ( l = 1 , K = 25 , N = 50) , ( l = 3 , K = 10 , N = 20) , and ( l = 3 , K = 25 , N = 50) (see
Figure 3(a)(b)).
Wheeled Robot Navigating (Generalized Reinforcement Learning tasks).
The agent is to navigate a two-wheeled robot to a randomly generated goal in 2-D space g =
( g x , g y ) .
We assume that there is a signal transmitter on the goal and a receiver on the robot.
The robot observes the signal intensity decided by A t =
A 0 − k · log ( d t /d 0 )
+ (cid:15) (inspired by the attenuation for electromagnetic wave, see
Friis (1946)), where d t is the current distance between the robot and the goal, (cid:15) ∼ N (0 , σ ) is the white noise in the observation.
g x , g y , A 0 , k are environment related conﬁgurations that is hidden from the agent.
For each task, we sample conﬁgurations by g x , g y ∼ U ( − 0 .
5 , 0 . 5) , A 0 ∼ U (0 . 5 , 2 . 0) , and k ∼ U (0 . 1 , 0 . 5) .
The action is the rotation speed of its two wheels that controls the orientation and velocity of the robot.
The reward at each step is r t = − d t , an episode terminates when the robot approaches the goal or steps reaches the maximum of 100.
We also hide its own position and orientation from the agent, such that the agent relies on recording its own action the signal strength
A t to locate itself.
We investigate three types of navigating circumstances with different level of noises in the observed signal: Low Noise ( σ = 0 . 01 ), Median Noise ( σ = 0 . 05 ), and High Noise ( σ = 0 . 2 ) (see Figure 3(c)).
4. E XPERIMENT S ETTINGS
We add the following methods into comparison, the methods share exactly the same outer loop and differ in the inner loop.
• ES-MAML (Song et al., 2019) :
We use four gradient descent steps in the inner loop, the learning rate of each step is treated as meta-parameters which is to be optimized by the outer loop.
As MAML can not utilize instant observation in zero-shot case, we show results of both ES-MAML (zero-shot) and ES-MAML (one-rollout) in Wheeled Robot Navigating .
Except for ES-MAML , the other methods are measured with zero-shot meta-testing score only in Wheeled Robot Navigating .
• ES-RNN : Vanilla RNN as the inner loop learner.
• ES-LSTM :
Long Short Term Memory (Hochreiter & Schmidhuber, 1997) as the inner loop learner.
• EPMLP (Soltoggio et al., 2018): Multi-Layer Perceptrons (MLP) with plasticity rules implemented.
• EPMLP (Random) (Najarro & Risi, 2020):
The main difference with EPMLP is randomly setting the parameters of the plastic layers at the beginning of each inner loop instead of using ﬁxed trainable initial parameters.
• EPRNN (w/o m) :
Removing neuron modulator ( m t ) from the plasticity rule in PRNN.
We add additional non-plastic fully connected layers before and after the plastic layers to increase the representation capability of the models.
For fairness, we kept those layers identical for all compared methods.
For every 100 outer loops in meta training , we add an extra meta testing epoch, evaluating the average ﬁtness of current meta-parameters on testing tasks.
Each run includes 15000 outer loops and 150 meta-testing epochs.
Each result is concluded from independent 3 runs.
Our coderelies on PARL 2 for parallelization.
We leave the detailed illustration of the model structures and hyper-parameters in the Appendices.
4. R ESULTS
We present the experiment results in Figure 4 (In Table 2, and Table 3, we also list the summarized performance by averaging the Top-3 meta-testing scores in the latest 10 meta-testing epochs of each run over 3 independent runs).
Generally, we can conclude that PRNN performs substantially better when compared with naive RNN .
In some cases, it even produces better results compared with LSTM , despite the simpler model architecture.
It is also interesting to notice that the gap between RNN and
PRNN are smaller in shorter sequences or low-noise environments, but larger in more challenging tasks with longer sequence or higher noise (In Wheeled Robot Navigating tasks, higher noise pushes the agent to maintain a longer memory in order to ﬁlter the noise and ﬁgure out the way to goal).
This phenomenon reafﬁrms the lack of long-term memories in RNN , and shows that PRNN signiﬁcantly improves this drawback.
Comparing EPRNN (w/o m) with RNN and EPRNN clearly demonstrates that simple ABCD rule (without the neural modulator) may also work to some extent, but introducing the neuron modulator m t can further beneﬁt the learner. [Table] Methods [Table]
− [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
− 13 .
[Table]
− 18 .
[Table]
.− 28 .
[Table]
[Table]
Among plasticity based methods, we show that recursion is more advantageous than MLP in evaluated tasks.
It is also worth noticing that EPMLP and EPMLP (Random) perform steadily beyond the gradient-based learner ( ES-MAML ).
In ES-MAML , the gradient can only be calculated after an episode is completed, while EPMLP is able to perform sequential learning even though no feedback is available during its life time.
This demonstrates the possibility of surpassing human-designed gradient-based learning rules with automatically learned unsupervised rules.
Moreover, comparison between EPMLP and EPMLP (Random) validate the proposal of Najarro & Risi (2020), implying the possibility of discovering global plastic learning rules instead of rules coupled with the initial parameters.
Yet we see optimization of the initial parameters is still advantageous, which can also be validated by evidences in nature that the newborn lives already have certain embedded skills (e [Figure] g.,
Newborn human babies have reﬂexes of suckling and grasping; Foals can stand shortly after being born).
[Figure]
To investigate whether plasticity rules update the connection weights as expected
[Figure]
we test the trained model with different tasks and record the updating trajectories of plastic connection weights W t .
As those weights are in relatively high dimension
[Figure]
we run t-SNE visualization to map W t s to 2-D space and show their trajectories in Figure 5.
We found that W t start from the same position and gradually move in different directions depending on the tasks.
The ﬁnal weights effectively capture environmental conﬁguration that was hidden from the agent.
Particularly, in Figure 5(d) for Wheeled
Robot Navigating tasks, we can see that W t captures only the signal transmission patterns ( A 0 , k ), but neglects the position of the goal ( g x , g y ). We guess that A 0 , k are important stationary patterns that helps the agent to interpret the observed signal strength ( A t ), while the absolute position of
[Figure]
l
[Figure]
[Figure] K=
[Figure]
N=50)
[Figure]
=
[Figure]
K=
[Figure]
N=50)
[Figure]
(l=
[Figure]
K=
[Figure]
N=50)
[Figure]
g=(
[Figure]
[Figure]
[Figure]
A=0.
[Figure]
k=0.1 g=(0.3, 0.3)
[Figure]
A=2.0, k=0.1 g=(0.3, 0.3), A=0.5, k=0.
g=(0.3, 0.3), A=2.0, k=0.5 g=(-0.3, -0.3), A=0.
[Figure]
k=0.1 g=(-0.3, -0.3)
[Figure]
A=2.0, k=0.
[Figure]
[Figure] [Figure] [Figure] goal is less important as its relative position to the robot is changing continuously.
This is clear that plasticity has made meaningful update on the connections weights depending on the tasks.
Note that we did not compare EPRNN with many gradient-based solutions such as differentiable plasticity (Miconi et al., 2018; 2019), since the classic Hebbian plasticity has been observed and conﬁrmed from neurons for more than half century (Hebb, 1949) — there is not evidences that the nature do differentiation or backpropagation (Lillicrap et al., 2020) to update plastic rules.
The purpose of our work is to reveal the potential of natural evolution of plasticity, and we don’t hope to claim that EPRNN could outperform gradient-based solutions on machine learning tasks.
C ONCLUSIONS
In this paper we present EPRNN a nature-inspired learning framework composed of Evolution
Strategies, Plastic rules, and Recursion.
Experiment results show that plasticity can be effectively forged with recursion to enhance the learning capability.
The proposed framework can achieve equivalent or even better performances compared with more sophisticated neural structures, by applying the simplest recurrent neural structures.
Moreover, we also show that under proper meta parameters, plasticity has a chance to surpass gradient descent methods.
We believe the learning framework of Figure 1 can be extended to more sophisticated plastic rules and model structures, uncovering better learners in the future.
Also, it would be more interesting if such framework can be validated in more complex environments such as natural language processing tasks and vision-related tasks.
Finally, we are looking forward that this work can shed light to new paradigm of building intelligent machines and inspire more efforts in this line of research.
A A PPENDICES A. M ODEL A RCHITECTURE AND T RAINING D ETAILS
To maintain fairness for comparison, the model architectures of the compared methods are similar, where the differences lie in only one of their layers (Figure 6).
We use 3 hidden layers for Sequence
Predicting tasks and 4 hidden layers for Wheeled Robot Navigating tasks.
For ES-MAML , we replace the PRNN layer with fully connected layer; For ES-RNN and ES-LSTM , we replace it with
RNN and LSTM respectively; For EPMLP , we replace it with plastic fully connected layer, where the plasticity rule is stated by Equation 1, and the neural modulator m is calculated by an additional dense layer with sigmoid activation.
The hidden sizes of all the hidden layers are 64 (for LSTM it is 64 hidden states and 64 cell states).
[Figure]
For Sequence Predicting tasks, the input observation and the output action has the dimension of l .
In the meta-training-training and meta-testing-train stages, we use the ground truth y t − 1 as input observation;
In the the meta-training-test and meta-testing-test stages, its previous action ( a t − 1 ) are taken as inputs.
For Wheeled Robot Navigating tasks, the output action is the control command of its two wheels (a length 2 vector), the input observation is the concatenation of its previous action and the current observed signal intensity ( A t ).
[Figure]
The training process is accelerated by utilizing the paralleling mechanism of PARL.
We employ 400 workers (400 Intel(R)
Xeon(R) CPU E5-2650) running inner-loops for 400 off-springs in each generation of the evolution, and additional 1 CPU to perform evolutionary update (shown in
Figure 7(b)).
It takes 3 to 12 hours for each run depending on the length of the inner loop and model architectures.
Following the previous work (Salimans et al., 2017a), we rank normalize the ﬁtness among 400 workers.
The learning rate α is set to 0 .
The mutation is performed by adding independent Gaussian noises to each parameter.
During the meta-training , we sample B different tasks and 400 meta-parameters.
Each worker run B meta-training-training and B meta-trainingtesting given the assigned meta-parameter, then the ﬁtness is averaged.
We set B = 16 for Sequence