Meta-Reinforcement Learning with Self-Modifying Networks
Anonymous Author(s)
AffiliationAddress
Abstract
Deep Reinforcement Learning has demonstrated the potential of neural networkstuned with gradient descent for solving complex tasks in well-delimited environments.
However, these neural systems are slow learners producing specializedagents with no mechanism to continue learning beyond their training curriculum.
On the contrary, biological synaptic plasticity is persistent and manifold, andhas been hypothesized to play a key role in executive functions such as workingmemory and cognitive flexibility, potentially supporting more efficient and genericlearning abilities.
Inspired by this, we propose to build networks with dynamicweights, able to continually perform self-reflexive modification as a function oftheir current synaptic state and action-reward feedback, rather than a fixed networkconfiguration.
The resulting model, MetODS (for Meta-Optimized Dynamical
Synapses) is a broadly applicable meta-reinforcement learning system able to learnefficient and powerful control rules in the agent policy space.
A single layer withdynamic synapses can perform one-shot learning, generalize navigation principlesto unseen environments and demonstrate a strong ability to learn adaptive motorpolicies.
Introduction
The algorithmic shift from hand-designed to learned features characterizing modern deep learninghas been transformative for Reinforcement Learning (RL), allowing to solve complex problemsranging from video games [1, 2] to multiplayer contests [3] or motor control [4, 5].
Yet "deep" RLhas mostly produced specialized agents unable to cope with rapid contextual changes or tasks withnovel or compositional structure [6–8].
The vast majority of models have relied on gradient-basedoptimization to learn static network parameters adjusted during a predefined curriculum, arguablypreventing the emergence of online adaptivity.
A potential solution to this challenge is to meta-learn[9–11] computational mechanisms able to rapidly capture a task structure and automatically operatecomplex feedback control: Meta-Reinforcement Learning constitutes a promising direction to buildmore adaptive artificial systems [12] and identify key neuroscience mechanisms that endow humanswith their versatile learning abilities [13].
In this work, we draw inspiration from biological fastsynaptic plasticity, hypothesized to orchestrate flexible cognitive functions according to contextdependent rules [14–17].
By tuning neuronal selectivity at fast time scales – from fast neuralsignaling (milliseconds) to experience-based learning (seconds and beyond) – fast plasticity cansupport in principle many cognitive faculties including motor and executive control.
From a dynamicalsystem perspective, fast plasticity can serve as an efficient mechanism for information storage andmanipulation and has led to modern theories of working memory [18–22].
Despite the fact that the magnitude of the synaptic gain variations may be small, such modifications are capable of profoundlyaltering the network transfer function [23] and constitute a plausible mechanism for rapidly convertingreward and choice history into tuned neural functions [24].
From a machine learning perspective, despite having a long history [25–29], fast weights have mostoften been investigated in conjunction with recurrent neural activations [30, 31] and rarely as afunction of an external reward signal or of the current synaptic state itself.
Recently, new proposalhave shown that models with dynamic modulation related to fast weights such as [32, 33] couldyield powerful meta-reinforcement learners.
In this work, we explore an original self-referentialupdate rule that allows the model to form synaptic updates conditionally on information present in itsown synaptic memory.
Additionally, environmental reward is injected continually in the model as arich feedback signal to drive the weight dynamics.
These features endow our model with a uniquerecursive control scheme, that support the emergence of a self-contained reinforcement learningprogram.
Contribution: We demonstrate that a neural network trained to continually self-modify its weightsas a function of sensory information and its own synaptic state can produce a powerful reinforcement learning program.
The resulting general-purpose meta-RL agent called ’MetODS’ (for MetaOptimized Dynamical Synapses) is theoretically presented as a model-free approach performingstochastic feedback control in the policy space.
In our experimental evaluation, we investigate thereinforcement learning strategies implemented by the model and demonstrate that a single layerwith lightweight parametrization can implement a wide spectrum of cognitive functions, from oneshot learning to continuous motor-control.
We hope that MetODS inspires more works aroundself-optimizing neural networks.
The remainder of the paper is organised as follows:
In Section 2 we introduce our mathematicalformulation of the meta-RL problem, which motivates MetODS computational principles presentedin Section 3.
In Section 4 we review previous approaches of meta-reinforcement learning and wediscuss other models of artificial fast plasticity and their relation to associative memory.
In Section5 we report experimental results in 3 different tasks.
Finally, in Section 6 we summarise the mainadvantages of MetODS and outline future work directions.
Background2.1 Notation
Throughout, we refer to “tasks" as Markov decision processes (MDP) defined by the following tupleτ =
( S , A , P , r, ρ 0 ) , where S and A are respectively the state and action sets, P : S×A×S (cid:55)→ [0 , 1]refers to the state transition distribution measure associating a probability to each tuple (state, action,new state), r : A × S
(cid:55)→ R is a bounded reward function and ρ 0 is the initial state distribution.
For simplicity, we consider finite-horizon MDP with T time-steps although our discussion can beextended to the infinite horizon case as well as partially observed MDP.
We further specify notationwhen needed by subscripting symbols with the corresponding task τ or time-step t .
Meta-Reinforcement learning considers the problem of building a program generating a distributionµ π of policies π ∈ Π that are "adapted" for a distribution µ τ of task τ ∈ T with respect to a givenmetric R . For instance, R ( τ, π ) can be the expected cumulative reward for a task τ and policy π ,i.e where state transitions are governed by s t +1 ∼ P τ ( . | s t , a t ) , actions are sampled according tothe policy π : a t ∼ π and initial state s 0 follows the distribution ρ 0 ,τ .
Provided the existence of anoptimal policy π ∗ for any task τ ∈ T , we can define the distribution measure µ π ∗ of these policiesover Π .
Arguably, an ideal system aims at associating to any task τ its optimal policy π ∗ , i.e, findingthe transport plan γ in the space Γ( µ T , µ π ∗ ) of coupling distributions with marginals µ T and µ π ∗ that maximizes R
[Equation]
where R ( τ, π ) = E π, P τ
[Equation]
Most generally, problem (1) is intractable, since µ π ∗ is unknow or has no explicit form.
Instead,previous approaches optimize a surrogate problem, by defining a parametric specialization procedurewhich builds for any task τ , a sequence ( π t ) of improving policies (see Fig. 1).
Defining θ somemeta-parameters governing the evolution of the sequences ( π t ) and µ θ ,τ,tπ the distribution measure ofthe policy π t after learning task τ during some period t , the optimization problem amounts to findingthe meta-parameters θ that best adapt π t ∼ µ θ ,τ,tπ over the task distribution.
[Equation]
Formulation (2) cast meta-Reinforcement learning as the problem of building automatic policy controlin Π .
We discuss
below
desirable properties of such control for which we show that our proposedmeta-learnt synaptic rule has a good potential.
⋄ Efficiency: How "fast" the distribution µ θ ,τ,tπ is transported towards a distribution of highperforming policies.
Arguably, an efficient learning mechanism should require few interactionsteps t with the environment to identify the task rule and adapt its policy accordingly. This can
be seen as minimizing the agent cumulative regret agent during learning [34].
This notion is also
connected to the policy exploration-exploitation trade-off [35, 36], where the agent learning program
should foster rapid discovery of policy improvements while retaining performance gains acquired
through exploration. We show that our learnt synaptic update rule is able to change the agent transfer function drastically in a few updates, and settle to a policy when task structure has been identified, thus supporting one-shot learning of a task-contingent association rule in the Harlow task, adapting amotor policy in a few steps or exploring original environments quickly in the Maze experiment.
⋄ Capacity: That property defines the learner sensitivity to particular task structures and its ability to convert them into precise state in the policy space, which determines the achievable level ofperformance for a distribution of tasks µ τ .
This notion is linked to the sensitivity of the learner, i.e how the agent captures and retains statistics and structures describing a task. Precedent work have shown that memory-based Meta-RL models operate a Bayesian task regression problem [37], and
recent work have shown that fostering task identification through diverse regularizations [38, 39, 36]
benefited performance of the optimized learning strategy.
Because our mechanism is continual, it allows for constant tracking of the environment information and policy update similar to memory- based models.
We particularly test this property in the maze experiment in Section 5, showing that tuned online synaptic updates obtain the best capacity under systematic variation of the environment.
⋄ Generality: We refer here to the overall ability of the meta-learnt policy flow to drive µ θ ,τ,tπ towardshigh-performing policy regions for a diverse set of tasks ( generic trainability ) but also to how generalis the resulting reinforcement learning program and how well it transfers to tasks unseen duringtraining ( transferability ).
In the former case.
since the proposed synaptic mechanism is model-free,it allows for tackling diverse types of policy learning, from navigation to motor control.
Arguably,to build reinforcing agents that learn in open situations, we should strive for generic and efficientcomputational mechanisms rather than learnt heuristics.
Transferability corresponds to the abilityof the meta-learned policy flow to generally yield improving updates even in unseen policy regionsof the space Π or conditioned by unseen task properties: new states, actions and transitions, newreward profile.
etc.
We show in a motor-control experiment using the Meta-World benchmark thatmeta-tuned synaptic updates are a potential candidate to produce a more systematic learner agnostic toenvironment setting and reward profile.
The generality property remains the hardest for current metaRL approaches, demonstrating the importance of building more stable and invariant control principles.
MetODS: Meta-Optimized Dynamical Synapses
[Figure] 3.1 Learning weights updates [Figure]
What if a neural agent could adjust the ruledriving the evolution of µ θ ,τ,tπ based on its ownknowledge? In this work, we test a neural computational model that learns to compress experience of a task τ into its dynamic weights W t .
[Equation]
Contrary to gradient-based rules whose analytical expression is a static and predefined functionof activations error ( ∆( W t ) = ∂ a ∂ W . ∂ L ∂ a ), wedefine an update rule that directly depends onthe current weight state through a parameterizednon-linear recursive scheme, making the expression of the update much more sensitive to weightcontent itself.
[Equation]
This self-reflexive property of the synaptic configuration W t allows for online variation of thelearning rule during adaptation for a given task τ and opens an original repertoire of dynamics for thepolicy evolution µ θ ,τ,tπ . Specifically, at every time-step t , network inference and learning consists inrecursive applications of read and write operations that we define below.⋄ Read-write operations: The core mechanism consists in two simple operations that respectivelylinearly project neurons activation v through the dynamic weights W followed by an element-wisenon-linearity, and build a local weight update with element-wise weighting α :
[Equation]
Here, α is a matrix of R N × N , ⊗ denotes the outer-product, ⊙ is the element-wise multiplication, σ isa non-linear activation function.
The read and write operations are motivated by biological synapticcomputation: Reading correspondd to the non-linear response of the neuron population to a specificactivation pattern.
Writing consists in an outer product that emulates a local Hebbian rule betweenneurons.
The element-wise weighting α allows locally tuning synaptic plasticity at every connection consistent with biology [40, 41] which generates a matrix update with potentially more than rank oneas in the classic hebbian rule.⋄ Recursive update: While a single iteration of these two operations can only retrieve a similaractivation pattern (reading) or add unfiltered external information into weights (writing), recursivelyapplying these operations offers a much more potent computational mechanism that mix informationbetween current neural activation and previous iterates.
Starting from an initial activation pattern v (0)and previous weight state W (0) =
W t − 1 , the model recursively applies equations (5) for s ∈ [1 , S ]on v ( s ) and W ( s ) such that:
[Equation]
Parameters κ ( l ) s and β ( l ) s are scalar values learnt along with plasticity parameters α , and correspondto delayed contributions of previous patterns and synaptic states to the current operations.
This ismotivated by biological evidence of complex cascades of temporal modulation mechanisms oversynaptic efficacy [42, 43].
Finally, ( v ( S ) , W ( S ) ) are respectively used as activation for the next layer,and as the new synaptic state W t .⋄
Computational interpretation: We note that if S=1 in equation (6), the operation boils down to asimple hebbian update with a synapse-specific weighting α i,j .
This perspective makes MetODS anoriginal form of modern Hopfield network [44] with hetero-associative memory that can dynamicallyaccess and edit stored representations driven by observations, rewards and actions. While patternretrieval from Hopfield networks has a dense litterature, our recursive scheme is an original proposalto learn automatic updates able to articulate representations across timesteps. We believe that thismechanism is particularly benefitial for meta-RL to filter external information with respect to pastexperience. The promising results shown in our experimental section suggest that such learnt updatescan generate useful self-modifications to sequentially adapt to incoming information at runtime.
Algorithm
1: Require: θ = [ f , g , α , κ , β ] and W2: for 1 ≤ t ≤ T do 3: v (0) ← f ( s t , a t − 1 , r t − 1 ) 4: W (0) ← W t −5: for 1 ≤ s ≤ S do 6: v ( s ) ← s − 1 (cid:80) l =0 κ ( l ) s v ( l ) + κ ( s ) s σ ( W ( s − 1) . v ( s − 1) ) 7:
W ( s ) ← s −(cid:80) l =0 β ( l ) s W ( l )
+ β ( s ) s (cid:0) α ⊙ v ( s − 1) ⊗ v ( s − 1) (cid:1) 8: end for 9: a t , v t ← g ( v ( s ) ) 10:
W t ← W ( s ) 11: end for In this work, we test a single dynamic layerfor MetODS and leave the extension ofthe synaptic plasticity to the full networkfor future work.
In order for the modelto learn a credit assignment strategy, statetransition and previous reward information[ s t , a t − 1 , r t − 1 ] are embedded into a vectorv t by a feedforward map f as in previousmeta-RL approaches [45, 46].
Action andadvantage estimate are read-out by a feedforward policy map g .
We meta-learn theplasticity and update coefficients, as well asthe embedding and read-out function altogether:
θ =
[ f , g , α , κ , β ] .
Additionally,the initial synaptic configuration W 0 canbe learnt, fixed a priori or sampled from anspecified distribution.
Related work
⋄ Meta-Reinforcement learning has recently flourished into several different approaches aiming atlearning high-level strategies for capturing task rules and structures.
A direct line of work consists inautomatically meta-learning components or parameters of the RL arsenal to improve over heuristicsettings [47–49].
Orthogonally, work building on the Turing-completeness of recurrent neuralnetworks has shown that simple recurrent neural networks can be trained to store past information in their persistent activity state to inform current decision, in such a way that the network implements aform of reinforcement learning over each episode [50, 45, 51].
It is believed that vanilla recurrentnetworks alone are not sufficient to meta-learn the efficient forms of episodic control found inbiological agents [52, 13].
Hence additional work has tried to enhance the system with a betterepisodic memory model [53–55] or by modeling a policy as an attention module over an explicitelystored set of past events [46].
Optimization based approaches have tried to cast episodic adaptationas an explicit optimization procedure either by treating the optimizer as a black-box system [56, 57]or by learning a synaptic configuration such that one or a few gradient steps are sufficient to adapt theinput/output mapping to a specific task [58].⋄ Artificial fast plasticity: Networks with dynamic weights that can adapt as a function of neuralactivation have shown promising results over regular recurrent neural networks to handle sequentialdata [59, 29, 60, 61, 31, 62].
However, contrary to our work, these models postulate a persistentneural activity orchestrating weights evolution.
On the contrary, we show that synaptic states arethe sole persistent components needed to perform fast adaptation.
Additionally, the possibility ofoptimizing synaptic dynamics with evolutionary strategies in randomly initialized networks [63]or through gradient descent [64] has been demonstrated, as well as in a time-continuous setting[65].
Recent results have shown that plasticity rules differentially tuned at the synapse level allowto dynamically edit and query networks memory [60, 66].
However another specificity of this workis that our model synaptic rule is a function of reward and synaptic state, allowing to drive weightdynamics conditionally on both an external feedback signal and the current model belief.⋄ Associative memory: As discussed above, efficient memory storage and manipulation is a crucialfeature for building rapidly learning agents.
To improve over vanilla recurrent neural network policies[45], some models have augmented recurrent agents with content-addressable dictionaries able toreinstate previously encoded patterns given the current state [67–69, 13].
However these slot-basedmemory systems are subject to interference with incoming inputs and their memory cost growslinearly with experience.
Contrastingly, attractor networks can be learnt to produce fast compressionof sensory information into a fixed size tensorial representation [70, 71].
One class of such networkare Hopfield networks [72–75] which benefit from a large storage capacity [75], can possibly performhetero-associative concept binding [76, 62] and produce fast and flexible information retrieval [44].
Experiments
In this section, we explore the potential of our meta-learnt synaptic update rule with respect to thethree properties of the meta-RL problem exposed in section 2.
Namely, 1) efficiency , 2) capacity and3) generality of the produced learning algorithm.
We compare it with three state-of-the-art meta-RLmodels based on different adaptation mechanisms: RL 2 [45] a memory-based algorithm based ontraining a GRU-cell to perform Reinforcement Learning within the hidden space of its recurrent unit, MAML [58] which performs online gradient descent on the weights of three-layer MLP with ReLUor hyperbolic tangeant non-linearities, and PEARL [77], which performs probabilistic task inferencefor conditioning policy.
Details for each experimental setting are further discussed in S.I.
5.1 Efficiency: One-shot reinforcement learning and rapid motor control
To first illustrate that learnt synaptic dynamics can support fast behavioral adaptation, we use aclassic experiment from the neuroscience literature originally presented by Harlow [78] and recentlyreintroduced in artificial meta-RL in [51] as well as a heavily-benchmarked MuJoCo directionallocomotion task (see Fig. 3).
To behave optimally in both settings, the agent must quickly identify thetask rule and implement a relevant policy : The Harlow task consists of five sequential presentationsof two random variables placed on a one-dimensional line with random permutation of their positionsthat an agent must select by reaching the corresponding position. One value is associated with apositive reward and the other with a negative reward. The five trials are presented in alternance withperiods of fixation where the agent should return to a neutral position between items.
In the MuJoCo robotic Ant-dir experiment, a 4-legged agent with must produce a locomotive policy given a randomrewarded direction.⋄ Harlow : Since values location are randomly permuted across presentations within one episode, thelearner cannot develop a mechanistic strategy to reach high rewards based on initial position.
Instead,to reach the maximal expected reward over the episode, the agent needs to perform one-shot learningof the task-contingent association rule during the first presentation.
We found that even a very smallnetwork of N=20 neurons proved to be sufficient to solve the task perfectly.
We investigated thesynaptic mechanism encoding the agent policy.
A principal component analysis reveals a policydifferentiation with respect to the initial value choice outcome supported by synaptic specializationin only a few time-steps (see Figure 1-c). We can further interpret this adaptation in terms of sharpmodifications of the Hopfield energy of the dynamic weights (Figure 3-e. Finally, the largest synapticvariations measured by the sum of absolute synaptic variations occur for states that carry a non-nullreward signal (see S.I).
Theses results suggest that the recursive hebbian update combined withreward feedback is sufficient to support one-shot reinforcement learning of the task association rule.
⋄ MuJoCo Ant- dir : We trained models for performing the task on an episode of lenght 200 stepsand found that MetODS can adapt in a few time-steps, similar to memory-based models such as RL 2 ,(Figure 3-f) thanks to its continual adaptation mechanism.
By design, MAML and PEARL do notpresent such a property, and they need multiple episodes before being able to perform adaptationcorrectly.
We still report MAML performance after running its gradient adaptation at timestep t=100.
We further note that our agent overall performance in a single episode is still superior to MAMLperformance reported in [58, 46] when more episodes are accessible for training.
5.2 Capacity : Maze exploration task
We further tested MetODS learnt reinforcement program systematicity on a partially observable
Markov decision process (POMDP): An agent must locate a target in a randomly generated maze while starting from random locations and observing a small portion of its environment (depicted in
Figure 4). While visual navigation has been previously explored in meta-RL [45, 46, 60], we herefocus on the mnemonic component of navigation by complexifying the task in two ways, we reducethe agent’s visual field to a small size of 3x3 pix.
and randomize the agent’s position after everyreward encounter.
The agent can take discrete actions in the set { up,down,left,right } which movesit accordingly by one coordinate.
The agent’s reward signal is solely received by hitting the targetlocation, thus receiving a reward of 10 .
Each time the agent hits the target, its position is randomlyreassigned on the map (orange) and the exploration resumes until 100 steps are accumulated duringwhich the agent must collect as much reward as possible.
Note that the reward is invisible to theagent, and thus the agent only knows it has hit the reward location because of the activation of thereward input.
The reduced observability of the environment and the sparsity of the reward signal(most of the state transitions yield no reward) requires the agent to perform logical binding betweendistant temporal events to navigate the maze.
Again, this setting rules out PEARL since its latentcontext encoding mechanism erases crucial temporal dependencies between state transitions forefficient exploration.
Despite having no particular inductive bias for efficient spatial exploration orpath memorization, a strong policy emerges spontaneously from training.
Agent 1st rew.
[Equation] ( ↓ )
Success ( ↑ )
Cum
[Equation]
Rew
[Equation]
Cum
[Equation]
Rew (Larger) ( ↑ )
Random 96.8 ± 0.
[Figure]
⋄ Ablation study and variations: We explored the contribution of the different features combined in MetODS update rule, showing that they all contribute to the final performance of our meta-learningmodel. First, we tested the importance of the element-wise tuning of plasticity in weight-basedlearning models and note that while it adversely affects MAML gradient update, it greatly improves
MetODS performance, suggesting different forms of weights update.
Second, we verified thataugmenting recursivity depth S was beneficial to performance, consistently with Harlow results. Third, we noted that transforming the rightmost vector of the writing equation in 5 with a linearprojection ( Linear in figure 4, see S.I for full experimental details) yields major improvementwhile non-linear ( MLP ) alters performance.
Finally, we additionally test the capability of the learnt navigation skills to generalize to a larger maze size of 10 × 10 pix. unseen during training. We showthat MetODS is able to retain its advantage (see figure 4 and table 5 for results).
5.3 Generality : Motor control
Finally, we test the generality of the reinforcement learning program learnt by our model for differentcontinuous control tasks:
MetaWorld:
First, we use the dexterous manipulation benchmark proposed in [80] using thebenchmark suite [81], in which a Sawyer robot is tasked with diverse operations.
A full adaptationepisode consists in N=10 rollouts of 500 timesteps of the same task across which dynamic weightsare carried over.
Observation consists in the robot’s joint angles and velocities, and the actions are itsjoint torques.
We compare MetODS with baseline methods in terms of meta-training and meta-testingsuccess rate for 3 settings, push, reach and ML-10 .
We show in Fig.
6 the meta-training resultsfor all the methods in the MetaWorld environments.
Due to computational resource constraints,we restrict our experiment to a budget of 10M steps per run.
While we note that our benchmarkdoes not reflect the final performance of previous methods reported in [80] at 300M steps, we notethat MetODS test performance outperforms these methods early in training and keeps improving at10M steps, potentially leaving room for improvement with additional training.
(see S.I for additionaldiscussion).
Finally, we note that all tested approaches performed modestly on ML10 for test tasks,which highlights the limitation of current methods.
We conjecture that this might be due to theabsence of inductive bias for sharing knowledge between tasks or fostering systematic explorationof the environment of the tested meta-learning algorithms.
Robot impairment: We also tested the robustness of MetODS learnt reinforcement programsby evaluating the agent ability to perform in a setting not seen during training: specifically, whenpartially impairing the agent motor capabilities.
We adopt the same experimental setting as section5.1 for the Ant and Cheetah robots and evaluate the performance when freezing one of the robotstorque.
We show that our model policy retains a better proportion of its performance compared toother approaches.
These results suggest that fast synaptic dynamics are not only better suited tosupport fast adaptation of a motor policy in the continuous domain, but also implement a more robustreinforcement learning program when impairing the agent motor capabilities.
Discussion
In this work, we introduce a novel meta-RL system, MetODS, which leverages a self-referentialweight update mechanism for rapid specialization at the episodic level.
Our approach is generic andsupports discrete and continuous domains, giving rise to a promising repertoire of skills such as oneshot adaptation, spatial navigation or motor coordination.
MetODS demonstrate that local synapticupdate whose expression depends directly on the network configuration can be meta-learnt for solvingcomplex reinforcement learning tasks.
We conjecture that further tuning the hyperparameters aswell as combining MetODS with more sophisticated reinforcement learning techniques can boostits performance.
Generally, the success of the approach provides evidence for the benefits of fastplasticity in artificial neural networks, and the exploration of self-referential networks.