Introducing Symmetries to Black Box Meta Reinforcement Learning
Abstract
Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction.
In so-called black-box approaches, the policy and the learning algorithm are jointly represented by a single neural network.
These methods are very ﬂexible, but they tend to underperform in terms of generalisation to new, unseen environments.
In this paper, we explore the role of symmetries in meta-generalisation.
We show that a recent successful meta RL approach that meta-learns an objective for backpropagation-based learning exhibits certain symmetries (speciﬁcally the reuse of the learning rule, and invariance to input and output permutations) that are not present in typical black-box meta
RL systems.
We hypothesise that these symmetries can play an important role in meta-generalisation.
Building off recent work in black-box supervised meta learning [12], we develop a black-box meta RL system that exhibits these same symmetries.
We show through careful experimentation that incorporating these symmetries can lead to algorithms with a greater ability to generalise to unseen action & observation spaces, tasks, and environments.
Introduction
Recent work in meta reinforcement learning (RL) has begun to tackle the challenging problem of automatically discovering general-purpose RL algorithms [13, 1, 17].
These methods learn to reinforcement learn by optimizing for earned reward over the lifetimes of many agents in multiple environments.
If the discovered learning principles are sufﬁciently general-purpose, then the learned algorithms should generalise to novel environments.
Depending on the structure of the learned algorithm, these methods can be partitioned into backpropagation-based methods, which learn to use the backpropagation algorithm to reinforcement learn and black-box-based methods, in which a single (typically recurrent) neural network jointly speciﬁes the agent and RL algorithm [25, 6].
While backpropagation-based methods are more prevalent due to their relative ease of implementation and theoretical guarantees, black-box methods can be more expressive and have the potential to avoid some of the issues with backpropagation-based optimization, such as memory requirements, catastrophic forgetting, and differentiability.
Unfortunately, black-box methods have not yet been successful at discovering general-purpose
RL algorithms.
In this work, we show that black-box methods exploit fewer symmetries than backpropagation-based methods.
We hypothesise that introducing more symmetries to black-box meta-learners can improve their generalisation capabilities.
We test this hypothesis by introducing a number of symmetries into an existing black-box meta learning algorithm, including (1) the use of the same learned learning rule across all nodes of the neural network (NN), (2) the ﬂexibility to work with any input, output, and architecture sizes, and (3) invariance to permutations of the inputs
[Figure]
and outputs (for dense layers). Permutation invariance implies that for any permutation of inputs and outputs the learning algorithm produces the same policy.
We refer to such agents as symmetric learning agents (SymLA).
To introduce these symmetries, we build on variable shared meta learning (VSML) [12], which we adapt to the RL setting.
VSML arranges multiple RNNs like weights in a NN and performs message passing between these RNNs.
We then perform meta training and meta testing similar to black-box MetaRNNs [25, 6].
We experimentally validate SymLA on bandits, classic control, and grid worlds, comparing generalisation capabilities to MetaRNNs.
SymLA improves generalisation when varying action dimensions, permuting observations and actions, and signiﬁcantly changing tasks and environments.
Preliminaries: Meta Reinforcement Learning
The RL setting in this work follows the standard (PO)MDP formulation.
At every time step, t = 1 , 2 , . . .
the agent receives a new observation o t ∈ O generated from the environment state s t ∈ S and performs an action a t ∈ A sampled from its (recurrent) policy π θ = p ( a t | o 1: t , a 1: t − 1 ) .
The agent receives a reward r t ∈ R ⊂ R and the environment transitions to the next state.
This transition is deﬁned by the environment dynamics e = p ( s t +1 , r t | s t , a t ) .
The initial environment state s 1 is sampled from the initial state distribution p ( s 1 ) .
The goal is to ﬁnd the optimal policy parameters θ ∗ that maximise
the expected return R = E
[ (cid:80) Tt =1 γ t r t ] where T is the episode length, and
0 < γ ≤is a discount factor ( T = ∞ , γ < 1 for non-episodic MDPs).
The meta reinforcement learning setting is concerned with discovering novel agents that learn throughout their multi-episode lifetime ( L ≥ T ) by making use of rewards r t to update their behavior.
This can be formulated as maximizing E e ∼ p
( e ) [
E [ (cid:80) Lt =1 γ t r t ]] where p ( e ) is a distribution of metatraining environments.
The objective itself is similar to a multi-task setting.
In this work, we discuss how the structure of the agent inﬂuences the degree to which it learns and generalises in novel tasks and environments.
We seek to discover general-purpose learning algorithms that generalise outside the meta-training distribution.
We can think of an agent that learns throughout its lifetime as a history-dependent map a t , h t = f
( h t − 1 , o t , r t − 1 , a t − 1 ) that produces an action a t and new agent state h t given its previous state h t − 1 , an observation o t , environment reward r t − 1 , and previous action a t − 1 .
In the case of backpropagationbased learning, f is decomposed into: (1) a stationary policy π ( s ) θ that maps the current state into an action, a t = π
( s ) θ ( o t ) ; and (2) a backpropagation-based update rule that optimizes a given objective
J by propagating the error signal backwards and updating the policy in ﬁxed intervals (e.g. after each episode).
In its simplest form, for any dense layer k ∈ { 1 , . . .
, K } of a NN policy with size
A ( k ) × B ( k ) , inputs x ( k ) , outputs x ( k +1) , and weights w ( k ) ⊂ θ , the backpropagation update rule is given by
[Equation]
where a ∈ { 1 , . . .
A ( k ) } , b ∈ { 1 , . . .
, B ( k ) } , α is the learning rate, δ are error terms, and the agent state h corresponds to parameters θ .
The initial error is given by the gradient at the NN outputs, δ ( k ) =
[Equation]
Transformations such as non-linearities are omitted here.
Works in metareinforcement learning that take this approach parameterise the objective J φ and meta-learn its parameters [13, 17].
In contrast, black-box meta RL [6, 25] meta-learns f directly in the form of a single non-stationary policy π θ with memory.
Parameters of f represent the learning algorithm (no explicit J φ ) while the state h represents the policy.
In the simplest form of an RNN representation of f , given a current hidden state h and inputs o, r, a (concatenated [ · ] ), updates to the policy take the form
[Equation]
with parameters θ = v and activation function σ , omitting the bias term.
We refer to this as the
MetaRNN.
The inputs must include, beyond the observation o , the previous reward r and action a , so that the meta-learner can learn to associate past actions with rewards [21, 25].
Further, black-box systems do not reset the state h between episode boundaries, so that the learning algorithm can accumulate knowledge through the agent’s lifetime.
For additional related work, refer to Appendix A.
Symmetries in Meta RL
In this section, we demonstrate how the learning dynamics in backpropagation-based systems (Equation 3) differ from the learning dynamics in black-box systems (Equation 4), and how this affects the generalisation of black-box methods to novel environments.
Symmetries in backpropagation-based Meta RL We ﬁrst identify three symmetries that backpropagation-based systems exhibit and discuss how they affect the generalisability of the learned learning algorithms.
Symmetric learning rule.
In Equation 3, each parameter w ab is updated by the same update rule based on information from the forward and backward pass.
Meta-learning an objective J φ affects the updates of each parameter symmetrically through backpropagation.
Flexible input, output, and architecture sizes.
Because the same rule is applied everywhere, the learning algorithm can be applied to arbitrarily sized neural networks, including variations in input and output sizes.
This involves varying A and B and the number of layers, affecting how often the learning rule is applied and how many parameters are being learned.
Invariance to input and output permutations.
Given a permutation of inputs and outputs in a layer, deﬁned by the bijections ρ : N → N and ρ (cid:48) :
N → N , the learning rule is applied as x ( k +1) ρ (cid:48)
( b ) = (cid:80) a x ( k ) ρ ( a ) w ( k ) ab , δ ( k − 1)
ρ ( a ) =
(cid:80)
b δ ( k ) ρ (cid:48) ( b ) w ( k ) ab , and ∆ w ( k ) ab = − αx ( k ) ρ ( a ) δ ( k ) ρ (cid:48)
( b ) .
Let w (cid:48) be a weight matrix with w (cid:48)
( k ) ρ ( a ) ρ (cid:48) ( b ) =
w ( k ) a,b , then we can equivalently write x ( k +1) ρ (cid:48) ( b ) = (cid:80) a x ( k ) ρ ( a ) w (cid:48)
( k ) ρ ( a ) ρ (cid:48) ( b ) , δ ( k − 1) ρ
( a ) = (cid:80)
b δ ( k ) ρ (cid:48) ( b ) w (cid:48) ( k ) ρ
( a ) ρ (cid:48) ( b ) , and ∆ w (cid:48) ( k ) ρ ( a ) ρ (cid:48) ( b ) = − αx ( k ) ρ ( a ) δ ( k ) ρ (cid:48)
( b ) .
If all elements of w (cid:48) ( k ) are initialized i.i.d.
, we can interchangeably use w in place of w (cid:48) in the above updates.
By doing so, we recover the original learning rule equations for any a, b .
Thus, the learning algorithm is invariant to input and output permutations.
While backpropagation has inherent symmetries, these symmetries would be violated if the objective function J φ would be asymmetric.
Formally, when permuting the NN outputs y = x ( K +1)
such that y (cid:48) b =
y ρ (cid:48) ( b ) , J φ should satisfy that the gradient under the permutation is also a permutation
[Equation]
( b ) where the environment accepts the action permuted by ρ (cid:48) in the case of
J φ ( y (cid:48) ) .
This is the case for policy gradients, for instance, if the action selection π ( a | s ) is permuted according to ρ (cid:48) .
When meta-learning objective functions, prior work carefully designed the objective function J φ to be symmetric.
In MetaGenRL [13], taken actions were processed element-wise with the policy outputs and sum-reduced by the loss function.
In LPG [17], taken actions and policy outputs were not directly fed to J φ , but instead only the log probability of the action distribution was used.
Insufﬁcient Symmetries in Black-box Meta RL Black-box meta learning methods are appealing as they require few hard-coded biases and are ﬂexible enough to represent a wide range of possible learning algorithms.
We hypothesize that this comes at the cost of the tendency to overﬁt to the given meta training environment(s) resulting in overly specialized learning algorithms.
Learning dynamics in backpropagation-based systems (Equation 3) differ signiﬁcantly from learning dynamics in black-box systems (Equation 4).
In particular, meta-learning J φ is signiﬁcantly more constrained, since J φ can only indirectly affect each policy parameter w ( k ) ab through the same learning rule from Equation 3.
In contrast, in black-box systems (Equation 4), each policy state h b is directly controlled by unique meta-parameters (vector v · b ), thereby encouraging the black-box meta-learner to construct speciﬁc update rules for each element of the policy state.
This results in sensitivity to permutations in inputs, outputs, and a general tendency to construct non-learning, biased solutions, over learning solutions.
Furthermore, input and output spaces must retain the same size as those are directly dependent on the number of RNN parameters.
Adding Symmetries to Black-box Meta RL
A solution to the illustrated over-ﬁtting problem with black-box methods is the introduction of symmetries into the parameterisation of the policy.
This can be achieved by generalising the forward pass (Equation 1), backward pass (Equation 2), and element-wise update (Equation 3) to parameterized versions.
We further subsume the loss computation into these parameterized update rules.
Together, they form a single recurrent policy with additional symmetries.
Prior work on variable shared meta learning (VSML) [12] used similar principles to meta-learn supervised learning algorithms.
In the following, we extend their approach to deal with the RL setting.
Variable Shared Meta Learning VSML describes neural architectures for meta learning with parameter sharing.
This can be motivated by meta learning how to update weights [4, 22] where the update rule is shared across the network.
Instead of designing a meta network that deﬁnes the weight updates explicitly, we arrange small parameter-shared RNNs (LSTMs) like weights in a NN and perform message passing between those.
In VSML, each weight w ab with w ∈ R A × B in a NN is replaced by a small RNN with parameters θ and hidden state h ab ∈ R N .
We restrict ourselves to dense NN layers here, where w corresponds to the weights of that layer with input size A and output size B .
This can be adapted to other architectures such as CNNs if necessary.
All these RNNs share the same parameters θ , deﬁning both what information propagates in the neural network, as well as how states are updated to implement learning.
Each RNN with state h ab receives the analogue to the previous activation, here called the vectorized forward message −→ m a ∈ R −→ M , and the backward message ←− m b ∈ R ←− M for information ﬂowing backwards in the network (asynchronously).
The backward message may contain information relevant to credit assignment, but is not constrained to this.
The RNN update equation (compare
Equation 3 and 4) is then given by
[Equation]
) for layer k where k ∈ { 1 , . . . , K } and a ∈ { 1 , . . .
A ( k ) } , b ∈ { 1 , . . .
, B ( k ) } .
Similarly, new forward messages are created by transforming the RNN states using a function f −→ m : R N → R −→ M (compare
Equation 1) such that
[Equation]
deﬁnes the new forward message for layer k + 1 with b ∈ { 1 , . . .
, B ( k ) =
A ( k +1) } .
The backward message is given by f ←− m : R N → R ←− M (compare Equation 2) such that
[Equation]
and a ∈ { 1 , . . .
, A ( k ) =
B ( k − 1) } .
For simplicity, we use θ below to denote all of the VSML parameters, including those of the RNN and forward and backward message functions.
In the following, we derive a black-box meta reinforcement learner based on VSML (visualized in
Figure 1).
[Figure]
θ h ( t ) h ( t +1)
θ ( i )
o t , r t −o t +1 , r t a t a t
+ θ ( i ) θ
( i ) θ ( i ) θ
( i +1)
Based on feedback r t , RNN states can be used as memory for learning.
The learning algorithm encoded in the RNN parameters θ is updated in the outer loop by meta-training using ES.
RL Agent Inputs and Outputs
At each time step in the environment, the agent’s inputs consist of the previously taken action a t − 1 , current observation o t and previous reward r t − 1 .
We feed r t − 1 as an additional input to each RNN, the observation o t ∈ R A (1) to the ﬁrst layer ( −→ m (1) · 1 := o t ), and the action a t − 1 ∈ { 0 , 1 } B ( K ) (onehot encoded) to the last layer ( ←− m ( K ) ·:= a t − 1 ).
The index 1 refers to the ﬁrst dimension of the −→ M or ←− Mdimensional message.
We interpret the agent’s output message y =
−→ m ( K +1) ·as the unnormalized logits of a categorical distribution over actions.
While we focus on discrete actions only in our present experiments, this can be adapted for probabilistic or deterministic continuous control.
Architecture Recurrence and Reward Signal Instead of using multiple layers ( K > 1 ), in this paper we use a single layer ( K = 1 ).
In Equation 5, RNNs in the same layer can not coordinate directly as their messages are only passed to the next and previous layer.
To give that single layer sufﬁcient expressivity for the RL setting, we make it ‘recurrent’ by processing the layer’s own messages −→ m ( k +1) b and ←− m ( k − 1) a .
The network thus has two levels of recurrence: (1) Each RNN that corresponds to a weight of a standard NN and (2) messages that are generated according to Equation 6 and 7 and fed back into the same layer.
Furthermore, each RNN receives the current reward signal r t − 1 as input.
The update equation is given by
[Equation]
where a ∈ { 1 , . . .
A ( k ) } , b ∈ { 1 , . . .
, B ( k ) } .
As we only use a single layer, k = 1 , we apply the update multiple times (multiple micro ticks) for each step in the environment.
This can also be viewed as multiple layers with shared parameters, where parameters correspond to states h .
For pseudo code, see Algorithm 1 in the appendix.
Symmetries in SymLA By incorporating the above changes to inputs, outputs, and architecture, we arrive at a black-box meta RL method with symmetries, here represented by our proposed symmetric learning agents (SymLA).
By construction, SymLA exhibits the same symmetries as those described in Section 3, despite not using the backpropagation algorithm.
Symmetric learning rule.
The learning rule as deﬁned by Equation 8 is replicated across a ∈ { 1 , . . .
, A } and b ∈ { 1 , . . .
, B } with the same parameter θ .
Flexible input, output, and architecture sizes.
Changes in A , B , and K correspond to input, output, and architecture size.
This does not affect the number of meta-parameters and therefore these quantities can also be varied at meta-test time.
Invariance
to input and output permutations.
When permuting messages using bijections ρ and ρ (cid:48) , the state update becomes h ( k ) ab ← f RNN ( h ( k ) ab , −→ m ( k ) ρ ( a ) ,
←− m ( k ) ρ (cid:48) ( b ) , r t − 1 , −→ m ( k +1) ρ (cid:48) ( b ) , ←− m ( k − 1) ρ ( a ) ) , and the message transformations are −→ m ( k +1)
ρ (cid:48) ( b ) = (cid:80) a f −→ m ( h ( k ) ab ) and ←− m ( k − 1) ρ
( a ) = (cid:80) b
f ←− m ( h ( k ) ab ) .
Similar to backpropagation, when RNN states h ab are initialized i.i.d.
, we can use h ρ ( a ) ,ρ (cid:48)
( b ) in place of h ab to recover the original Equations 6, 7, 8.
Learning / Inner Loop Learning corresponds to updating RNN states h ab (see Figure 2).
This is the same as the MetaRNN [25, 6] but with a more structured neural model.
For ﬁxed RNN parameters θ which encode the learning algorithm, we randomly initialize all states h ab .
Next, the agent steps through the environment, updating h ab in each step.
If the environment is episodic with T steps, the agent is run for a lifetime of L ≥ T steps with environment resets in-between, carrying the agent state h ab over.
Meta Learning / Outer Loop Each outer loop step unrolls the inner loop for L environment steps to update θ .
The SymLA objective is to maximize the agent’s lifetime sum of rewards,
i.e. (cid:80) Lt =1 r t ( θ ) .
We optimize this objective using evolutionary strategies [26, 18] by following the gradient
[Equation]
with some ﬁxed diagonal covariance matrix Σ and environments e ∼ p ( e ) .
We chose evolution strategies due to its ability to optimize over long inner-loop horizons without memory constraints that occur due to backpropagation-based meta optimization.
Furthermore, it was shown that meta-loss landscapes are difﬁcult to navigate and the search distribution helps in smoothing those [14].
Experiments
[Table]
[Figure]
U n i f I n d e p
[Figure]
E a s y U
n
i f D e p .
M e d i u m H a r d
U n i f I n d e p .
E a s y U
n
i f D e p .
M e d i u m H a r d
U n i f I n d e p .
E a s y U
n
i f D e p .
M e d i u m H a r d where the meta-training environments are similar to the meta-test environments.
Secondly, we demonstrate generalisation to unseen action spaces, applying the learned algorithm to bandits with varying numbers of arms at meta-test time—something that MetaRNNs are not capable of.
Thirdly, we demonstrate how symmetries improve generalisation to unseen observation spaces by creating permutations of observations and actions in classic control benchmarks.
Fourthly, we show how permutation invariance leads to generalisation to unseen tasks by learning about states and their associated rewards at meta-test time.
Finally, we demonstrate how symmetries result in better learning algorithms for unseen environments, generalising from a grid world to
CartPole.
Hyper-parameters are in Appendix C.
[Figure]
r a i n
[Figure]
Learning to Learn on Similar Environments We ﬁrst compare SymLA and the MetaRNN on the two-armed (dependent) bandit experiments from Wang et al.
[25] where there is no large variation in the meta-test environments.
These consist of ﬁve different settings of varying difﬁculty that we use for meta-training and meta-testing (see Appendix B).
There are no observations (no context), only two arms, and a meta-training distribution where each arm has the same marginal distribution of payouts.
Thus, we expect the symmetries from SymLA to have no signiﬁcant effect on performance .
We meta-train for an agent lifetime of L = 100 arm-pulls and report the expected cumulative regret at meta-test time in Figure 3.
We metatrain on each of the ﬁve settings, and meta-test across all settings.
The performance of the MetaRNN reproduces the average performance of Wang et al.
[25], here trained with ES instead of A2C. When using symmetries (as in
SymLA), we recover a similar performance compared to the MetaRNN.
Generalisation to Unseen Action Spaces In contrast to the MetaRNN, in SymLA we can vary the number of arms at meta-test time.
The architecture of SymLA allows to change the network size arbitrarily by replicating existing RNNs, thus adding or removing arms at meta-test time while retaining the same meta-parameters from meta-training.
In Figure 4 we train on different numbers of arms and test on seen and unseen conﬁgurations.
All arms are independently drawn from the uniform distribution p i ∼ U [0 , 1] .
We observe that SymLA works well within-distribution (diagonal) and generalises to unseen numbers of arms (off-diagonal).
We also observe that for two arms a more specialized solution can be discovered, impeding generalisation when only training on this conﬁguration.
Generalisation to Unseen Observation Spaces In the next experiments we want to speciﬁcally analyze the permutation invariance created by our architecture.
In the previous bandit environments, actions occurred in all permutations in the training distribution.
In contrast, RL environments usually have some structure to their observations and actions.
For example in CartPole the ﬁrst observation is usually the pole angle and the ﬁrst action describes moving to the left.
Human-engineered learning algorithms are usually invariant to permutations and thus generalise to new problems with different structure.
The same should apply for our black-box agent with symmetries.
We demonstrate this property in the classic control tasks CartPole , Acrobot , and MountainCar .
We meta-train on each environment respectively with the original observation and action order.
We then meta-test on either (1) the same conﬁguration or (2) across a permuted version.
The results are visualized in Figure 5.
Due to the built-in symmetries, the performance does not degrade in the shufﬂed setting.
Instead, our method quickly learns about the ordering of the relevant observations and actions at meta-test time.
In comparison, the MetaRNN baseline fails on the permuted setting where it was not trained on, indicating over-specialization.
Thus, symmetries help to generalise to observation permutations that were not encountered during meta training.
[Figure]
We then meta-test on a setting where both the observations and actions are shufﬂed.
In this setting
SymLA still performs well as it has meta-learned to identify observations and actions at meta-test time.
In contrast, the MetaRNN fails to do so.
Standard deviations are over 3 meta-training andmeta-testing runs.
[Figure]
Experiments: Generalize to similar environments ● Permutation invariance extends to object types ● One-hot image ○ Channel (cid:248): Player ○ Channel (cid:249): Heart ○ Channel (cid:250):
Trap ● Meta-Training on this conﬁguration
[Figure]
Generalisation to Unseen Tasks The permutation invariance has further reaching consequences.
It extends to learning about tasks at meta-test time.
This enables generalisation to unseen tasks.
We construct a grid world environment (see Figure 6) with two object types: A trap and a heart.
The agent and the two objects (one of each type) are randomly positioned every episode.
Collecting the heart gives a reward of +1, whereas the trap gives -1.
All other rewards are zero.
The agent observes its own position and the position of both objects.
The observation is constructed as an image with binary channels for the position and each object type.
When meta-training on this environment, at meta-test time we observe in Figure 6 that the MetaRNN learns to directly collect hearts in each episode throughout its lifetime.
This is due to having overﬁtted to the association of hearts with positive rewards.
In comparison, SymLA starts with near-zero rewards and learns through interactions which actions need to be taken when receiving particular observations to collect the heart instead of the trap.
With sufﬁcient environment interactions L we would expect SymLA to eventually (after sufﬁcient learning) match the average reward per time of the MetaRNN in the non-shufﬂed grid world.
Next, we swap the rewards of the trap and heart, i.e. the trap now gives a positive reward, whereas the heart gives a negative reward.
This is equivalent to swapping the input channels corresponding to the heart and trap.
We observe that SymLA still generalises, learning at meta-test time about observations and their associated rewards.
In contrast, the MetaRNN now collects the wrong item, receiving negative rewards.
These results show that black-box meta RL with symmetries discovers a more general update rule that is less speciﬁc to the training tasks than typical MetaRNNs.
[Figure]
Generalisation to Unseen
Environments We have demonstrated how permutation invariance can lead to increased generalisation.
But can
SymLA also generalise between entirely different environments?
We show-case how metatraining on a grid world environment allows generalisation to CartPole.
To simplify credit-assignment, we use a dense-reward grid world where the reward is proportional to the change in distance toward a target position.
Both the target position, as well as the agent position are randomized.
The agent observes its own position, all obstacles, and the target position as a binary image with multiple channels.
In the CartPole environment the agent is rewarded for being as upright and centered as possible [24].
Further, during meta-training, we randomly project observations linearly for each lifetime.
This is necessary as in the grid world environment all observations are binary whereas the CartPole environment has continuously varying observations.
This mismatch would inhibit generalisation.
In Figure 7 we demonstrate that meta-training with
SymLA only on the GridWorld environment allows reusing the same meta-learned learning algorithm to the CartPole environment.
In contrast, the MetaRNN does not exhibit such generalisation.
Thus, learning algorithms with symmetries can to some extent generalise between signiﬁcantly different environments.
Conclusion
In this work, we identiﬁed symmetries that exist in backpropagation-based methods for meta RL but are missing from black-box methods.
We hypothesized that these symmetries lead to better generalisation of the resulting learning algorithms.
To test this, we extended a black-box meta learning method [12] that exhibits these same symmetries to the meta RL setting.
This resulted in
SymLA, a ﬂexible black-box meta RL algorithm that is less prone to over-ﬁtting.
We demonstrated generalisation to varying numbers of arms in bandit experiments (unseen action spaces), permuted observations and actions with no degradation in performance (unseen observation spaces), and observed the tendency of the meta-learned RL algorithm to learn about states and their associated rewards at meta-test time (unseen tasks).
Finally, we showed that the discovered learning behavior also transfers between grid world and (unseen) classic control environments.
A Related Work
Learning to reinforcement learn can be implemented with varying degrees of inductive biases.
Black-Box Meta RL Black-box meta RL can be implemented by policies that receive the reward signal as input [21] and use memory to learn, such as recurrence in RNNs [10, 25, 6].
These approaches do not feature the symmetries discussed in this paper which leads to a tendency of overﬁtting.
Learned Learning Rules & Fast Weights In the supervised and reinforcement learning contexts, learned learning rules [4] or fast weights [20, 22, 15, 19, 16] describe (meta-)learned mechanisms (slow weights) that update fast weights to implement learning.
This often involves outer-products and can be generalised to black-box meta learning with parameter sharing [12].
None of these approaches feature all of the symmetries we discuss above to meta learn RL algorithms.
Backpropagation-based Meta RL Alternatives to black-box meta RL include learning a weight initialization and adapting it with a human-engineered RL algorithm [7], warping computed gradients [8], meta-learning hyper-parameters [23, 28] or meta-learning objective functions corresponding to the learning algorithm
Neural Network Symmetries Symmetries in neural networks have mainly been investigated to reﬂect the structure of the input data.
This includes applications of convolutions [9], deep sets [30], graph neural networks [27], and geometric deep learning [5].
While many meta learning algorithms exhibit symmetries [4], in particular backpropagation-based meta learning [2, 7, 8, 13], the effects of these symmetries have not been discussed in detail.
In this work, we provide such a discussion and experimental investigation in the context of meta RL.
Algorithm
Require: Distribution over RL environment(s) p ( e ) θ ← initialize LSTM parameters while meta loss has not converged do (cid:46)
Outer loop in parallel over envs e ∼ p
( e ) and samples φ ∼ N ( φ | θ, Σ) { h ab } ← initialize LSTM states ∀ a, b o 1 ∼ p ( o 1 ) (cid:46)
Initialize environment e for t ∈ { 1 , . . .
, L } do (cid:46) Inner loop over lifetime in environment e h ab ← f LSTM ( h ab , o t,a , a t − 1 ,b , r t − 1 , −→ m b , ←− m a ) ∀ a, b (cid:46) Equation 8 −→ m b ← (cid:80) a f −→ m ( h ab )
∀ b (cid:46) Create forward messages ←− m a ← (cid:80) b f ←− m ( h ab ) ∀ a (cid:46) Create backward messages y ← −→ m ·(cid:46) Read out action a t ∼ p ( a t ; y )
(cid:46) Sample action from distribution parameterized by y
Send action a t to environment e , observe o t +1 and r t θ ← θ + α ∇
[Equation]
(cid:46) Update θ strategies (Equation 9)
B Bandits from Wang et al. [25]
In our experiments, we use bandits of varying difﬁculty from Wang et al.
Let p 1 be the probability of the ﬁrst arm for a payout of r = 1 , r = 0 otherwise, and p 2 the payout for the second arm.
Then, we deﬁne the
• uniform independent bandit with p 1 ∼ U [0 , 1] and p 2 ∼ U [0 , 1] , • uniform dependent bandit with p 1 ∼ U [0 , 1] and p 2 = 1 − p 1 , • easy dependent bandit with p 1 ∼ U { 0 . 1 , 0 . 9 } and p 2 = 1 − p 1 , • medium dependent bandit with p 1 ∼ U { 0 . 25 , 0 . 75 } and p 2 = 1 − p 1 , • hard dependent bandit with p 1 ∼ U { 0 . 4 , 0 . 6 } and p 2 = 1 − p 1 .
C Hyper-parameters C.1 SymLA Architecture
We use a single recurrent layer, K = 1 , with a message size of ←− M = 8 and −→ M = 8 .
To produce the next state h ab according to Equation 8, we use parameter-shared LSTMs with a hidden size of
N = 16 ( N = 64 for bandits to match Wang et al.
[25]) and run the recurrent cell for 2 micro ticks.
C.2 Meta Learning / Outer Loop
We estimate gradients ∇ θ using evolutionary strategies [18] with 10 evaluations per population sample to estimate the ﬁtness value ( 100 evaluations for bandits).
Then, we apply those using Adam with a learning rate of α = 0 .
01 , β 1 = 0 .
9 , and β 2 = 0 .
999 ( α = 0 .
2 for bandits).
We use a ﬁxed noise standard deviation of σ = 0 .
035 ( σ = 0 .
2 for bandits) and a population size of 512 .
Our inner loop has a length of L = {+
+} ( L = 100 for bandits), concatenating multiple episodes.
We meta-optimize for 4 , 000 outer steps for bandit experiments, and 20 , 000 otherwise.
C.3 Generalisation to Unseen Environments
We apply a random linear transformation (Glorot normal) to environment observations, mapping those to a 16 -dimensional vector.