Original: Introducing Symmetries to Black Box Meta Reinforcement Learning
Revised: Introducing Symmetries to Black Box Meta Reinforcement Learning

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction.
Revised: Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction.

Original: In so-called black-box approaches, thepolicy and the learning algorithm are jointly represented by a single neural network.
Revised: In so-called black-box approaches, the policy and the learning algorithm are jointly represented by a single neural network.

Original: These methods are very ﬂexible, but they tend to underperform in termsof generalisation to new, unseen environments.
Revised: These methods are very ﬂexible, but they tend to underperform in terms of generalisation to new, unseen environments.

Original: In this paper, we explore the roleof symmetries in meta-generalisation.
Revised: In this paper, we explore the role of symmetries in meta-generalisation.

Original: We show that a recent successful meta RLapproach that meta-learns an objective for backpropagation-based learning exhibitscertain symmetries (speciﬁcally the reuse of the learning rule, and invariance toinput and output permutations) that are not present in typical black-box meta RLsystems.
Revised: We show that a recent successful meta RL approach that meta-learns an objective for backpropagation-based learning exhibits certain symmetries (speciﬁcally the reuse of the learning rule, and invariance to input and output permutations) that are not present in typical black-box meta RL systems.

Original: We hypothesise that these symmetries can play an important role in metageneralisation.
Revised: We hypothesise that these symmetries can play an important role in meta-generalisation.

Original: Building off recent work in black-box supervised meta learning[Kirsch and Schmidhuber, 2020], we develop a black-box meta RL system thatexhibits these same symmetries.
Revised: Building off recent work in black-box supervised meta learning [12], we develop a black-box meta RL system that exhibits these same symmetries.

Original: We show through careful experimentation thatincorporating these symmetries can lead to algorithms with a greater ability togeneralise to unseen action & observation spaces, tasks, and environments.
Revised: We show through careful experimentation that incorporating these symmetries can lead to algorithms with a greater ability to generalise to unseen action & observation spaces, tasks, and environments.

Original:  Introduction
Revised:  Introduction

Original: Recent work in meta reinforcement learning (RL) has begun to tackle the challenging problem ofautomatically discovering general-purpose RL algorithms [Kirsch et al., 2019, Alet et al., 2020, Ohet al., 2020].
Revised: Recent work in meta reinforcement learning (RL) has begun to tackle the challenging problem of automatically discovering general-purpose RL algorithms [13, 1, 17].

Original: These methods learn to reinforcement learn by optimizing for earned reward over thelifetimes of many agents in multiple environments.
Revised: These methods learn to reinforcement learn by optimizing for earned reward over the lifetimes of many agents in multiple environments.

Original: If the discovered learning principles are sufﬁcientlygeneral-purpose, then the learned algorithms should generalise to novel environments.
Revised: If the discovered learning principles are sufﬁciently general-purpose, then the learned algorithms should generalise to novel environments.

Original: Depending onthe structure of the learned algorithm, these methods can be partitioned into backpropagation-basedmethods which learn to use the backpropagation algorithm to reinforcement learn and black-boxbased methods, in which a single (typically recurrent) neural network jointly speciﬁes the agent and RL algorithm [Wang et al., 2016, Duan et al., 2016].
Revised: Depending on the structure of the learned algorithm, these methods can be partitioned into backpropagation-based methods, which learn to use the backpropagation algorithm to reinforcement learn and black-box-based methods, in which a single (typically recurrent) neural network jointly speciﬁes the agent and RL algorithm [25, 6].

Original: While backpropagation-based methods are moreprevalent due to their relative ease of implementation and theoretical guarantees, black-box methodscan be more expressive and have the potential to avoid some of the issues with backpropagation-basedoptimization, such as memory requirements, catastrophic forgetting, and differentiability.
Revised: While backpropagation-based methods are more prevalent due to their relative ease of implementation and theoretical guarantees, black-box methods can be more expressive and have the potential to avoid some of the issues with backpropagation-based optimization, such as memory requirements, catastrophic forgetting, and differentiability.

Original: Unfortunately, black-box methods have not yet been successful at discovering general-purpose
Revised: Unfortunately, black-box methods have not yet been successful at discovering general-purpose

Original: RL algorithms.
Revised: RL algorithms.

Original: In this work, we show that black-box methods exploit fewer symmetries thanbackpropagation-based methods.
Revised: In this work, we show that black-box methods exploit fewer symmetries than backpropagation-based methods.

Original: We hypothesise that introducing more symmetries to black-boxmeta-learners can improve their generalisation capabilities.
Revised: We hypothesise that introducing more symmetries to black-box meta-learners can improve their generalisation capabilities.

Original: We test this hypothesis by introducinga number of symmetries into an existing black-box meta learning algorithm, including (1) the useof the same learned learning rule across all nodes of the neural network (NN), (2) the ﬂexibility towork with any input, output, and architecture sizes, and (3) invariance to permutations of theinputsand outputs (for dense layers). Permutation invariance implies that for any permutation of inputs
Revised: We test this hypothesis by introducing a number of symmetries into an existing black-box meta learning algorithm, including (1) the use of the same learned learning rule across all nodes of the neural network (NN), (2) the ﬂexibility to work with any input, output, and architecture sizes, and (3) invariance to permutations of the inputs

Original: [Figure]
Revised: [Figure]

Original: and outputs  the learning algorithm produces the same policy.
Revised: and outputs (for dense layers). Permutation invariance implies that for any permutation of inputs and outputs the learning algorithm produces the same policy.

Original: We refer to such agents as symmetriclearning agents (SymLA).
Revised: We refer to such agents as symmetric learning agents (SymLA).

Original: To introduce these symmetries, we build on variable shared meta learning (VSML) [Kirsch and Schmidhuber, 2020], which we adapt to the RL setting.
Revised: To introduce these symmetries, we build on variable shared meta learning (VSML) [12], which we adapt to the RL setting.

Original: VSML arranges multiple RNNs like weightsin a NN and performs message passing between these RNNs.
Revised: VSML arranges multiple RNNs like weights in a NN and performs message passing between these RNNs.

Original: We then perform meta training and metatesting similar to black-box MetaRNNs [Wang et al., 2016, Duan et al., 2016].
Revised: We then perform meta training and meta testing similar to black-box MetaRNNs [25, 6].

Original: We experimentallyvalidate SymLA on bandits, classic control, and grid worlds, comparing generalisation capabilities to MetaRNNs.
Revised: We experimentally validate SymLA on bandits, classic control, and grid worlds, comparing generalisation capabilities to MetaRNNs.

Original: SymLA improves generalisation when varying action dimensions, permutingobservations and actions, and signiﬁcantly changing tasks and environments.
Revised: SymLA improves generalisation when varying action dimensions, permuting observations and actions, and signiﬁcantly changing tasks and environments.

Original:  Preliminaries2.1 Reinforcement Learning
Revised:  Preliminaries: Meta Reinforcement Learning

Original: The RL setting in this work follows the standard (PO)MDP formulation.
Revised: The RL setting in this work follows the standard (PO)MDP formulation.

Original: At every time step, t =1 , 2 , . . .
Revised: At every time step, t = 1 , 2 , . . .

Original: the agent receives a new observation o t 2 O generated from the environment state s t 2 S49 and performs an action a t 2 A sampled from its (recurrent) policy ⇡ ✓ = p ( a t | o 1: t , a 1: t  1 ) .
Revised: the agent receives a new observation o t ∈ O generated from the environment state s t ∈ S and performs an action a t ∈ A sampled from its (recurrent) policy π θ = p ( a t | o 1: t , a 1: t − 1 ) .

Original: The50 agent receives a reward r t 2 R ⇢ R and the environment transitions to the next state.
Revised: The agent receives a reward r t ∈ R ⊂ R and the environment transitions to the next state.

Original: This transition51 is deﬁned by the environment dynamics e = p ( s t +1 , r t | s t , a t ) .
Revised: This transition is deﬁned by the environment dynamics e = p ( s t +1 , r t | s t , a t ) .

Original: The initial environment state s 1 issampled from the initial state distribution p ( s 1 ) .
Revised: The initial environment state s 1 is sampled from the initial state distribution p ( s 1 ) .

Original: The goal is to ﬁnd the optimal policy parameters ✓ ⇤that maximise the expected return R = EP
Revised: The goal is to ﬁnd the optimal policy parameters θ ∗ that maximise the expected return R = E

Original:  Tt =1  t r t ] where T is the episode length, and 0 <   1is a discount factor ( T = 1 ,  < 1 for non-episodic MDPs).
Revised: [ (cid:80) Tt =1 γ t r t ] where T is the episode length, and 0 < γ ≤is a discount factor ( T = ∞ , γ < 1 for non-episodic MDPs).

Original: 2.2 Meta Reinforcement Learning

Original: The meta reinforcement learning setting is concerned with discovering novel agents that learnthroughout their multi-episode lifetime ( L  T ) by making use of rewards r t to update their behavior.
Revised: The meta reinforcement learning setting is concerned with discovering novel agents that learn throughout their multi-episode lifetime ( L ≥ T ) by making use of rewards r t to update their behavior.

Original: This can be formulated as maximizing E e ⇠ p ( e ) [E [
Revised: This can be formulated as maximizing E e ∼ p ( e ) [

Original: P Lt =1  t r t ]] where p ( e ) is a distribution of metatraining environments.
Revised: E [ (cid:80) Lt =1 γ t r t ]] where p ( e ) is a distribution of metatraining environments.

Original: The objective itself is similar to a multi-task setting.
Revised: The objective itself is similar to a multi-task setting.

Original: In this work, we discusshow the structure of the agent inﬂuences the degree to which it learns and generalises in novel tasksand environments.
Revised: In this work, we discuss how the structure of the agent inﬂuences the degree to which it learns and generalises in novel tasks and environments.

Original: We seek to discover general-purpose learning algorithms that generalise outsidethe meta-training distribution.
Revised: We seek to discover general-purpose learning algorithms that generalise outside the meta-training distribution.

Original: We can think of an agent that learns throughout its lifetime as a history-dependent map a t , h t = f
Revised: We can think of an agent that learns throughout its lifetime as a history-dependent map a t , h t = f

Original: ( h t  1 , o t , r t  1 , a t  1 ) that produces an action a t and new agent state h t given its previous state h t  1 , an observation o t , environment reward r t  1 , and previous action a t  1 .
Revised: ( h t − 1 , o t , r t − 1 , a t − 1 ) that produces an action a t and new agent state h t given its previous state h t − 1 , an observation o t , environment reward r t − 1 , and previous action a t − 1 .

Original: In the case of backpropagationbased learning, f is decomposed into: (1) a stationary policy ⇡ ( s ) ✓ that maps the current state into an action, a t = ⇡ ( s ) ✓ ( o t ) ; and (2) a backpropagation-based update rule that optimizes a given objective
Revised: In the case of backpropagationbased learning, f is decomposed into: (1) a stationary policy π ( s ) θ that maps the current state into an action, a t = π ( s ) θ ( o t ) ; and (2) a backpropagation-based update rule that optimizes a given objective

Original: J by propagating the error signal backwards and updating the policy in ﬁxed intervals (e.g. after each episode).
Revised: J by propagating the error signal backwards and updating the policy in ﬁxed intervals (e.g. after each episode).

Original: In its simplest form, for any dense layer k 2 { 1 , . . .
Revised: In its simplest form, for any dense layer k ∈ { 1 , . . .

Original: , K } of a NN policy with size
Revised: , K } of a NN policy with size

Original: A ( k ) ⇥ B ( k ) , inputs x ( k ) , outputs x ( k +1) , and weights w ( k ) ⇢ ✓ , the backpropagation update rule is given by[Equation] forward pass [Equation]
Revised: A ( k ) × B ( k ) , inputs x ( k ) , outputs x ( k +1) , and weights w ( k ) ⊂ θ , the backpropagation update rule is given by

Original: ab |

Original: [Equation]
Revised: [Equation]

Original: backward pass

Original: [Equation] w [Equation] =
Revised: where a ∈ { 1 , . . .

Original: [Equation]

Original: [Equation]

Original: @ w ( k ) ab =

Original:  ↵ x ( k ) a  ( k ) b | {z } update (3) where a 2 { 1 , . . .

Original: A ( k ) } , b 2 { 1 , . . .
Revised: A ( k ) } , b ∈ { 1 , . . .

Original: , B ( k ) } , ↵ is the learning rate,  are error terms, and theagent state h corresponds to parameters ✓ .
Revised: , B ( k ) } , α is the learning rate, δ are error terms, and the agent state h corresponds to parameters θ .

Original: The initial error is given by the gradient at the NNoutputs,  ( k ) =@
Revised: The initial error is given by the gradient at the NN outputs, δ ( k ) =

Original: J @ x ( K +1) .
Revised: [Equation]

Original: Transformations such as non-linearities are omitted here.
Revised: Transformations such as non-linearities are omitted here.

Original: Works in metareinforcement learning that take this approach parameterise the objective J  and meta-learn itsparameters [Kirsch et al., 2019, Oh et al., 2020].
Revised: Works in metareinforcement learning that take this approach parameterise the objective J φ and meta-learn its parameters [13, 17].

Original: In contrast, black-box meta RL [Duan et al., 2016, Wang et al., 2016] meta-learns f directly inthe form of a single non-stationary policy ⇡ ✓ with memory.
Revised: In contrast, black-box meta RL [6, 25] meta-learns f directly in the form of a single non-stationary policy π θ with memory.

Original: Parameters of f represent the learningalgorithm (no explicit J  ) while the state h represents the policy.
Revised: Parameters of f represent the learning algorithm (no explicit J φ ) while the state h represents the policy.

Original: In the simplest form of an RNNrepresentation of f , given a current hidden state h and inputs o, r, a (concatenated [ · ] ), updates to thepolicy take the form
Revised: In the simplest form of an RNN representation of f , given a current hidden state h and inputs o, r, a (concatenated [ · ] ), updates to the policy take the form

Original: [Equation]
Revised: [Equation]

Original: with parameters ✓ = v and activation function  , omitting the bias term.
Revised: with parameters θ = v and activation function σ , omitting the bias term.

Original: We refer to this as the
Revised: We refer to this as the

Original: MetaRNN.
Revised: MetaRNN.

Original: The inputs must include, beyond the observation o , the previous reward r and action a , sothat the meta-learner can learn to associate past actions with rewards [Schmidhuber, 1992b, Wanget al., 2016].
Revised: The inputs must include, beyond the observation o , the previous reward r and action a , so that the meta-learner can learn to associate past actions with rewards [21, 25].

Original: Further, black-box systems do not reset the state h between episode boundaries, so thatthe learning algorithm can accumulate knowledge through the agent’s lifetime.
Revised: Further, black-box systems do not reset the state h between episode boundaries, so that the learning algorithm can accumulate knowledge through the agent’s lifetime.

Original: For additional relatedwork, refer to Appendix A.
Revised: For additional related work, refer to Appendix A.

Original:  Symmetries in Meta RL
Revised:  Symmetries in Meta RL

Original: In this section, we demonstrate how the learning dynamics in backpropagation-based systems(Equation 3) differ from the learning dynamics in black-box systems (Equation 4), and how thisaffects the generalisation of black-box methods to novel environments.
Revised: In this section, we demonstrate how the learning dynamics in backpropagation-based systems (Equation 3) differ from the learning dynamics in black-box systems (Equation 4), and how this affects the generalisation of black-box methods to novel environments.

Original:  We ﬁrst identify three symmetries that backpropagation-based systems exhibit and discuss how theyaffect the generalisability of the learned learning algorithms.
Revised: Symmetries in backpropagation-based Meta RL We ﬁrst identify three symmetries that backpropagation-based systems exhibit and discuss how they affect the generalisability of the learned learning algorithms.

Original: Symmetric learning rule.
Revised: Symmetric learning rule.

Original: In Equation 3, each parameter w ab is updated by the same update rule based on information from the forward and backward pass.
Revised: In Equation 3, each parameter w ab is updated by the same update rule based on information from the forward and backward pass.

Original: Meta-learning an objective J  affects the updates of each parameter symmetrically through backpropagation.
Revised: Meta-learning an objective J φ affects the updates of each parameter symmetrically through backpropagation.

Original: Flexible input, output, and architecture sizes.
Revised: Flexible input, output, and architecture sizes.

Original: Because the same rule is applied every- where, the learning algorithm can be applied to arbitrarily sized neural networks, includingvariations in input and output sizes.
Revised: Because the same rule is applied everywhere, the learning algorithm can be applied to arbitrarily sized neural networks, including variations in input and output sizes.

Original: This involves varying A and B and the number of layers,affecting how often the learning rule is applied and how many parameters are being learned.
Revised: This involves varying A and B and the number of layers, affecting how often the learning rule is applied and how many parameters are being learned.

Original: Invariance to input and output permutations.
Revised: Invariance to input and output permutations.

Original: Given a permutation of inputs and outputs in a layer, deﬁned by the bijections ⇢ : N ! N and ⇢ 0 :N ! N
Revised: Given a permutation of inputs and outputs in a layer, deﬁned by the bijections ρ : N → N and ρ (cid:48) :

Original:  , the learning rule is appliedas x ( k +1) ⇢ 0 ( b ) =
Revised: N → N , the learning rule is applied as x ( k +1) ρ (cid:48)

Original: P a x ( k ) ⇢ ( a ) w ( k ) ab ,  ( k  1) ⇢ ( a ) =
Revised: ( b ) = (cid:80) a x ( k ) ρ ( a ) w ( k ) ab , δ ( k − 1) ρ ( a ) =

Revised: (cid:80)

Original: P b  ( k ) ⇢ 0 ( b ) w ( k ) ab , and  w ( k ) ab =  ↵ x ( k ) ⇢ ( a )  ( k ) ⇢ 0 ( b ) .
Revised: b δ ( k ) ρ (cid:48) ( b ) w ( k ) ab , and ∆ w ( k ) ab = − αx ( k ) ρ ( a ) δ ( k ) ρ (cid:48) ( b ) .

Original: Letw 0 be a weight matrix with w 0 ( k ) ⇢ ( a ) ⇢ 0 ( b ) =
Revised: Let w (cid:48) be a weight matrix with w (cid:48) ( k ) ρ ( a ) ρ (cid:48) ( b ) =

Original: w ( k ) a,b , then we can equivalently write x ( k +1) ⇢ 0 ( b ) =
Revised: w ( k ) a,b , then we can equivalently write x ( k +1) ρ (cid:48)

Original: P
Revised: ( b ) = (cid:80) a x ( k ) ρ ( a ) w (cid:48) ( k ) ρ ( a ) ρ (cid:48) ( b ) , δ ( k − 1) ρ

Original: [Equation]
Revised: ( a ) = (cid:80)

Original: x ( k ) ⇢ ( a ) w 0 ( k ) ⇢ ( a ) ⇢ 0 ( b ) ,  ( k  1) ⇢ ( a ) = P b  ( k ) ⇢ 0 ( b ) w 0 ( k ) ⇢ ( a ) ⇢ 0 ( b ) , and  w 0 ( k ) ⇢ ( a ) ⇢ 0 ( b ) 
Revised: b δ ( k ) ρ (cid:48) ( b ) w (cid:48) ( k ) ρ ( a ) ρ (cid:48) ( b ) , and ∆ w (cid:48) ( k ) ρ ( a ) ρ (cid:48) ( b ) = − αx ( k ) ρ ( a ) δ ( k ) ρ (cid:48)

Original: =  ↵ x ( k ) ⇢ ( a )  ( k ) ⇢ 0 ( b ) .
Revised: ( b ) .

Original: If allelements of w 0 ( k ) are initialized i.i.d.
Revised: If all elements of w (cid:48) ( k ) are initialized i.i.d.

Original: , we can interchangeably use w in place of w 0 in theabove updates.
Revised: , we can interchangeably use w in place of w (cid:48) in the above updates.

Original: By doing so, we recover the original learning rule equations for any a, b .
Revised: By doing so, we recover the original learning rule equations for any a, b .

Original: Thus, the learning algorithm is invariant to input and output permutations.
Revised: Thus, the learning algorithm is invariant to input and output permutations.

Original: While backpropagation has inherent symmetries, these symmetries would be violated if the objectivefunction J  would be asymmetric.
Revised: While backpropagation has inherent symmetries, these symmetries would be violated if the objective function J φ would be asymmetric.

Original: Formally, when permuting the NN outputs y = x ( K +1)
Revised: Formally, when permuting the NN outputs y = x ( K +1)

Original: suchthat y 0 b =
Revised: such that y (cid:48) b =

Original: y ⇢ 0 ( b ) , J  should satisfy that the gradient under the permutation is also a permutation@ J  ( y 0 ) @ y 0 b = h @ J  ( y ) @
Revised: y ρ (cid:48) ( b ) , J φ should satisfy that the gradient under the permutation is also a permutation

Original: [-y i ⇢ 
-]
Revised: [-y i ⇢ 
-][Equation]

Original: ( b ) where the environment accepts the action permuted by ⇢ 0 in the case of
Revised: ( b ) where the environment accepts the action permuted by ρ (cid:48) in the case of

Original: J  ( y 0 ) .
Revised: J φ ( y (cid:48) ) .

Original: This is the case for policy gradients, for instance, if the action selection ⇡ ( a | s ) is permutedaccording to ⇢ 0 .
Revised: This is the case for policy gradients, for instance, if the action selection π ( a | s ) is permuted according to ρ (cid:48) .

Original: When meta-learning objective functions, prior work carefully designed the objectivefunction J  to be symmetric.
Revised: When meta-learning objective functions, prior work carefully designed the objective function J φ to be symmetric.

Original: In MetaGenRL [Kirsch et al., 2019], taken actions were processedelement-wise with the policy outputs and sum-reduced by the loss function.
Revised: In MetaGenRL [13], taken actions were processed element-wise with the policy outputs and sum-reduced by the loss function.

Original: In LPG [Oh et al. , 2020],taken actions and policy outputs were not directly fed to J  , but instead only the log probability ofthe action distribution was used.
Revised: In LPG [17], taken actions and policy outputs were not directly fed to J φ , but instead only the log probability of the action distribution was used.

Original:  Black-box meta learning methods are appealing as they require few hard-coded biases and are ﬂexibleenough to represent a wide range of possible learning algorithms.
Revised: Insufﬁcient Symmetries in Black-box Meta RL Black-box meta learning methods are appealing as they require few hard-coded biases and are ﬂexible enough to represent a wide range of possible learning algorithms.

Original: We hypothesize that this comesat the cost of the tendency to overﬁt to the given meta training environment(s) resulting in overlyspecialized learning algorithms.
Revised: We hypothesize that this comes at the cost of the tendency to overﬁt to the given meta training environment(s) resulting in overly specialized learning algorithms.

Original: Learning dynamics in backpropagation-based systems (Equation 3) differ signiﬁcantly from learningdynamics in black-box systems (Equation 4).
Revised: Learning dynamics in backpropagation-based systems (Equation 3) differ signiﬁcantly from learning dynamics in black-box systems (Equation 4).

Original: In particular, meta-learning J  is signiﬁcantly moreconstrained, since J  can only indirectly affect each policy parameter w ( k ) ab through the same learningrule from Equation 3.
Revised: In particular, meta-learning J φ is signiﬁcantly more constrained, since J φ can only indirectly affect each policy parameter w ( k ) ab through the same learning rule from Equation 3.

Original: In contrast, in black-box systems (Equation 4), each policy state h b is directlycontrolled by unique meta-parameters (vector v · b ), thereby encouraging the black-box meta-learnerto construct speciﬁc update rules for each element of the policy state.
Revised: In contrast, in black-box systems (Equation 4), each policy state h b is directly controlled by unique meta-parameters (vector v · b ), thereby encouraging the black-box meta-learner to construct speciﬁc update rules for each element of the policy state.

Original: This results in sensitivity topermutations in inputs and outputs.
Revised: This results in sensitivity to permutations in inputs, outputs, and a general tendency to construct non-learning, biased solutions, over learning solutions.

Original: Furthermore, input and output spaces must retain the same sizeas those are directly dependent on the number of RNN parameters.
Revised: Furthermore, input and output spaces must retain the same size as those are directly dependent on the number of RNN parameters.

Original: As an example, consider a meta-training distribution of two-armed bandits where the expected payoutof the ﬁrst arm is much larger than the second.

Original: If we meta-train a MetaRNN on these environmentsthen when meta-testing the MetaRNN will have learned to immediately increase the probabilityof pulling the ﬁrst arm, independent of any observed rewards.

Original: If instead the action probability isadapted using REINFORCE or a meta-learned symmetric objective function then, due to the implicitsymmetries, the learning algorithm could not differentiate between the two arms to favor one overthe other.

Original: While the MetaRNN behavior is optimal when meta-testing on the same meta-trainingdistribution, it completely fails to generalise to other distributions.

Original: Thus, the MetaRNN results ina non-learning, biased solution, whereas the backpropagation-based approach results in a learningsolution.

Original: In the former case, the learning algorithm is overﬁtted to only produce a ﬁxed policythat always samples the ﬁrst arm.

Original: In the latter case, the learning algorithm is unbiased and willlearn a policy from observed rewards to sample the ﬁrst arm.

Original: Beyond bandits, for reasonably sizedmeta-training distributions, we may have any number of biases in the data that a MetaRNN willinherit, impeding generalisation to unseen tasks and environments.

Original:  Adding Symmetries to Black-box Meta RL
Revised:  Adding Symmetries to Black-box Meta RL

Original: A solution to the illustrated over-ﬁtting problem with black-box methods is the introduction ofsymmetries into the parameterisation of the policy.
Revised: A solution to the illustrated over-ﬁtting problem with black-box methods is the introduction of symmetries into the parameterisation of the policy.

Original: This can be achieved by generalising theforward pass (Equation 1), backward pass (Equation 2), and element-wise update (Equation 3)to parameterized versions.
Revised: This can be achieved by generalising the forward pass (Equation 1), backward pass (Equation 2), and element-wise update (Equation 3) to parameterized versions.

Original: We further subsume the loss computation into these parameterized updaterules.
Revised: We further subsume the loss computation into these parameterized update rules.

Original: Together, they form a single recurrent policy with additional symmetries.
Revised: Together, they form a single recurrent policy with additional symmetries.

Original: Prior work onvariable shared meta learning (VSML) [Kirsch and Schmidhuber, 2020] used similar principles tometa-learn supervised learning algorithms.
Revised: Prior work on variable shared meta learning (VSML) [12] used similar principles to meta-learn supervised learning algorithms.

Original: In the following, we extend their approach to deal withthe RL setting.
Revised: In the following, we extend their approach to deal with the RL setting.

Original:  VSML describes neural architectures for meta learning with parameter sharing.
Revised: Variable Shared Meta Learning VSML describes neural architectures for meta learning with parameter sharing.

Original: This can be motivatedby meta learning how to update weights [Bengio et al., 1992, Schmidhuber, 1993] where the updaterule is shared across the network.
Revised: This can be motivated by meta learning how to update weights [4, 22] where the update rule is shared across the network.

Original: Instead of designing a meta network that deﬁnes the weight updates explicitly, we arrange small parameter-shared RNNs (LSTMs) like weights in a NN and performmessage passing between those.
Revised: Instead of designing a meta network that deﬁnes the weight updates explicitly, we arrange small parameter-shared RNNs (LSTMs) like weights in a NN and perform message passing between those.

Original: In VSML, each weight w ab with w 2 R A ⇥ B in a NN is replaced by a small RNN with parameters ✓and hidden state h ab 2 R N .
Revised: In VSML, each weight w ab with w ∈ R A × B in a NN is replaced by a small RNN with parameters θ and hidden state h ab ∈ R N .

Original: We restrict ourselves to dense NN layers here, where w correspondsto the weights of that layer with input size A and output size B .
Revised: We restrict ourselves to dense NN layers here, where w corresponds to the weights of that layer with input size A and output size B .

Original: This can be adapted to otherarchitectures such as CNNs if necessary.
Revised: This can be adapted to other architectures such as CNNs if necessary.

Original: All these RNNs share the same parameters ✓ , deﬁning bothwhat information propagates in the neural network, as well as how states are updated to implementlearning.
Revised: All these RNNs share the same parameters θ , deﬁning both what information propagates in the neural network, as well as how states are updated to implement learning.

Original: Each RNN with state h ab receives the analogue to the previous activation, here calledthe vectorized forward message ! m a 2 R ! M , and the backward message  m b 2 R  M for informationﬂowing backwards in the network (asynchronously).
Revised: Each RNN with state h ab receives the analogue to the previous activation, here called the vectorized forward message −→ m a ∈ R −→ M , and the backward message ←− m b ∈ R ←− M for information ﬂowing backwards in the network (asynchronously).

Original: The backward message may contain informationrelevant to credit assignment, but is not constrained to this.
Revised: The backward message may contain information relevant to credit assignment, but is not constrained to this.

Original: The RNN update equation (compare
Revised: The RNN update equation (compare

Original: Equation 3 and 4) is then given by
Revised: Equation 3 and 4) is then given by

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  for layer k where k 2 { 1 , . . . , K } and a 2 { 1 , . . .
Revised: ) for layer k where k ∈ { 1 , . . . , K } and a ∈ { 1 , . . .

Original: A ( k ) } , b 2 { 1 , . . .
Revised: A ( k ) } , b ∈ { 1 , . . .

Original: , B ( k ) } .
Revised: , B ( k ) } .

Original: Similarly, new forwardmessages are created by transforming the RNN states using a function f ! m : R N ! R ! M (compare
Revised: Similarly, new forward messages are created by transforming the RNN states using a function f −→ m : R N → R −→ M (compare

Original: Equation 1) such that
Revised: Equation 1) such that

Original: [Equation]
Revised: [Equation]

Original: (6) deﬁnes the new forward message for layer k + 1 with b 2 { 1 , . . .
Revised: deﬁnes the new forward message for layer k + 1 with b ∈ { 1 , . . .

Original: , B ( k ) =
Revised: , B ( k ) =

Original: A ( k +1) } .
Revised: A ( k +1) } .

Original: The backwardmessage is given by f  m : R N ! R  M (compare Equation 2) such that
Revised: The backward message is given by f ←− m : R N → R ←− M (compare Equation 2) such that

Original: [Equation]
Revised: [Equation]

Original: and a 2 { 1 , . . .
Revised: and a ∈ { 1 , . . .

Original:  A ( k ) =
Revised: , A ( k ) =

Original: B ( k  1) } .
Revised: B ( k − 1) } .

Original: For simplicity, we use ✓ below to denote all of the VSMLparameters, including those of the RNN and forward and backward message functions.
Revised: For simplicity, we use θ below to denote all of the VSML parameters, including those of the RNN and forward and backward message functions.

Original: In the following, we derive a black-box meta reinforcement learner based on VSML (visualized in
Revised: In the following, we derive a black-box meta reinforcement learner based on VSML (visualized in

Original: Figure 1).
Revised: Figure 1).

Original: Inner loop Updates RNN states h ab
Revised: [Figure]

Original: Outer loop Updates RNN parameters θ h ( t ) h ( t +1)
Revised: θ h ( t ) h ( t +1)

Original: θ ( i )
Revised: θ ( i )

Original: o t , r t −o t +1 , r t a t a t
Revised: o t , r t −o t +1 , r t a t a t

Original: + θ ( i ) θ
Revised: + θ ( i ) θ

Original: ( i ) θ ( i ) θ
Revised: ( i ) θ ( i ) θ

Original: ( i +1)
Revised: ( i +1)

Original: Based on feedback r t , RNN states can be used as memory for learning.
Revised: Based on feedback r t , RNN states can be used as memory for learning.

Original: The learning algorithm encoded in the RNN parameters ✓ is updated in the outer loop by meta-training using ES.
Revised: The learning algorithm encoded in the RNN parameters θ is updated in the outer loop by meta-training using ES.

Original:  At each time step in the environment, the agent’s inputs consist of the previously taken action a t  1 , current observation o t and previous reward r t  1 .
Revised: RL Agent Inputs and Outputs At each time step in the environment, the agent’s inputs consist of the previously taken action a t − 1 , current observation o t and previous reward r t − 1 .

Original: We feed176 r t  1 as an additional input to each RNN, the observationo t 2 R A (1) to the ﬁrst layer ( ! m (1) · 1 := o t ), and the action a t  1 2 { 0 , 1 } B ( K ) (one-hot encoded) to the last layer(  m ( K ) ·:= a t  1 ).
Revised: We feed r t − 1 as an additional input to each RNN, the observation o t ∈ R A (1) to the ﬁrst layer ( −→ m (1) · 1 := o t ), and the action a t − 1 ∈ { 0 , 1 } B ( K ) (onehot encoded) to the last layer ( ←− m ( K ) ·:= a t − 1 ).

Original: The index 1 refers to the ﬁrst dimensionof the ! M or  M -dimensional message.
Revised: The index 1 refers to the ﬁrst dimension of the −→ M or ←− Mdimensional message.

Original: We interpret theagent’s output message y =
Revised: We interpret the agent’s output message y =

Original: ! m ( K +1) ·as the unnormalizedlogits of a categorical distribution over actions.
Revised: −→ m ( K +1) ·as the unnormalized logits of a categorical distribution over actions.

Original: While wefocus on discrete actions only in our present experiments,this can be adapted for probabilistic or deterministic continuous control.
Revised: While we focus on discrete actions only in our present experiments, this can be adapted for probabilistic or deterministic continuous control.

Original:  Instead of using multiple layers ( K > 1 ), in this paper we use a single layer ( K = 1 ).
Revised: Architecture Recurrence and Reward Signal Instead of using multiple layers ( K > 1 ), in this paper we use a single layer ( K = 1 ).

Original: In Equation5, RNNs in the same layer can not coordinate directly as their messages are only passed to the nextand previous layer.
Revised: In Equation 5, RNNs in the same layer can not coordinate directly as their messages are only passed to the next and previous layer.

Original: To give that single layer sufﬁcient expressivity for the RL setting, we make it‘recurrent’ by processing the layer’s own messages !
Revised: To give that single layer sufﬁcient expressivity for the RL setting, we make it ‘recurrent’ by processing the layer’s own messages −→ m

Original: m ( k +1) b and  m ( k  1) a .
Revised: ( k +1) b and ←− m ( k − 1) a .

Original: The network thus has twolevels of recurrence: (1) Each RNN that corresponds to a weight of a standard NN and (2) messagesthat are generated according to Equation 6 and 7 and fed back into the same layer.
Revised: The network thus has two levels of recurrence: (1) Each RNN that corresponds to a weight of a standard NN and (2) messages that are generated according to Equation 6 and 7 and fed back into the same layer.

Original: Furthermore, each RNN receives the current reward signal r t  1 as input.
Revised: Furthermore, each RNN receives the current reward signal r t − 1 as input.

Original: The update equation is given by
Revised: The update equation is given by

Original: [Equation]
Revised: [Equation]

Original: from previous step [Equation] (8) where a 2 { 1 , . . .
Revised: where a ∈ { 1 , . . .

Original: A ( k ) } , b 2 { 1 , . . .
Revised: A ( k ) } , b ∈ { 1 , . . .

Original: , B ( k ) } .
Revised: , B ( k ) } .

Original: As we only use a single layer, k = 1 , we apply theupdate multiple times (multiple micro ticks) for each step in the environment.
Revised: As we only use a single layer, k = 1 , we apply the update multiple times (multiple micro ticks) for each step in the environment.

Original: This can also be viewedas multiple layers with shared parameters, where parameters correspond to states h .
Revised: This can also be viewed as multiple layers with shared parameters, where parameters correspond to states h .

Original: For pseudo code,see Algorithm 1 in the appendix.
Revised: For pseudo code, see Algorithm 1 in the appendix.

Original:  By incorporating the above changes to inputs, outputs, and architecture, we arrive at a black-box meta RL method with symmetries, here represented by our proposed symmetric learning agents (SymLA).
Revised: Symmetries in SymLA By incorporating the above changes to inputs, outputs, and architecture, we arrive at a black-box meta RL method with symmetries, here represented by our proposed symmetric learning agents (SymLA).

Original: By construction, SymLA exhibits the same symmetries as those described in Section 3.1, despite notusing the backpropagation algorithm.
Revised: By construction, SymLA exhibits the same symmetries as those described in Section 3, despite not using the backpropagation algorithm.

Original: Symmetric learning rule.
Revised: Symmetric learning rule.

Original: The learning rule as deﬁned by Equation 8 is replicated acrossa 2 { 1 , . . .
Revised: The learning rule as deﬁned by Equation 8 is replicated across a ∈ { 1 , . . .

Original: , A } and b 2 { 1 , . . .
Revised: , A } and b ∈ { 1 , . . .

Original: , B } with the same parameter ✓ .2.
Revised: , B } with the same parameter θ .

Original: Flexible input, output, and architecture sizes.
Revised: Flexible input, output, and architecture sizes.

Original: Changes in A , B , and K correspond toinput, output, and architecture size.
Revised: Changes in A , B , and K correspond to input, output, and architecture size.

Original: This does not affect the number of meta-parameters and therefore these quantities can also be varied at meta-test time.
Revised: This does not affect the number of meta-parameters and therefore these quantities can also be varied at meta-test time.

Original: Invariance to input and output permutations.
Revised: Invariance to input and output permutations.

Original: When permuting mes sages using bijections ⇢ and ⇢ 0 , the state update becomes [Equation] [Equation]
Revised: When permuting messages using bijections ρ and ρ (cid:48) , the state update becomes h ( k ) ab ← f RNN ( h ( k ) ab , −→ m ( k ) ρ ( a ) ,

Original:  , ! m ( k +1) ⇢ 0 ( b ) ,  m ( k  1) ⇢ ( a ) ) , and the message transformationsare ! m ( k +1)⇢ 0 ( b ) =
Revised: ←− m ( k ) ρ (cid:48) ( b ) , r t − 1 , −→ m ( k +1) ρ (cid:48) ( b ) , ←− m ( k − 1) ρ ( a ) ) , and the message transformations are −→ m ( k +1)

Original: P a f ! m ( h ( k ) ab ) and  m ( k  1) ⇢ ( a ) = 
Revised: ρ (cid:48) ( b ) = (cid:80) a f −→ m ( h ( k ) ab ) and ←− m ( k − 1) ρ ( a ) = (cid:80) b

Original: P b f  m ( h ( k ) ab ) .
Revised: f ←− m ( h ( k ) ab ) .

Original: Similar to backpropagation,when RNN states h ab are initialized i.i.d.
Revised: Similar to backpropagation, when RNN states h ab are initialized i.i.d.

Original: , we can use h ⇢ ( a ) , ⇢ 0 ( b ) in place of h ab to recoverthe original Equations 6, 7, 8.
Revised: , we can use h ρ ( a ) ,ρ (cid:48) ( b ) in place of h ab to recover the original Equations 6, 7, 8.

Original: Learning  corresponds to updating RNN states h ab (see Figure 2).
Revised: Learning / Inner Loop Learning corresponds to updating RNN states h ab (see Figure 2).

Original: This is the same as the MetaRNN [Wang et al., 2016, Duan et al., 2016] but with a more structured neural model.
Revised: This is the same as the MetaRNN [25, 6] but with a more structured neural model.

Original: Forﬁxed RNN parameters ✓ which encode the learning algorithm, we randomly initialize all statesh ab .
Revised: For ﬁxed RNN parameters θ which encode the learning algorithm, we randomly initialize all states h ab .

Original: Next, the agent steps through the environment, updating h ab in each step.
Revised: Next, the agent steps through the environment, updating h ab in each step.

Original: If the environmentis episodic with T steps, the agent is run for a lifetime of L  T steps with environment resetsin-between, carrying the agent state h ab over.
Revised: If the environment is episodic with T steps, the agent is run for a lifetime of L ≥ T steps with environment resets in-between, carrying the agent state h ab over.

Original:  Each outer loop step unrolls the inner loop for L environment steps to update ✓ .
Revised: Meta Learning / Outer Loop Each outer loop step unrolls the inner loop for L environment steps to update θ .

Original: The SymLA objectiveis to maximize the agent’s lifetime sum of rewards,i.e.
Revised: The SymLA objective is to maximize the agent’s lifetime sum of rewards,

Original: P Lt = [Equation] t ( ✓ ) .
Revised: i.e. (cid:80) Lt =1 r t ( θ ) .

Original: We optimize this objective usingevolutionary strategies [Wierstra et al., 2008, Salimans et al., 2017] by following the gradient[Equation]
Revised: We optimize this objective using evolutionary strategies [26, 18] by following the gradient

Original: [Equation]
Revised: [Equation]

Original: (9) with some ﬁxed diagonal covariance matrix ⌃ and environments e ⇠ p ( e ) .
Revised: with some ﬁxed diagonal covariance matrix Σ and environments e ∼ p ( e ) .

Original: We chose evolutionstrategies due to its ability to optimize over long inner-loop horizons without memory constraints thatoccur due to backpropagation-based meta optimization.
Revised: We chose evolution strategies due to its ability to optimize over long inner-loop horizons without memory constraints that occur due to backpropagation-based meta optimization.

Original: Furthermore, it was shown that meta-losslandscapes are difﬁcult to navigate and the search distribution helps in smoothing those [Metz et al.
Revised: Furthermore, it was shown that meta-loss landscapes are difﬁcult to navigate and the search distribution helps in smoothing those [14].

Original:  Experiments
Revised:  Experiments

Original: Equipped with a symmetric black-box learner, we now investigate how its learning properties differfrom a standard MetaRNN. Firstly, we learn to learn on bandits from Wang et al. [2016] where the meta-training environments are similar to the meta-test environments. Secondly, we demonstrategeneralisation to unseen action spaces, applying the learned algorithm to bandits with varying numbersof arms at meta-test time—something that MetaRNNs are not capable of.
Revised: [Table]

Original: Thirdly, we demonstratehow symmetries improve generalisation to unseen observation spaces by creating permutationsof observations and actions in classic control benchmarks. Fourthly, we show how permutationinvariance leads to generalisation to unseen tasks by learning about states and their associated rewardsat meta-test time. Finally, we demonstrate how symmetries result in better learning algorithms forunseen environments, generalising from a grid world to CartPole.
Revised: [Figure]

Original: Hyper-parameters are in Appendix C. 5.1 Learning to Learn on Similar Environments
Revised: U n i f I n d e p

Original: [Table]
Revised: [Figure]

Revised: E a s y U

Original:  i f D e p .
Revised: n i f D e p .

Original: M e d i u m H a r d
Revised: M e d i u m H a r d

Original: U n i f I n d e p .
Revised: U n i f I n d e p .

Original: E a s y U
Revised: E a s y U

Original: n
Revised: n

Original: i f D e p .
Revised: i f D e p .

Original: M e d i u m H a r d
Revised: M e d i u m H a r d

Original: U n i f I n d e p .
Revised: U n i f I n d e p .

Original: E a s y U
Revised: E a s y U

Original: n
Revised: n

Original: i f D e p .
Revised: i f D e p .

Original: M e d i u m H a r d 
Revised: M e d i u m H a r d where the meta-training environments are similar to the meta-test environments. Secondly, we demonstrate generalisation to unseen action spaces, applying the learned algorithm to bandits with varying numbers of arms at meta-test time—something that MetaRNNs are not capable of. Thirdly, we demonstrate how symmetries improve generalisation to unseen observation spaces by creating permutations of observations and actions in classic control benchmarks. Fourthly, we show how permutation invariance leads to generalisation to unseen tasks by learning about states and their associated rewards at meta-test time.

Original: 5.2 Generalisation to Unseen Action Spaces
Revised: Finally, we demonstrate how symmetries result in better learning algorithms for unseen environments, generalising from a grid world to CartPole. Hyper-parameters are in Appendix C. [Figure]

Original: 2 a rm s 8 a r m s
Revised: r a i n

Original: [Figure]
Revised: [Figure]

Original: 3 meta-training and 
Revised: Learning to Learn on Similar Environments We ﬁrst compare SymLA and the MetaRNN on the two-armed (dependent) bandit experiments from Wang et al.

Original: meta
Revised: [25] where there is no large variation in the meta-test environments. These consist of ﬁve different settings of varying difﬁculty that we use for meta-training and meta-testing (see Appendix B). There are no observations (no context), only two arms, and a meta-training distribution where each arm has the same marginal distribution of payouts. Thus, we expect the symmetries from SymLA to have no signiﬁcant effect on performance .

Original: runs with 
Revised: We meta-train for an agent lifetime of L = 100 arm-pulls and report the expected cumulative regret at meta-test time in Figure 3. We metatrain on each of the ﬁve settings, and meta-test across all settings. The performance of the MetaRNN reproduces the average performance of Wang et al. [25], here trained with ES instead of A2C. When using symmetries (as in

Original: -pulls (smaller is Particularly relevant scenarios (off In contrast to the MetaRNN, in SymLA we can vary thenumber of arms at meta-test time.
Revised: SymLA), we recover a similar performance compared to the MetaRNN. Generalisation to Unseen Action Spaces In contrast to the MetaRNN, in SymLA we can vary the number of arms at meta-test time.

Original: The architecture of

Original:  SymLA allows to change the network size arbitrarily byreplicating existing RNNs, thus adding or removing armsat meta-test time while retaining the same meta-parametersfrom meta-training.
Revised: The architecture of SymLA allows to change the network size arbitrarily by replicating existing RNNs, thus adding or removing arms at meta-test time while retaining the same meta-parameters from meta-training.

Original: In Figure 4 we train on different numbers of arms and test on seen and unseen conﬁgurations.
Revised: In Figure 4 we train on different numbers of arms and test on seen and unseen conﬁgurations.

Original: All arms are independently drawn from the uniform distribution p i ⇠ U [0 , 1] .
Revised: All arms are independently drawn from the uniform distribution p i ∼ U [0 , 1] .

Original: We observe that SymLA workswell within-distribution (diagonal) and generalises to unseen numbers of arms (off-diagonal).
Revised: We observe that SymLA works well within-distribution (diagonal) and generalises to unseen numbers of arms (off-diagonal).

Original: We also observe thatfor two arms a more specialized solution can be discovered, impeding generalisation when only training on thisconﬁguration.
Revised: We also observe that for two arms a more specialized solution can be discovered, impeding generalisation when only training on this conﬁguration.

Original:  In the next experiments we want to speciﬁcally analyzethe permutation invariance created by our architecture.
Revised: Generalisation to Unseen Observation Spaces In the next experiments we want to speciﬁcally analyze the permutation invariance created by our architecture.

Original: Inthe previous bandit environments, actions occurred in allpermutations in the training distribution.
Revised: In the previous bandit environments, actions occurred in all permutations in the training distribution.

Original: In contrast, RLenvironments usually have some structure to their observations and actions.
Revised: In contrast, RL environments usually have some structure to their observations and actions.

Original: For example in CartPolethe ﬁrst observation is usually the pole angle and the ﬁrst action describes moving to the left.
Revised: For example in CartPole the ﬁrst observation is usually the pole angle and the ﬁrst action describes moving to the left.

Original: Human- engineered learning algorithms are usually invariant to permutations and thus generalise to newproblems with different structure.
Revised: Human-engineered learning algorithms are usually invariant to permutations and thus generalise to new problems with different structure.

Original: The same should apply for our black-box agent with symmetries.
Revised: The same should apply for our black-box agent with symmetries.

Original: We demonstrate this property in the classic control tasks CartPole , Acrobot , and MountainCar .
Revised: We demonstrate this property in the classic control tasks CartPole , Acrobot , and MountainCar .

Original: Wemeta-train on each environment respectively with the original observation and action order.
Revised: We meta-train on each environment respectively with the original observation and action order.

Original: Wethen meta-test on either (1) the same conﬁguration or (2) across a permuted version.
Revised: We then meta-test on either (1) the same conﬁguration or (2) across a permuted version.

Original: The resultsare visualized in Figure 5.
Revised: The results are visualized in Figure 5.

Original: Due to the built-in symmetries, the performance does not degrade in theshufﬂed setting.
Revised: Due to the built-in symmetries, the performance does not degrade in the shufﬂed setting.

Original: Instead, our method quickly learns about the ordering of the relevant observationsand actions at meta-test time.
Revised: Instead, our method quickly learns about the ordering of the relevant observations and actions at meta-test time.

Original: In comparison, the MetaRNN baseline fails on the permuted settingwhere it was not trained on, indicating over-specialization.
Revised: In comparison, the MetaRNN baseline fails on the permuted setting where it was not trained on, indicating over-specialization.

Original: Thus, symmetries help to generalise toobservation permutations that were not encountered during meta training.
Revised: Thus, symmetries help to generalise to observation permutations that were not encountered during meta training.

Original: 5.4 Generalisation to Unseen Tasks

Original: [Figure]
Revised: [Figure]

Revised: We then meta-test on a setting where both the observations and actions are shufﬂed.

Original: (cid:21)(cid:198)(cid:170)(cid:123)(cid:173)(cid:137)(cid:154)(cid:123)(cid:155)(cid:183)(cid:177)(cid:234)(cid:270)(cid:31)(cid:123)(cid:155)(cid:123)(cid:173)(cid:103)(cid:148)(cid:137)(cid:202)(cid:123)(cid:270)(cid:183)(cid:160)(cid:270)
Revised: In this setting SymLA still performs well as it has meta-learned to identify observations and actions at meta-test time. In contrast, the MetaRNN fails to do so. Standard deviations are over 3 meta-training andmeta-testing runs.

Original: [Table]
Revised: [Figure]

Revised: Experiments: Generalize to similar environments ● Permutation invariance extends to object types ● One-hot image ○ Channel (cid:248): Player ○ Channel (cid:249): Heart ○ Channel (cid:250):

Revised: Trap ● Meta-Training on this conﬁguration

Original: [Figure]
Revised: [Figure]

Original:  The permutation invariance has further reaching consequences.
Revised: Generalisation to Unseen Tasks The permutation invariance has further reaching consequences.

Original: It extends to learning about tasks atmeta-test time.
Revised: It extends to learning about tasks at meta-test time.

Original: This enables generalisation to unseen tasks.
Revised: This enables generalisation to unseen tasks.

Original: We construct a grid world environment(see Figure 6) with two object types: A trap and a heart.
Revised: We construct a grid world environment (see Figure 6) with two object types: A trap and a heart.

Original: The agent and the two objects (one of eachtype) are randomly positioned every episode.
Revised: The agent and the two objects (one of each type) are randomly positioned every episode.

Original: Collecting the heart gives a reward of +1, whereas thetrap gives -1.
Revised: Collecting the heart gives a reward of +1, whereas the trap gives -1.

Original: All other rewards are zero.
Revised: All other rewards are zero.

Original: The agent observes its own position and the position of both objects.
Revised: The agent observes its own position and the position of both objects.

Original: The observation is constructed as an image with binary channels for the position and eachobject type.
Revised: The observation is constructed as an image with binary channels for the position and each object type.

Original: When meta-training on this environment, at meta-test time we observe in Figure 6 that the MetaRNNlearns to directly collect hearts in each episode throughout its lifetime.
Revised: When meta-training on this environment, at meta-test time we observe in Figure 6 that the MetaRNN learns to directly collect hearts in each episode throughout its lifetime.

Original: This is due to having overﬁttedto the association of hearts with positive rewards.
Revised: This is due to having overﬁtted to the association of hearts with positive rewards.

Original: In comparison, SymLA starts with near-zerorewards and learns through interactions which actions need to be taken when receiving particularobservations to collect the heart instead of the trap.
Revised: In comparison, SymLA starts with near-zero rewards and learns through interactions which actions need to be taken when receiving particular observations to collect the heart instead of the trap.

Original: With sufﬁcient environment interactions L wewould expect SymLA to eventually (after sufﬁcient learning) match the average reward per time ofthe MetaRNN in the non-shufﬂed grid world.
Revised: With sufﬁcient environment interactions L we would expect SymLA to eventually (after sufﬁcient learning) match the average reward per time of the MetaRNN in the non-shufﬂed grid world.

Original: Next, we swap the rewards of the trap and heart, i.e.the trap now gives a positive reward, whereas the heart gives a negative reward.
Revised: Next, we swap the rewards of the trap and heart, i.e. the trap now gives a positive reward, whereas the heart gives a negative reward.

Original: This is equivalentto swapping the input channels corresponding to the heart and trap.
Revised: This is equivalent to swapping the input channels corresponding to the heart and trap.

Original: We observe that SymLA stillgeneralises, learning at meta-test time about observations and their associated rewards.
Revised: We observe that SymLA still generalises, learning at meta-test time about observations and their associated rewards.

Original: In contrast,the MetaRNN now collects the wrong item, receiving negative rewards.
Revised: In contrast, the MetaRNN now collects the wrong item, receiving negative rewards.

Original: These results show that black-box meta RL with symmetries discovers a more general update rule that is less speciﬁc to thetraining tasks than typical MetaRNNs.
Revised: These results show that black-box meta RL with symmetries discovers a more general update rule that is less speciﬁc to the training tasks than typical MetaRNNs.

Original: 5.5 Generalisation to Unseen Environments

Original: [Figure]
Revised: [Figure]

Revised: Generalisation to Unseen

Original:  demonstrated howpermutation invariance can[Figure] lead to increased generalisation.
Revised: Environments We have demonstrated how permutation invariance can lead to increased generalisation.

Original: But can SymLAalso generalise betweenentirely different environments?
Revised: But can SymLA also generalise between entirely different environments?

Original: We show-case[Figure] how meta-training on agrid world environment allows generalisation to CartPole.
Revised: We show-case how metatraining on a grid world environment allows generalisation to CartPole.

Original: To simplify creditassignment, we use a densereward grid world where thereward is proportional to thechange in distance toward atarget position.
Revised: To simplify credit-assignment, we use a dense-reward grid world where the reward is proportional to the change in distance toward a target position.

Original: Both the target position, as well as the agent position are randomized.
Revised: Both the target position, as well as the agent position are randomized.

Original: The agentobserves its own position, all obstacles, and the target position as a binary image with multiplechannels.
Revised: The agent observes its own position, all obstacles, and the target position as a binary image with multiple channels.

Original: In the CartPole environment the agent is rewarded for being as upright and centeredas possible [Tassa et al., 2020].
Revised: In the CartPole environment the agent is rewarded for being as upright and centered as possible [24].

Original: Further, during meta-training, we randomly project observationslinearly for each lifetime.
Revised: Further, during meta-training, we randomly project observations linearly for each lifetime.

Original: This is necessary as in the grid world environment all observations arebinary whereas the CartPole environment has continuously varying observations.
Revised: This is necessary as in the grid world environment all observations are binary whereas the CartPole environment has continuously varying observations.

Original: This mismatchwould inhibit generalisation.
Revised: This mismatch would inhibit generalisation.

Original: In Figure 7 we demonstrate that meta-training withSymLA only on the
Revised: In Figure 7 we demonstrate that meta-training with

Original:  GridWorld environment allows reusing the same meta-learned learning algorithm to the CartPole environment.
Revised: SymLA only on the GridWorld environment allows reusing the same meta-learned learning algorithm to the CartPole environment.

Original: In contrast, the MetaRNN does not exhibit such generalisation.
Revised: In contrast, the MetaRNN does not exhibit such generalisation.

Original: Thus, learning algorithmswith symmetries can to some extent generalise between signiﬁcantly different environments.
Revised: Thus, learning algorithms with symmetries can to some extent generalise between signiﬁcantly different environments.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we identiﬁed symmetries that exist in backpropagation-based methods for meta RLbut are missing from black-box methods.
Revised: In this work, we identiﬁed symmetries that exist in backpropagation-based methods for meta RL but are missing from black-box methods.

Original: We hypothesized that these symmetries lead to bettergeneralisation of the resulting learning algorithms.
Revised: We hypothesized that these symmetries lead to better generalisation of the resulting learning algorithms.

Original: To test this, we extended a black-box meta learningmethod [Kirsch and Schmidhuber, 2020] that exhibits these same symmetries to the meta RL setting.
Revised: To test this, we extended a black-box meta learning method [12] that exhibits these same symmetries to the meta RL setting.

Original: This resulted in SymLA, a ﬂexible black-box meta RL algorithm that is less prone to over-ﬁtting.
Revised: This resulted in SymLA, a ﬂexible black-box meta RL algorithm that is less prone to over-ﬁtting.

Original: We demonstrated generalisation to varying numbers of arms in bandit experiments (unseen actionspaces), permuted observations and actions with no degradation in performance (unseen observationspaces), and observed the tendency of the meta-learned RL algorithm to learn about states and theirassociated rewards at meta-test time (unseen tasks).
Revised: We demonstrated generalisation to varying numbers of arms in bandit experiments (unseen action spaces), permuted observations and actions with no degradation in performance (unseen observation spaces), and observed the tendency of the meta-learned RL algorithm to learn about states and their associated rewards at meta-test time (unseen tasks).

Original: Finally, we showed that the discovered learningbehavior also transfers between grid world and (unseen) classic control environments. 
Revised: Finally, we showed that the discovered learning behavior also transfers between grid world and (unseen) classic control environments. A Related Work

Revised: Learning to reinforcement learn can be implemented with varying degrees of inductive biases.

Revised: Black-Box Meta RL Black-box meta RL can be implemented by policies that receive the reward signal as input [21] and use memory to learn, such as recurrence in RNNs [10, 25, 6].

Revised: These approaches do not feature the symmetries discussed in this paper which leads to a tendency of overﬁtting.

Revised: Learned Learning Rules & Fast Weights In the supervised and reinforcement learning contexts, learned learning rules [4] or fast weights [20, 22, 15, 19, 16] describe (meta-)learned mechanisms (slow weights) that update fast weights to implement learning.

Revised: This often involves outer-products and can be generalised to black-box meta learning with parameter sharing [12].

Revised: None of these approaches feature all of the symmetries we discuss above to meta learn RL algorithms.

Revised: Backpropagation-based Meta RL Alternatives to black-box meta RL include learning a weight initialization and adapting it with a human-engineered RL algorithm [7], warping computed gradients [8], meta-learning hyper-parameters [23, 28] or meta-learning objective functions corresponding to the learning algorithm

Revised: Neural Network Symmetries Symmetries in neural networks have mainly been investigated to reﬂect the structure of the input data.

Revised: This includes applications of convolutions [9], deep sets [30], graph neural networks [27], and geometric deep learning [5].

Revised: While many meta learning algorithms exhibit symmetries [4], in particular backpropagation-based meta learning [2, 7, 8, 13], the effects of these symmetries have not been discussed in detail.

Revised: In this work, we provide such a discussion and experimental investigation in the context of meta RL.

Revised: Algorithm

Revised: Require: Distribution over RL environment(s) p ( e ) θ ← initialize LSTM parameters while meta loss has not converged do (cid:46)

Revised: Outer loop in parallel over envs e ∼ p

Revised: ( e ) and samples φ ∼ N ( φ | θ, Σ) { h ab } ← initialize LSTM states ∀ a, b o 1 ∼ p ( o 1 ) (cid:46)

Revised: Initialize environment e for t ∈ { 1 , . . .

Revised: , L } do (cid:46) Inner loop over lifetime in environment e h ab ← f LSTM ( h ab , o t,a , a t − 1 ,b , r t − 1 , −→ m b , ←− m a ) ∀ a, b (cid:46) Equation 8 −→ m b ← (cid:80) a f −→ m ( h ab )

Revised: ∀ b (cid:46) Create forward messages ←− m a ← (cid:80) b f ←− m ( h ab ) ∀ a (cid:46) Create backward messages y ← −→ m ·(cid:46) Read out action a t ∼ p ( a t ; y )

Revised: (cid:46)

Revised: Sample action from distribution parameterized by y

Revised: Send action a t to environment e , observe o t +1 and r t θ ← θ + α ∇

Revised: [Equation]

Revised: (cid:46) Update θ strategies (Equation 9)

Revised: B Bandits from Wang et al. [25]

Revised: In our experiments, we use bandits of varying difﬁculty from Wang et al.

Revised: Let p 1 be the probability of the ﬁrst arm for a payout of r = 1 , r = 0 otherwise, and p 2 the payout for the second arm.

Revised: Then, we deﬁne the

Revised: • uniform independent bandit with p 1 ∼ U [0 , 1] and p 2 ∼ U [0 , 1] , • uniform dependent bandit with p 1 ∼ U [0 , 1] and p 2 = 1 − p 1 , • easy dependent bandit with p 1 ∼ U { 0 . 1 , 0 . 9 } and p 2 = 1 − p 1 , • medium dependent bandit with p 1 ∼ U { 0 . 25 , 0 . 75 } and p 2 = 1 − p 1 , • hard dependent bandit with p 1 ∼ U { 0 . 4 , 0 . 6 } and p 2 = 1 − p 1 .

Revised: C Hyper-parameters C.1 SymLA Architecture

Revised: We use a single recurrent layer, K = 1 , with a message size of ←− M = 8 and −→ M = 8 .

Revised: To produce the next state h ab according to Equation 8, we use parameter-shared LSTMs with a hidden size of

Revised: N = 16 ( N = 64 for bandits to match Wang et al.

Revised: [25]) and run the recurrent cell for 2 micro ticks.

Revised: C.2 Meta Learning / Outer Loop

Revised: We estimate gradients ∇ θ using evolutionary strategies [18] with 10 evaluations per population sample to estimate the ﬁtness value ( 100 evaluations for bandits).

Revised: Then, we apply those using Adam with a learning rate of α = 0 .

Revised: 01 , β 1 = 0 .

Revised: 9 , and β 2 = 0 .

Revised: 999 ( α = 0 .

Revised: 2 for bandits).

Revised: We use a ﬁxed noise standard deviation of σ = 0 .

Revised: 035 ( σ = 0 .

Revised: 2 for bandits) and a population size of 512 .

Original: {+Our inner loop has a length of L = 
+}
Revised: {+Our inner loop has a length of L = 
+}

Revised: ( L = 100 for bandits), concatenating multiple episodes.

Revised: We meta-optimize for 4 , 000 outer steps for bandit experiments, and 20 , 000 otherwise.

Revised: C.3 Generalisation to Unseen Environments

Revised: We apply a random linear transformation (Glorot normal) to environment observations, mapping those to a 16 -dimensional vector.