Original: Deep Policy Generators Anonymous Author(s)
Revised: Goal-Conditioned Generators of Deep Policies

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Traditional Reinforcement Learning (RL) learns policies that maximize expectedreturn.
Revised: Goal-conditioned Reinforcement Learning (RL) aims at learning optimal policies, given goals encoded in special command inputs.

Original: Here we study  neural nets (NNs) that learn to generate  policies in form ofcontext-specific weight matrices, similar to Fast Weight Programmers and othermethods from the 1990s.
Revised: Here we study goal-conditioned neural nets (NNs) that learn to generate deep NN policies in form of contextspecific weight matrices, similar to Fast Weight Programmers and other methods from the 1990s.

Original: Using context commands of the form "generate a policythat achieves a desired expected return," our NN generators combine powerfulexploration of parameter space with greedy command choices to iteratively findbetter and better policies.
Revised: Using context commands of the form “generate a policy that achieves a desired expected return,” our NN generators combine powerful exploration of parameter space with generalization across commands to iteratively find better and better policies.

Original: A form of weight-sharing HyperNetworks and policyembeddings scales our method to generate deep NNs.
Revised: A form of weight-sharing HyperNetworks and policy embeddings scales our method to generate deep NNs.

Original: Experiments show how asingle learned policy generator can produce policies that achieve any return seenduring training.
Revised: Experiments show how a single learned policy generator can produce policies that achieve any return seen during training.

Original: Finally, we evaluate our algorithm on a set of continuous controltasks where it exhibits competitive performance.
Revised: Finally, we evaluate our algorithm on a set of continuous control tasks where it exhibits competitive performance.

Original:  Introduction
Revised: 1. Introduction

Original: Reinforcement learning (RL)(Kaelbling et al., 1996; Sutton and Barto, 2018) is a frameworkfor trainingartificial agents to  maximize  rewards in possibly non-continuous, non-differentiableenvironments.
Revised: General reinforcement learning (RL) is about training agents to execute action sequences that maximize cumulative rewards (Kaelbling et al., 1996; van Hasselt, 2012; Schmidhuber, 1990).

Original: Typical RL methods learn value functions (Sutton, 1988) or estimate stochastic policygradients (with possibly high variance)
Revised: Goal-conditioned RL agents can learn to solve many different tasks, where the present task is encoded by special command inputs (Schmidhuber & Huber, 1991; Schaul et al., 2015).

Original: (Williams, 1992; Sutton et al., 1999). In contrast, Upside Down Reinforcement Learning (UDRL) (Srivastava et al., 2019; Schmidhuber,2019) uses supervised learning to train  RL  agents conditioned on sequences of special commandinputs of the form "act in the environment and achieve a desired return within so much time.
Revised: Upside-down RL (UDRL) (Srivastava et al., 2019; Schmidhuber, 2019) and related methods (Ghosh et al., 2019), however, use supervised learning to train goal-conditioned RL agents. UDRL agents receive command inputs of the form "act in the environment and achieve a desired return within so much time" (Schmidhuber, 2019).

Original: Usinghindsight learning, UDRL transforms the RL problem into the problem of predicting actions, givenreward commands.
Revised: Typically, hindsight learning (Andrychowicz to: et al., 2017; Rauber et al., 2018) is used to transform the RL problem into the problem of predicting actions, given reward commands.

Original: This is quite powerful: when an UDRL agent interacts with an environment, givena random command c , and achieves return r , its behavior would have been optimal if the commandhad been r .
Revised: This is quite powerful. Consider a commandbased agent interacting with an environment, given a random command c , and achieving return r . Its behavior would have been optimal if the command had been r .

Original: Hence UDRL’s parameters can be learned by maximizing the likelihood of the agent’sbehavior, given command r .
Revised: Hence the agent’s parameters can be learned by maximizing the likelihood of the agent’s behavior, given command r .

Original: Unfortunately, in the episodic setting, many behaviors may sastisfy thesame command.
Revised: Unfortunately, in the episodic setting, many behaviors may satisfy the same command.

Original: Hence the function to be learned may be highly multimodal, and a simple maximumlikelihood approach may fail to capture the variability in the data.
Revised: Hence the function to be learned may be highly multimodal, and a simple maximum likelihood approach may fail to capture the variability in the data.

Original: To overcome this limitation, we introduce PoliGen , a novel method to generate return-conditionedpolicies evaluated in parameter space.
Revised: To overcome this limitation, we introduce PoliGen , a novel method for return-conditioned generation of policies evaluated in parameter space.

Original: First, we use a Fast Weight Programmer (FWP) (Schmidhuber,1992, 1993; Ha et al., 2016) to generate the parameters of a desired policy.
Revised: First, we use a Fast Weight Programmer (FWP) (Schmidhuber, 1992; 1993; Ha et al., 2016) to generate the parameters of a desired policy, given a “desired return” command.

Original: Then, we evaluate thepolicy using a parameter-based value function (Harb et al., 2020; Faccio et al., 2020).
Revised: Then, we evaluate the policy using a parameter-based value function (Harb et al., 2020; Faccio et al., 2020).

Original: This allowsfor end-to-end optimization of the  generator producing return-conditioned policies by matching thecommands (desired returns) to the observed returns.
Revised: This allows for end-to-end optimization of the return-conditioned generator producing deep NN policies by matching the commands (desired returns) to the evaluated returns.

Original: The paper is structured as follows: Section 2 introduces the MDP frameworks for action-based andparameter-based methods; Section 3 reviews the concept of Fast Weight Programmers; Section 4describes our method (PoliGen) including architectural choices; Section 5 evaluates our method oncontinuous control tasks where it demonstrates competitive performance.

Original: Our analysis shows how a single learned policy generator can produce policies with any desired return seen during training.

Original: Finally, we discuss related and future work in Sections 6 and 7.

Original:  Background
Revised: 2. Background

Original: We consider a Markov Decision Process (MDP) (Stratonovich, 1960; Puterman, 2014)
Revised: We consider a Markov Decision Process (MDP) (Puterman, 2014)

Original: M =( S , A , P, R, γ, µ 0 ) .
Revised: M = ( S , A , P, R, γ, µ 0 ) .

Original: At each time step t , an artificial agent observes a state s t ∈ S , chooses anaction a t ∈ A , obtains a reward r t =
Revised: At each time step t , an artificial agent observes a state s t ∈ S , chooses an action a t ∈ A , obtains a reward r t =

Original: R ( s t , a t ) and transitions to a new state with probability P ( s t +1 | s t , a t ) .
Revised: R ( s t , a t ) and transitions to a new state with probability P ( s t +1 | s t , a t ) .

Original: The initial state of the agent is chosen with probability µ 0 .
Revised: The initial state of the agent is chosen with probability µ 0 .

Original: The behavior of the agentis expressed through its stochastic policy π θ : S → ∆( A ) , where θ ∈ Θ are the policy parameters.
Revised: The behavior of the agent is expressed through its stochastic policy π θ : S → ∆( A ) , where θ ∈ Θ are the policy parameters.

Original: Iffor each state s there is an action a such that π θ ( a | s ) = 1 , we will call the policy deterministic.
Revised: If for each state s there is an action a such that π θ ( a | s ) = 1 , we will call the policy deterministic.

Original: Theagent interacts with the environment through episodes, starting from the initial states, and endingeither when the agent reaches a set of particular states—these can be failing states or goal states—orwhen it hits a time horizon H .
Revised: The agent interacts with the environment through episodes, starting from the initial states, and ending either when the agent reaches a set of particular states—these can be failing states or goal states—or when it hits a time horizon H .

Original: We define a trajectory τ ∈ T as the sequence of state-action pairs thatan agent encounters during an episode in the MDP τ =
Revised: We define a trajectory τ ∈ T as the sequence of state- action pairs that an agent encounters during an episode in the MDP τ =

Original: ( s τ, 0 , a τ, 0 , s τ, 1 , a τ, 1 , . . .
Revised: ( s τ, 0 , a τ, 0 , s τ, 1 , a τ, 1 , . . .

Original: , s τ,T , a τ,T ) , where49 T denotes the time-step at the end of the episode ( T ≤ H ).
Revised: , s τ,T , a τ,T ) , where T denotes the time-step at the end of the episode ( T ≤ H ).

Original: The return of a trajectory R ( τ ) is definedas the cumulative discounted sum of rewards over the trajectory R ( τ ) =
Revised: The return of a trajectory R ( τ ) is defined as the cumulative discounted sum of rewards over the trajectory R ( τ ) =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: γ t R ( s τ,t , a τ,t ) , whereγ ∈ (0 , 1] is the discount factor.
Revised: γ t R ( s τ,t , a τ,t ) , where γ ∈ (0 , 1] is the discount factor.

Original: The RL problem consists in finding the policy π θ ∗ that maximizes the expected return obtained fromthe environment, i.e.π θ ∗ = arg max π θ J
Revised: The RL problem consists in finding the policy π θ ∗ that maximizes the expected return obtained from the environment, i.e.

Original: [Equation]
Revised: [Equation]

Original: θ [Equation] [Equation] where p ( τ | θ ) = µ 0 ( s 0) (cid:81)
Revised: where p ( τ | θ ) = µ 0 ( s 0) (cid:81)

Original: Tt =

Revised: Tt =

Original: π θ ( a t | s t ) P ( s t +1 | s t , a t ) is the distribution over trajectories inducedby π θ in the MDP.
Revised: π θ ( a t | s t ) P ( s t +1 | s t , a t ) is the distribution over trajectories induced by π θ in the MDP.

Original: When the policy is stochastic and differentiable, by taking the gradient of J ( θ )with respect to the policy parameters we obtain an algorithm called REINFORCE (Williams, 1992): [Equation]
Revised: In parameter-based methods (Sehnke et al., 2010; 2008;

Original: In parameter-based methods (Sehnke et al., 2010, 2008; Salimans et al., 2017; Mania et al., 2018),at the beginning of each episode, the weights of a policy are sampled from a distribution ν ρ ( θ ) ,called the hyperpolicy, which is parametrized by ρ .
Revised: Salimans et al., 2017; Mania et al., 2018), at the beginning of each episode, the weights of a policy are sampled from a distribution ν ρ ( θ ) , called the hyperpolicy, which is parametrized by ρ .

Original: Typically, the stochasticity of the hyperpolicy issufficient for exploration, and deterministic policies are used.
Revised: Typically, the stochasticity of the hyperpolicy is sufficient for exploration, and deterministic policies are used.

Original: The RL problem translates into findingthe hyperpolicy parameters ρ maximizing expected return, i.e.
Revised: The RL problem translates into finding the hyperpolicy parameters ρ maximizing expected return, i.e.

Original: [Equation]
Revised: [Equation]

Original: ρ ∗ = arg max ν

Original: [Equation]

Original: J [Equation] [Equation] This objective is maximized by taking the gradient of J ( ρ ) with respect to the hyperpolicy parameters:∇
Revised: This objective is maximized by taking the gradient of J ( ρ ) with respect to the hyperpolicy parameters.

Original: [Equation]

Original: [Equation] ) = (cid:82) Θ (cid:82) T [Equation] ( θ ) ∇
Revised: It is intuitive to observe that, when the hyperpolicy becomes deterministic and the policy is stochastic, the dependency on ρ is lost and the policy parameters θ can be directly maximized using

Original: [Equation]
Revised: Equation 1.

Original: log ν ρ ( θ ) p ( τ | θ ) R ( τ )

Original: d τ d θ .

Original: This gradient can be either approximatedthrough samples (Sehnke et al., 2010, 2008; Salimans et al., 2017) or estimated using finite differencemethods (Mania et al., 2018).

Original: This only requires differentiability and stochasticity of the hyperpolicy.

Original: It is intuitive to observe that, when the hyperpolicy becomes deterministic and the policy is stochastic,the dependency on ρ is lost and the policy parameters θ can be directly maximized using Equation 1.

Original: Since the optimization problem is episodic, we can set the discount factor γ to 1.
Revised: Since the optimization problem is episodic, we can set the discount factor γ to 1.

Original:  Fast Weight Programmers
Revised: 3. Fast Weight Programmers

Original: Fast Weight Programmers (FWPs) (Schmidhuber, 1992, 1993)  generate changes of weights of anothernetwork conditioned on some contextual input.
Revised: Fast Weight Programmers (FWPs) (Schmidhuber, 1992; 1993) are NNs that generate changes of weights of another NN conditioned on some contextual input.

Original: In our  case, the context corresponds to the desiredreturn a generated policy should obtain.
Revised: In our UDRLlike case, the context is the desired return to be obtained by a generated policy.

Original: The outputs of the FWP are the policy parameters θ ∈ Θ .
Revised: The outputs of the FWP are the policy parameters θ ∈ Θ .

Original: Formally, our FWP is adeterministic function G ρ : R n c −→ Θ , where c ∈ R n c is the contextinput and ρ ∈ P are the FWP parameters.
Revised: Formally, our FWP is a function G ρ : R n c −→ Θ , where c ∈ R n c is the context-input and ρ ∈ P are the FWP parameters.

Original: Here, we consider a probabilistic FWP of the formg ρ ( θ | c ) =
Revised: Here, we consider a probabilistic FWP of the form g ρ ( θ | c ) =

Original: G ρ ( c ) + ϵ , with ϵ ∼ N (0 , σ 2 I ) and σ is fixed.
Revised: G ρ ( c ) + ϵ , with ϵ ∼ N (0 , σ 2 I ) and σ is fixed.

Original: In this setting, the FWP conditioned oncontext c induces a probability distribution over the parameter space, similar to the one induced bythe hyperpolicy in Section 2.
Revised: In this setting, the FWP conditioned on context c induces a probability distribution over the parameter space, similar to the one induced by the hyperpolicy in Section 2.

Original: If the FWPis used to generate the weights of a policy, we can rewritethe RL objective, making it context-dependent:
Revised: Using the FWP to generate the weights of a policy, we can rewrite the RL objective, making it context-dependent:

Original: [Equation]
Revised: [Equation]

Original: Compared to Eq. 2, J ( ρ, c ) induces a set of optimization problems that now are context-specific 1 .
Revised: Compared to Eq. 2, J ( ρ, c ) induces a set of optimization problems that now are context-specific 2 .

Original: Here, J ( ρ, c ) is the expected return for generating a policy with a generator parametrized by ρ , whenobserving context c .
Revised: Here, J ( ρ, c ) is the expected return for generating a policy with a generator parametrized by ρ , when observing context c .

Original: Instead of optimizing Eq.
Revised: Instead of optimizing Eq.

Original: 2 using policy gradient methods, we are interestedin learning a good policy through pure supervised learning by following a sequence of contextcommands of the form "generate a policy that achieves a desired expected return.
Revised: 2 using policy gradient methods, we are interested in learning a good policy through pure supervised learning by following a sequence of context-commands of the form “generate a policy that achieves a desired expected return.”

Original: " Under suchcommands, for any c , the objective J ( ρ, c ) can be optimized with respect to ρ toequal c .
Revised: Under such commands, for any c , the objective J ( ρ, c ) can be optimized with respect to ρ to

Original: FWPs offera suitable framework for this setting, since the generator network can learn to create weights of thepolicy network so that it achieves what the given context requires.
Revised: offer a suitable framework for this setting, since the generator network can learn to create weights of the policy network so that it achieves what the given context requires.

Original:  Deep Policy Generators (PoliGen)
Revised: 4. Deep Policy Generators (PoliGen)

Original: [Figure]
Revised: [Figure]

Revised: c o n c a t .

Revised: c

Revised: o n c a t .

Original:  Here we develop PoliGen , our algorithm togenerate policies that achieve any desired return.
Revised: c o n c a t . Environment Here we develop PoliGen , our algorithm to generate policies that achieve any desired return.

Original: In the supervised learning scenario, itis straightforward to learn the parameters ofthe FWP that minimize the error L G ( ρ ) =94 E
Revised: In the supervised learning scenario, it is straightforward to learn the parameters of the FWP that minimize the error L G ( ρ ) = E

Original: [Equation] [Equation]
Revised: [Equation] [Figure]

Original: θ

Original: [Equation]
Revised: [Equation]

Original: − c

Original: [Equation]
Revised: [Equation]

Original:  2 ] , where the context95 c comes from some set of possible commands D .
Revised: J ( θ ) − c ) 2 ] , where the context c comes from some set of possible commands D .

Original: This is because in supervised learning J ( θ )the return is a differentiable function of the policy parameters. However, in RL, it is not thecase.
Revised: This is because in supervised learning J ( θ ) , the expected return, is a differentiable function of the policy parameters, unlike in general RL.

Original: Therefore,  we learn an evaluator function V w : Θ −→ R parametrized by w thatestimates J ( θ ) using supervised learning (Faccio et al.,2020).
Revised: Therefore, to make the objective differentiable, we learn an evaluator function V w : Θ −→ R parametrized by w that estimates J ( θ ) using supervised learning (Faccio et al.,

Original: This function is a map fromthe policy parameters to the expected return.
Revised: This function is a map from the policy parameters to the expected return.

Original: Once V is learned, the objective L G ( ρ ) canbe optimized end-to-end, like in the supervisedlearning scenario, to directly learn the generator’s parameters.
Revised: Once V is learned, the objective L G ( ρ ) can be optimized end-to-end, like in the supervised learning scenario, to directly learn the generator’s parameters.

Original: Concretely, we minimize L G ( ρ ) =
Revised: Concretely, we minimize L G ( ρ ) =

Original: E c ∈ D
Revised: E c ∈ D

Original: [( V w ( G ρ ( c )) − c ) 2 ] to learn theparameters ρ .
Revised: [( V w ( G ρ ( c )) − c ) 2 ] to learn the parameters ρ .

Original: Our method is described in Algorithm 1 andconsists of three steps.
Revised: Our method is described in Algorithm 1 and consists of three steps.

Original: First , at each iteration acommand c is chosen following some strategy.
Revised: First , in each iteration, a command c is chosen following some strategy.

Original: Ideally, to ensure that the generated policies arebetter and better, one should query the generator for larger returns.
Revised: Ideally, to ensure that the generated policies improve over time, the generator should be instructed to produce larger and larger returns.

Original: We discuss commandstrategies in the next paragraph.
Revised: We discuss command strategies in the next section.

Original: The generatorobserves c and produces policy π θ which is runin the environment.
Revised: The generator observes c and produces policy π θ which is run in the environment.

Original: The return and the policy ( r, θ ) are then stored in a replay buffer.
Revised: The return and the policy ( r, θ ) are then stored in a replay buffer.

Original: Second , theevaluator function is trained to predict the return of the policies observed during training.
Revised: Second , the evaluator function is trained to predict the return of the policies observed during training.

Original: This isachieved by minimizing MSE loss L V ( w ) =
Revised: This is achieved by minimizing MSE loss L V ( w ) =

Original: E ( r,θ ) ∈ B
Revised: E ( r,θ ) ∈ B

Original: [( r − V w ( θ )) 2 ] .
Revised: [( r − V w ( θ )) 2 ] .

Original: Third , we use the learnedevaluator to directly minimizeL G ( ρ ) =
Revised: Third , we use the learned evaluator to directly minimize

Original: E r ∈ B [( r − V w ( Gρ ( r ))) 2 ] . Output : Learned G
Revised: L G ( ρ ) =

Original: Initialize generator and critic weights ρ, w , set initial return command c

Original: repeat :

Original: Sample policy parameters θ ∼ g ρ ( θ, c )

Original: Generate an episode s 0 , a 0 , r 1 , s 1 , a 1 , r 2 , . . .

Original: , s T − 1 , a T − 1 , r T with policy π θ

Original: Compute return r = (cid:80) Tk =1 r k

Original: Store ( r, θ ) in the replay buffer D for many steps do :

Original: Sample a batch B = { ( r, θ ) } from D

Original: Update evaluator by stochastic gradient descent: ∇ w E ( r,θ

Original: ) ∈ B
Revised: E r ∈ B

Original: ( r − V w ( θ )) 2 ] end for for do :

Original: Sample a batch B = { r from D

Original: Update generator by stochastic gradient descent: ρ E r

Original: B

Original: ( r − V w ( G ρ  r )) 2 ] end for
Revised: [( r − V w ( G ρ ( r ))) 2 ] .

Original: Set next command using strategy until convergence

Original: Choosing the command The strategy of choosing the command c before interacting with theenvironment is important.

Original: Intuitively, asking the generator to produce low return policies will nothelp finding better policies.

Original: On the other hand, asking for too much will produce policies that areout of distribution from the training data, and the generator cannot be trusted to produce such values.

Original: Hence it is reasonable to ask the generator to produce a return close to the highest one observed sofar.

Original: More on the command strategy can be found in Section 5.1.

Original: Scaling to deep policies Both generating and evaluating the weights of a deepNN policy usingsimple feedforward architectures like an MLP isvery difficult for larger policies.
Revised: Scaling to deep policies Both generating and evaluating the weights of a deep feedforward MLP-based policy is difficult for large policies.

Original: The sheer numberof policy weights, as well as their lack of easily recognizable structure, requires special solutions forgenerator and evaluator.
Revised: The sheer number of policy weights, as well as their lack of easily recognizable structure, requires special solutions for generator and evaluator.

Original: To scale FWPs to deep policies, we rely on the relaxed weight-sharing of HyperNetworks (Ha et al., 2016) for the generator, and on  a fingerprinting mechanism (Harb et al. for the evaluator.
Revised: To scale FWPs to deep policies, we rely on the relaxed weightsharing of hypernetworks (Ha et al., 2016) for the generator, and on parameter-based value functions (Faccio et al., 2020) using a fingerprinting mechanism (Harb et al., 2020) for the evaluator.

Original: We discuss these two approaches in the next section.
Revised: We discuss these two approaches in Appendix B.

Original: 4.1 HyperNetwork

Original: The idea behind certain feed-forward FWPs called HyperNetworks (Ha et al., 2016) is to split theparameters of the generated network θ into smaller slices s l .

Original: A shared NN H with parametersξ receives as input a learned embedding z l and outputs the slice s l , i.e. s l = H ξ ( z l ) for each l .

Original: Following von Oswald et al.

Original: (2020), further context information can be given to H in form of anadditional conditioning input c , which can be either either scalar or vector-sized: s l = H ξ ( z l , c ) .

Original: Then the weights are combined by concatenating all generated slices:

Original: [Equation]

Original: The splitting of θ into slices and the choice of H depend on the specific architecture of the generatedpolicy.

Original: Here we are interested in generating MLP policies.

Original: The policy’s parameters θ consist ofweight matrices K j , where j indicates the layer.

Original: We use an MLP H ξ to generate each slice of eachweight matrix: the HyperNetwork generator G ρ splits each weight matrix into slices s j mn ∈ R f × f ,where j is the policy layer, and m, n are indexes of the slice in weight matrix of layer l .

Original: For each ofthese slices, a small embedding vector z jmn ∈ R d is learned.

Original: Our network H ξ is an MLP, followed bya reshaping operation that turns a vector of size f 2 into a f × f matrix:

Original: [Equation]

Original: The slices are then concatenated over two dimensions to obtain the full weight matrices:

Original: [Equation]

Original: The full HyperNetwork generator G ρ consists of the shared network H ξ , as well as all embeddingsz jmn .

Original: Its learnable parameters are ρ = { ξ, z jmn ∀ m, n, j } .

Original: MLP r e - s h a p e

Original: Our aim for the generator G ρ is to dynamically generatepolicy parameters, conditioned on the total return thesepolicies should achieve.

Original: For this, our conditioning inputc is simply this scalar return command.

Original: It is appended toeach learned slice embedding z jmn .

Original: The resulting vectorsare the inputs to the network H .

Original: Figure 2 shows a diagramthis process.

Original: For the the slicing to work, the width and height of theweight matrices have to be multiples of f .

Original: For the hidden layers of an MLP, this is easily achieved since wecan freely choose the number neurons.

Original: For the input andoutput layers, however, we are constrained by the environment’s observation and action dimension, respectively.

Original: Toaccommodate any number of input and output neurons,we use dedicated networks h i and h o for the input andoutput layer.

Original: The generated slices have the shape f × N ifor the input layer ( N i is the number of input neurons) and

Original: N o × f for the output layer ( N o is the number of outputneurons).

Original: 4.2 Policy Fingerprinting

Original: We use a policy fingerprinting mechanism (Harb et al., 2020) as an effective method to evaluate theperformance of multiple NNs while using a single function.

Original: Policy fingerprinting works by giving aset of learnable probing states as input to the policy π θ .

Original: The resulting outputs of the policy—calledprobing actions—are concatenated and given as input to an MLP U that computes the prediction

Original: V w ( θ ) .

Original: Here the set of parameters w of this evaluator consists of the MLP parameters ϕ

Original: and all theparameters of the probing states.

Original: When training V w , the probing states learn to query the policy inmeaningful situations, so that the policy’s success can be judged by its probing actions.

Original: Fingerprintingis similar to a previous technique (Schmidhuber, 2015) where an NN learns to send queries (sequencesof activation vectors) into another already trained NN, and learns to use the answers (sequences ofactivation vectors) to improve its own performance.

Original: Figure 1 shows a diagram of our method with a

Original: HyperNetwork generator and a fingerprinting value function.

Original: The benefits of policy fingerprinting over directly observing policy weights become apparent assoon as we have at least one hidden layer in an MLP policy: the weights then have a large numberof symmetries, i.e., many different weight configurations that are entirely equivalent in terms ofthe input-output mapping of the network.

Original: The main symmetries are across permutations of hiddenneurons and scalings of the weight matrices (K˚urková and Kainen, 1994).

Original: The probing actions of the fingerprinting mechanism are invariant with respect to these symmetries.

Original: They are, in fact, invariant even with respect to the general policy architecture.

Original: For our purposes, thisbrings advantages not only for the value function V w , but also for the generator: the gradients w.r.t.the generator’s weights ρ are obtained by backpropagating through V w .

Original: If V w is fingerprinting-based,these gradients will point only in directions which actually change the policy’s probing actions.

Original: Consequently, the generator will waste no capacity trying to capture differences in the policy weightsthat have no effect on the policy’s probing actions (which are proxies for the policy’s general behaviorin the environment).

Original: [Figure] Experiments
Revised: 5. Experiments

Original: [Figure]

Original: We empirically evaluate PoliGen as follows:
Revised: We empirically evaluate PoliGen as follows:

Original: First, we show competitive performance on commoncontinuous control problems.
Revised: First, we show competitive performance on common continuous control problems.

Original: Then we use the the learned fingerprinting mechanism to visualize thepolicies created by the generator over the course of training, and investigate its learning behavior.
Revised: Then we use the the learned fingerprinting mechanism to visualize the policies created by the generator over the course of training, and investigate its learning behavior.

Original: 5.1 Results on continuous control RL environments
Revised: 5.1. Results on continuous control RL environments

Original: [Figure] We evaluate our method on continuous control tasks from the MuJoCo (Todorov et al., 2012) suite.
Revised: We evaluate our method on continuous control tasks from the MuJoCo (Todorov et al., 2012) suite.

Original: Parameter-based Augmented Random Search (ARS) (Mania et al., 2018) serves as a strong baseline.
Revised: Parameter-based Augmented Random Search (ARS) (Mania et al., 2018) serves as a strong baseline.

Original: We also compare our method to the Deep Deterministic Policy Gradient (DDPG) algorithm (Silveret al., 2014), another popular method for continuous control tasks.
Revised: We also compare our method to the Deep Deterministic Policy Gradient (DDPG) algorithm (Silver et al., 2014), another popular method for continuous control tasks.

Original: In the experiments, all policies are MLPs with two hidden layers, each having 256 neurons.
Revised: In the experiments, all policies are MLPs with two hidden layers, each having 256 neurons.

Original: Ourmethod uses the same set of hyper-parameters in all environments.
Revised: Our method uses the same set of hyperparameters in all environments.

Original: For ARS, we tune step size,population size, and noise independently for each environment.
Revised: For ARS, we tune step size, population size, and noise independently for each environment.

Original: For DDPG, we use the establishedset of default hyper-parameters.
Revised: For DDPG, we use the established set of default hyperparameters.

Original: Details can be found in Appendix A.
Revised: Details can be found in Appendix B.

Original: We find that while always  asking to generate a policy with return equal to the best return seen in thehistory, there is a slight advantage when asking for more than that.
Revised: We find that while always [Figure] asking to generate a policy with return equal to the best return ever seen, there is a slight advantage when asking for more than that.

Original: In particular, we demonstrate thata simple strategy such that "produce a policy whose return is 20 above the one of the best policyseen so far" can be very effective.
Revised: In particular, we demonstrate that a simple strategy such as “produce a policy whose return is 20 above the one of the best policy seen so far” can be very effective.

Original: We present an ablation showing that this strategy is slightly betterthan the strategy "produce a policy whose return equal to the one of the best policy seen so far" in
Revised: We present an ablation showing that this strategy is slightly better than the strategy “produce a policy whose return equal to the one of the best policy seen so far” in

Original: Appendix B.4.
Revised: Appendix C.4.

Original: This suggests that our method’s improvements are duenot only to the exploration inparameter space but also to the greediness of the command choice.
Revised: This suggests that our method’s success is partially due to its ability to learn to explore performance improvements.

Original: For our method and ARS, we use observation normalization (see Mania et al. ; Faccio et al.(2020)).
Revised: For our method and ARS, we use observation normalization (see (Mania et al., 2018; Faccio et al., 2020)).

Original: Furthermore, following ARS, the survival bonus of +1 for every timestep is removed for the Hopper-v3 environment, since for parameter-based methods it leads to the local optimum of onlystaying alive without any movement.
Revised: Furthermore, following ARS, the survival bonus of +1 for every timestep is removed for the Hopper-v3 environment, since for parameter-based methods it leads to the local optimum of staying alive without any movement.

Original: In environments that do not have a fixed episode length, quickly failing bad policies from the earlystages of training tend to dominate the replay buffer.
Revised: In tasks without fixed episode length, quickly failing bad policies from the early stages of training tend to dominate the replay buffer.

Original: To counteract this, we introduce a recency biaswhen sampling training batches from the buffer, assigning higher probability to newer policies.
Revised: To counteract this, we introduce a recency bias when sampling training batches from the buffer, assigning higher probability to newer policies.

Original: It istreated as an additional hyperparameter.
Revised: It is treated as an additional hyperparameter.

Original: Below we provide an ablation showing the importance ofthis component.
Revised: In Appendix C.2 we provide an ablation showing the importance of this component.

Original: Figure 3 shows our main experimental result.
Revised: Figure 2 shows our main experimental result.

Original: Our Algorithm 1 performs competitively in the tested environments.
Revised: Our Algorithm 1 performs competitively in the tested environments.

Original: In Swimmer and Hopperenvironments, our method learns faster than ARS, while eventually reaching the same asymptotic performance.
Revised: In Swimmer and Hopper environments, our method learns faster than ARS, while eventually reaching the same asymptotic performance.

Original: In MountainCarContinuous, DDPG is unable to explore the action space, and parameterbased methods quickly learn the optimal policy.
Revised: In MountainCarContinuous, DDPG is unable to explore the action space, and parameter-based methods quickly learn the optimal policy.

Original: For a comparison to UDRL with episodic resets, see Appendix B.1.
Revised: For a comparison to UDRL with episodic resets, see Appendix C.1.

Original: Obtaining suitable policies from the start When using a randomly initialized generator, we foundthat the weights of the policy generated were far from those of typical initialization schemes.

Original: Inparticular, the standard PyTorch (Paszke et al., 2019) initialization is uniform in [ − 1 / √ n, 1 / √ n ] ,where n is the number of neurons in the previous layer, resulting in a distribution uniform in[ − 0 .

Original: in the second and last layers.

Original: Our network was approximately generating weightsmuch larger and roughly uniform in every layer of the NN.

Original: We therefore scale our output suchthat it is close to the default initialization.

Original: Concretely, we multiply for each layer the output of thegenerator by 2 / √ n , where n is the number of neurons in the previous layer.

Original: Here we provide anablation showing that this choice is crucial.

Original: Figure 4 shows the importance of scaling the output ofthe generator in Swimmer and Hopper.

Original: We compare this with and without weighted sampling fromthe replay buffer.

Original: We observe that in Swimmer, output scaling is very important, while in Hopper,most of the performance gain is due to weighted sampling.

Original: This choice of output scaling is rather anheuristic and does not match the standard PyTorch initialization for all environments.

Original: Therefore itmight happen that a randomly initialized generator produces policies that are difficult to perturb.

Original: Wefound some difficulties in training policies for the InvertedDoublePendulum due to this explorationissue.

Original: This highlights a possible limitation of our method.

Original: [Figure]

Original: t u r n

Original: [Figure]

Original: 5.2 Analyzing the generator’s learning process
Revised: 5.2. Analyzing the generator’s learning process

Original: The probing actions from the fingerprinting mechanism of the value function V w can be seen asa compact meaningful policy embedding which can be used to visualize policies for a specificenvironment.
Revised: The probing actions created by the fingerprinting mechanism of the value function V w can be seen as a compact meaningful policy embedding useful to visualize policies for a specific environment.

Original: We make use of this in Figure 5. It shows the return of all policies in the buffer aftertraining, as well asthe returns of policies created by the generator at different stages of training whengiven the same range of return commands. 
Revised: In Figure 3 we apply PCA to probing actions to show all policies in the buffer after training, as well as policies created by the generator at different stages of training when given the same range of return commands. Policies are colored in line with achieved return.

Original: The generator’s objective can be seen as finding a trajectory through policy space, defined by thereturn commands, connecting the lowest with the highest return.
Revised: The generator’s objective can be seen as finding a trajectory through policy space, defined by the return commands, connecting the lowest with the highest return.

Original: In Figure 5, this corresponds to atrajectory going from a dark to a bright area.
Revised: In Figure 3, this corresponds to a trajectory going from a dark to a bright area.

Original: Indeed, we observe that the generator starts out beingconfined to the dark region (producing only bad policies) and over the course of training finds atrajectory leading from the darkest (low return) to the brightest (high return) regions.
Revised: Indeed, we observe that the generator starts out being confined to the dark region (producing only bad policies) and over the course of training finds a trajectory leading from the darkest (low return) to the brightest (high return) regions.

Original: Figure 6 shows the the returns achieved by policies that are created by a fully trained generator whengiven a range of return commands.
Revised: Figure 4 shows the the returns achieved by policies that are created by a fully trained generator when given a range of return commands.

Original: This highlights a feature of the policy generator: while most RLalgorithms generate only the best-performing policy, our generator is in principle able to produceby command policies across the whole performance spectrum.
Revised: This highlights a feature of the policy generator: while most RL algorithms generate only the best-performing policy, our generator is in principle able to produce by command policies across the whole performance spectrum.

Original: For the environments Swimmer and
Revised: For the environments Swimmer and

Original: Hopper (Figures 6a and 6b), this works in a relatively reliable fashion.
Revised: Hopper, this works in a relatively reliable fashion.

Original: In Hopper the return useddoes not include survival bonus.
Revised: In Hopper the return used does not include survival bonus.

Original: A return of 2000 without survival bonus corresponds roughly to areturn of 3000 with survival bonus.
Revised: A return of 2000 without survival bonus corresponds roughly to a return of 3000 with survival bonus.

Original: It is worth noting, however, that in some environments it is hard or even impossible to achieve anygiven intermediate return.
Revised: It is worth noting, however, that in some environments it is hard or even impossible to achieve every given intermediate return.

Original: This might be the case, for example, if the optimal policy is much simplerthan a slightly sub-optimal one, or if a large reward is given once a goal state is reached.
Revised: This might be the case, for example, if the optimal policy is much simpler than a slightly sub-optimal one, or if a large reward is given once a goal state is reached.

Original: We canobserve this effect for the environments InvertedPendulum and MountainCar—see Figures 6c and 6d.
Revised: We can observe this effect for the environments InvertedPendulum and MountainCar.

Original: There the generator struggles to produce the desired identityline of return command and achievedreturn—instead we get something closer to a step function.
Revised: There the generator struggles to produce the desired identity of return command and achieved return—instead we get something closer to a step function.

Original: However, this does not prevent ourmethod from quickly finding optimal policies in these environments.
Revised: However, this does not prevent our method from quickly finding optimal policies in these environments.

Original: More details in Appendix B.3.
Revised: More details in Appendix C.3.

Original: Related Work 
Revised: [Figure] 6. Conclusion and Future Work Our PoliGen is an RL framework for generating policies yielding given desired returns. Hypernetworks in conjunction with fingerprinting-based value functions can be used to train a Fast Weight Programmer through supervised learning to directly generate parameters of a policy that achieves a given return.

Revised: By iteratively asking for higher returns than those observed so far, our algorithm trains the generator to produce highly performant policies from scratch.

Revised: Empirically, PoliGen is competitive with ARS and DDPG on continuous control tasks

Revised: [Figure]

Revised: and able to generate policies with any desired return.

Revised: Future work will consider context commands other than those asking for particular returns, as well as generators based on latent variable models (e.g., conditional variational autoencoders) allowing for capturing diverse sets of policies, to improve exploration of complex

Revised: RL environments.

Revised: Acknowledgements

Revised: We thank Kazuki Irie, Mirek Strupl, Dylan Ashley, Róbert

Revised: Csordás, Aleksandar Stani´c and Anand Gopalakrishnan for their feedback.

Revised: This work was supported by the ERC Advanced Grant (no: 742870) and by the Swiss National Supercomputing Centre (CSCS, projects: s1090, s1154).

Revised: We also thank NVIDIA Corporation for donating a DGX-1 as part of the Pioneers of AI Research Award and to IBM for donating a Minsky machine.

Original: Policy conditioned value functions Compared to standard value functions that are conditioned on aspecific policy, policy conditioned value functions can generate values across several policies (Faccioet al., 2020;
Revised: Policy conditioned value functions Compared to standard value functions that are conditioned on a specific policy, policy conditioned value functions can generate values across several policies (Faccio et al., 2020;

Original: Harb et al., 2020).
Revised: Harb et al., 2020).

Original: This has been used to directly maximize the value using gradientascent in the policy parameters.
Revised: This has been used to directly maximize the value using gradient ascent in the policy parameters.

Original: In the context of the present work, this can be used to evaluate anypolicy generated by our policy generator.
Revised: In the context of the present work, this can be used to evaluate any policy generated by our policy generator.

Original: In contrast to previous work, this allows for generatingpolicies of an arbitrary quality in a zero-shot manner, without any gradient-based iterative trainingprocedure.
Revised: In contrast to previous work, this allows for generating policies of an arbitrary quality in a zero-shot manner, without any gradient-based iterative training procedure.

Original: Hindsight and Upside Down RL Upside Down RL (UDRL) transforms the RL problem into asupervised learning problem by conditioning the policy on commands such as the instruction toachieve a desired return (Schmidhuber, 2019; Srivastava et al., 2019).
Revised: Hindsight and Upside Down RL Upside Down RL (UDRL) transforms the RL problem into a supervised learning problem by conditioning the policy on commands such as the instruction to achieve a desired return (Schmidhuber, 2019; Srivastava et al., 2019).

Original: The required dataset of states,actions, and rewards can be collected online with iterative improvements to the policy (Srivastavaet al., 2019), or offline (Janner et al., 2021;
Revised: The required dataset of states, actions, and rewards can be collected online with iterative improvements to the policy (Srivastava et al., 2019), or offline (Janner et al., 2021;

Original: Chen et al., 2021).
Revised: Chen et al., 2021).

Original: UDRL methods are related to hindsight RL when the instructed commands correspond to desired goals in the environment (Schmidhuber,1991; Kaelbling, 1993; Andrychowicz et al., 2017; Rauber et al., 2018).
Revised: UDRL methods are related to hindsight RL when the instructed commands correspond to desired goals in the environment (Schmidhuber, 1991; Kaelbling, 1993; Andrychowicz et al., 2017; Rauber et al., 2018).

Original: Instead of optimizing thepolicy to achieve a desired reward in action space, our method PoliGen evaluates the generated[Figure] policies in command space.
Revised: Instead of optimizing the policy to achieve a desired reward in action space, our method PoliGen evaluates the generated policies in command space.

Original: This is done by generating a command-conditioned policy that is thenevaluated using a parameter-based value function and trained to match the command to the evaluatedreturn.
Revised: This is done by generating a command-conditioned policy that is then evaluated using a parameter-based value function and trained to match the command to the evaluated return.

Original: Fast Weight Programmers and Hyper Networks
Revised: Fast Weight Programmers and Hyper Networks

Original: The idea of using a neural network (NN) togenerate weights for another NN dates back to Fast Weight Programmers (FWPs) (Schmidhuber,1992, 1993), later scaled up to deeper neural networks under the name of HyperNetworks (Ha et al.,2016).
Revised: The idea of using a neural network (NN) to generate weights for another NN dates back to Fast Weight Programmers (FWPs) (Schmidhuber, 1992; 1993), later scaled up to deeper neural networks under the name of HyperNetworks (Ha et al., 2016).

Original: While in traditional NNs the weight matrix remains fixed after training, FWPs make theseweights context dependent.
Revised: While in traditional NNs the weight matrix remains fixed after training, FWPs make these weights context dependent.

Original: More generally, FWPs can be viewed as a neural function that involvesmultiplicative interactions and parameter sharing (Kirsch and Schmidhuber, 2021).
Revised: More generally, FWPs can be viewed as a neural function that involves multiplicative interactions and parameter sharing (Kirsch & Schmidhuber, 2021).

Original: When updated inrecurrent fashion, FWPs can be used as memory mechanisms.
Revised: When updated in recurrent fashion, FWPs can be used as memory mechanisms.

Original: Linear transformers are a type of FWPwhere information is stored through outer products of keys and values (Schlag et al., 2021).
Revised: Linear transformers are a type of FWP where information is stored through outer products of keys and values (Schlag et al., 2021).

Original: FWPs areused in the context of memory-based meta learning (Schmidhuber, 1993; Miconi et al., 2018; Gregor,2020; Kirsch and Schmidhuber, 2021;
Revised: FWPs are used in the context of memory-based meta learning (Schmidhuber, 1993; Miconi et al., 2018; Gregor, 2020; Kirsch & Schmidhuber, 2021;

Original: Irie et al., 2021; Kirsch et al., 2022), predicting parameters forvarying architectures (Knyazev et al., 2021), and reinforcement learning (Gomez and Schmidhuber,2005; Najarro and Risi, 2020; Kirsch et al., 2022).
Revised: Irie et al., 2021; Kirsch et al., 2022), predicting parameters for varying architectures (Knyazev et al., 2021), and reinforcement learning (Gomez & Schmidhuber, 2005; Najarro & Risi, 2020; Kirsch et al., 2022).

Original: In contrast to all of these approaches, ours uses FWPs to generate policies conditioned on a command (target return).
Revised: In contrast to all of these approaches, ours uses FWPs to generate policies conditioned on a command (target return).

Revised: B. Implementation details B.1. HyperNetwork

Revised: The idea behind certain feed-forward FWPs called hypernetworks (Ha et al., 2016)

Revised: is to split the parameters of the generated network θ into smaller slices s l .

Original: Conclusion and Future Work
Revised: A shared NN H with parameters ξ receives as input a learned embedding z l and outputs the slice s l , i.e. s l = H ξ ( z l ) for each l . Following (von Oswald et al., 2020), further context information can be given to H in form of an additional conditioning input c , which can be either either scalar or vector-valued: s l = H ξ ( z l , c ) . Then the weights are combined by concatenating all generated slices: [Equation]

Original: This work introduced PoliGen, a framework for generating deep RL policies that achieve a givendesired return.
Revised: The splitting of θ into slices and the choice of H depend on the specific architecture of the generated policy. Here we are interested in generating MLP policies whose parameters θ consist of weight matrices K j with j ∈ { 1 , 2 , . . . , n K } , where n K is the policy’s number of layers. We use an MLP H ξ to generate each slice of each weight matrix: the hypernetwork generator G ρ splits each weight matrix into slices s jmn ∈ R f × f , where j is the policy layer, and m, n are indexes of the slice in weight matrix of layer l .

Revised: For each of these slices, a small embedding vector z jmn ∈ R d is learned.

Original: We showed that Hyper Networks in conjunction with fingerprinting-based valuefunctions can be used to train a Fast Weight Programmer using supervised learning techniques thatdirectly generates policy parameters with the desired return.
Revised: Our network H ξ is an MLP, followed by a reshaping operation that turns a vector of size f 2 into an f × f matrix: [Equation] [Equation] The slices are then concatenated over two dimensions to obtain the full weight matrices:

Original: By iteratively asking for higher returnsthan those that have been observed so far, our algorithm trains the generator to produce highlyperformant policies from scratch.
Revised: [Equation] [Equation] The full hypernetwork generator G ρ consists of the shared network H ξ , as well as all embeddings z jmn . Its learnable parameters are ρ = { ξ, z

Original: We empirically demonstrate PoliGen’s competitiveness to ARSand DDPG on continuous control tasks, and its ability to  generate  policies with any desired return.
Revised: [Equation] [Figure] Generator G ρ is supposed to dynamically generate policy parameters, conditioned on the total return these policies should achieve.

Original: A natural extension of this work will be the consideration of other context commands besides thedesired return.
Revised: The conditioning input c is simply this scalar return command. It is appended to each learned slice embedding z jmn . The resulting vectors are the inputs to the network H . Figure 5 shows a diagram of this process.

Revised: For the the slicing to work, the widths and heights of the weight matrices have to be multiples of f .

Original: Furthermore, latent variable models (e.g., conditional variational autoencoders) wouldallow the generator to capture a more diverse set of policies, which might benefit effective explorationin complex RL environments.
Revised: For the hidden layers of an MLP, this is easily achieved since we can freely choose the numbers of neurons. For the input and output layers, however, we are constrained by the dimensions of environmental observations and actions. To accommodate any number of input and output neurons, we use dedicated networks H i and H o for the input and output layers. The generated slices have the shape f × n i for the input layer ( n i is the number of input neurons) and n o × f for the output layer ( n o is the number of output neurons).

Revised: B.2. Policy Fingerprinting

Revised: We use a policy fingerprinting mechanism (Harb et al., 2020) as an effective method to evaluate the performance of multiple NNs through a single function.

Revised: Policy fingerprinting works by giving a set of learnable probing states as input to the policy π θ .

Revised: The resulting outputs of the policy—called probing actions—are concatenated and given as input to an MLP U that computes the prediction V w ( θ ) .

Revised: Here the set of parameters w of this evaluator consists of the MLP parameters ϕ

Revised: and all the parameters of the probing states.

Revised: When training V w , the probing states learn to query the policy in meaningful situations, so that the policy’s success can be judged by its probing actions.

Revised: Fingerprinting is similar to a previous technique (Schmidhuber, 2015) where an NN learns to send queries (sequences of activation vectors) into another already trained NN, and learns to use the answers (sequences of activation vectors) to improve its own performance.

Original: A Implementation detailsA.1 Hyperparameters
Revised: Figure 1 shows a diagram of our method with a hypernetwork generator and a fingerprinting value function. The benefits of policy fingerprinting over directly observing policy weights become apparent as soon as we have at least one hidden layer in an MLP policy: the weights then have a large number of symmetries, i.e., many different weight configurations that are entirely equivalent in terms of the input-output mapping of the network. The main symmetries reflect possible permutations of hidden neurons and scalings of the weight matrices (K˚urková & Kainen, 1994). The probing actions of the fingerprinting mechanism are invariant with respect to such symmetries.

Revised: In fact, they are invariant even with respect to the general policy architecture.

Revised: This entails advantages not only for the value function V w , but also for the generator: the gradients w.r.t.

Revised: the generator’s weights ρ are obtained by backpropagating through V w .

Revised: If V w is fingerprinting-based, these gradients will point only in directions which, when followed, actually change the generated policy’s probing actions.

Revised: Consequently, the generator will ignore potential policy weight changes that have no effect on the policy’s probing actions (which are proxies for the policy’s general behavior in the environment).

Revised: B.3. Hyperparameters

Original: Here we report the hyperparameters used for PoliGen and the baselines.
Revised: Here we report the hyperparameters used for PoliGen and the baselines.

Original: For DDPG, we use thespinning-up RL implementation (Achiam, 2018), whose results are on par with the best reportedresults.
Revised: For DDPG, we use the spinning-up RL implementation (Achiam, 2018), whose results are on par with the best reported results.

Original: For ARS, we use the implementation of the authors (Mania et al., 2018), adapted to Deep NNpolicies.
Revised: For ARS, we use the implementation of the authors (Mania et al., 2018), adapted to Deep NN policies.

Original: Shared hyperparameters The table below shows hyperparametersthat are relevant to at least twoof the three methods.
Revised: Shared hyperparameters The table below shows hyperparameters relevant to at least two of the three methods.

Original: They stay fixed across environments.
Revised: They stay fixed across environments.

Original: [Table]
Revised: [Table]

Original: 100k
Revised: , 2 hidden layers, 256 neurons each, with bias [Table]

Original: [Table]
Revised: [Table]

Original: InvertedPendulum
Revised: see below

Original: [Table]
Revised: [Table]

Original: MountCarContinuous

Original: [Table]
Revised: [Table]

Revised: 100k for InvertedPendulum and MountCarContinuous, 3M for all other environ- ments

Original: Hyperparameters for specific algorithms Fixed across environments:
Revised: Hyperparameters for specific algorithms Fixed across environments:

Original: PoliGen:
Revised: PoliGen:

Original: • Architecture of the networks H in the generator: MLP with bias, two hidden layers of size 256, ReLU nonlinearity and no output activation function • Size of learnable HyperNetwork embeddings z jmn : 8• Size of slices s jmn produced by the HyperNetwork: 16 × • Number of probing  states: {+Uniformly random in [0 , 
+}
Revised: • Architecture of the networks H in the generator: MLP with bias, two hidden layers of size 256, ReLU nonlinearity, no output activation function • Size of learnable hypernetwork embeddings z jmn :• Size of slices s jmn produced by the hypernetwork: 16 ×• Number of probing states:• Initialization of probing states: {+Uniformly random in [0 , 
+}

Original:  •Initialization of probing states: Uniformly random in [0 , 1)• Priority sampling from replay buffer:
Revised: ) • Priority sampling from replay buffer:

Original: True, with weights 1 /x 1 . 1 , where x is the number ofepisodes since the data was stored in the buffer
Revised: True, with weights 1 /x 1 . 1 , where x is the number of episodes since the data was stored in the buffer

Original: DDPG:
Revised: DDPG:

Original: • Start-steps (random actions): 10000 time steps • Update after (no training): 1000 time steps • Polyak parameter: 0.
Revised: • Start-steps (random actions): 10000 time steps • Update after (no training): 1000 time steps • Polyak parameter: 0.

Original: Tuned hyperparameters For ARS, we tune the following hyperparameters for each environmentseparately using grid search:
Revised: Tuned hyperparameters For ARS, we tune the following hyperparameters for each environment separately using grid search:

Original: • Step size for ARS: tuned with values in { 1 e − 2 , 1 e − 3 , 1 e − 4 }• Number of directions and elite directions for ARS: tuned with values in{ [1 , 1] , [8 , 4] , [8 , 8] , [32 , 4] , [32 , 16] , [64 , 8] , [64 , 32] } , where the first element denotes thenumber of directions and the second element the number of elite directions• Noise for exploration in ARS: tuned with values in { 0 . 1 , 0 . 05 , 0 . 025 }
Revised: • Step size for ARS: tuned with values in { 1 e − 2 , 1 e − 3 , 1 e − 4 } • Number of directions and elite directions for ARS: tuned with values in { [1 , 1] , [8 , 4] , [8 , 8] , [32 , 4] , [32 , 16] , [64 , 8] , [64 , 32] } , where the first element denotes the number of directions and the second element the number of elite directions • Noise for exploration in ARS: tuned with values in { 0 . 1 , 0 . 05 , 0 . 025 }

Original: Here we report the best hyperparameters found for each environment:
Revised: Here we report the best hyperparameters found for each environment:

Original: [Table]
Revised: [Table]

Revised: Swimmer

Original: [Table]
Revised: [Table]

Revised: Inverted-Pendulum

Revised: MountainCar-Continuous

Revised: Step Size 0.

Revised: Number of

Revised: [Table]

Revised: , Number of Elite

Revised: [Table]

Original:  UDRL For UDRL we use a previous implementation (Srivastava et al., 2019) for discrete controlenvironments, and implemented additional classes to use it in continuous control tasks with episodicresets (although the original UDRL report (Schmidhuber, 2019) focused on continuous control insingle-life settings without resets).
Revised: Exploration [Table] Scale 0. UDRL For UDRL we use a previous implementation (Srivastava et al., 2019) for discrete control environments, and implemented additional classes to use it in continuous control tasks with episodic resets (although the original UDRL report (Schmidhuber, 2019) focused on continuous control in single-life settings without resets).

Original: We use the previous hyperparameters (Srivastava et al. , 2019)and tune learning rate (in { 1 e − 3 , 1 e − 4 , 1 e − 5 } ), activation (ReLU, tanh), and their "last_few"parameter (1, 10, 100), which is used to select the command for exploration.
Revised: We use the previous hyperparameters (Srivastava et al., 2019) and tune learning rate (in { 1 e − 3 , 1 e − 4 , 1 e − 5 } ), activation (ReLU, tanh), and their “last_few” parameter (1, 10, 100), which is used to select the command for exploration.

Original: For Swimmer, we arenot able to reproduce the performance with the original reported hyperparameters.
Revised: For Swimmer, we are not able to reproduce the performance with the original reported hyperparameters.

Original: Like for the otheralgorithms, we use a NN with 2 hidden layers and 256 neurons per layer.
Revised: Like for the other algorithms, we use an NN with 2 hidden layers and 256 neurons per layer.

Original: Below we report the besthyperparameters found for UDRL.
Revised: Below we report the best hyperparameters found for UDRL.

Original: [Table]
Revised: [Table]

Original: A.2 Generator implementation details
Revised: B.4. Generator implementation details

Original: Generating bias vectors Here we describe how to generate the bias vectors of the policies, whichis not explicitly mentioned in section 4.1.
Revised: Generating bias vectors Here we describe how to generate the bias vectors of the policies, which is not explicitly mentioned in section B.1.

Original: Analogously to Equations 5 and 6, the embeddings z jmnare fed to a dedicated bias-generating network H χ that produces slices of the shape f × 1 , andthose slices are concatenated.
Revised: Analogously to Equations 5 and 6, the embeddings z jmn are fed to a dedicated bias-generating network H χ that produces slices of the shape f × 1 , and those slices are concatenated.

Original: Since we have a two-dimensional grid of learned embeddings z (see Figure 2), we take the mean across the input dimensions of the concatenated slices so that we end upwith a bias vector (and not a matrix).
Revised: Since we have a two-dimensional grid of learned embeddings z (see Figure 5), we take the mean across the input dimensions of the concatenated slices so that we end up with a bias vector (and not a matrix).

Original: A.3 GPU usage / compute
Revised: B.5. GPU usage / compute

Original: We use cloud computing resources for our experiments.
Revised: We use cloud computing resources for our experiments.

Original: Our nodes have an Intel Xeon 12 core CPU and an NVIDIA Tesla P100 GPU with 16GB of memory.
Revised: Our nodes have an Intel Xeon 12 core CPU and an NVIDIA Tesla P100 GPU with 16GB of memory.

Original: We were able to run four PoliGenexperiments on one node in parallel.
Revised: We were able to run four PoliGen experiments on one node in parallel.

Original: Our estimate of computation time for the main results is 40 nodehours.
Revised: Our estimate of computation time for the main results is 40 node hours.

Original: B Experimental details 
Revised: C. Experimental details C.1. Additional experimental results

Original: [Figure] B.1 Additional experimental results

Original: [Figure] In Figure 7 we provide additional results to compare our method to UDRL with episodic resets.
Revised: In Figure 6 we provide additional results to compare our method to UDRL with episodic resets.

Original: Weconfirm that UDRL is not sample efficient for continuous control in environments with episodic resets (Schmidhuber, 2019), in line with previous experimental results.
Revised: We confirm that UDRL is not sample efficient for continuous control in environments with episodic resets (Schmidhuber, 2019), in line with previous experimental results.

Original: We argue that the multimodalityissue discussed in the introduction is the main issue with UDRL.
Revised: We argue that the multimodality issue discussed in the introduction is the main issue with UDRL.

Original: Note that UDRL is also known todiverge in stochastic environments with episodic resets (Štrupl et al., 2022).

Original: B.2 Main experiments on MuJoCo
Revised: C.2. Main experiments on MuJoCo

Original: For ARS and UDRL, the best hyperparameters for each environment are determined by running thealgorithm with each hyperparameter configuration across 5 random seeds.
Revised: For ARS and UDRL, the best hyperparameters for each environment are determined by running the algorithm with each hyperparameter configuration across 5 random seeds.

Original: The best configurationsare those reported in section A.1{+B.
+} We use them for the final 20 evaluation runs shown in our mainresults.
Revised: The best configurations are those reported in section {+B.
+} We use them for the final 20 evaluation runs shown in our main results.

Original: For DDPG and PoliGen, we use the same hyperparameters for all environments during the 20evaluation runs.
Revised: For DDPG and PoliGen, we use the same hyperparameters for all environments.

Original: For the results shown in Figures 3 and 7, we evaluate each run every 10000 time steps (in the caseof Swimmer and Hopper, every 1000  for InvertedPendulum and MountainCarContinuous) for 10episodes.
Revised: For 10 episodes, Figures 2 and 6 evaluate each run every 10000 time steps for Swimmer and Hopper, every 1000 steps for InvertedPendulum and MountainCarContinuous.

Original: Table 1 shows the final return and standard deviation of each algorithm.
Revised: Table 1 shows the final return and standard deviation of each algorithm.

Original: [Table]
Revised: [Table]

Revised: Obtaining suitable policies from the start Randomly initialized policy generators produce weights far from those of typical initialization schemes.

Revised: In particular, the standard PyTorch (Paszke et al., 2019) initialization is uniform in [ − 1 / √ n, 1 / √ n ] , where n is the number of neurons in the previous layer, resulting in a distribution uniform in [ − 0 .

Revised: in the second and last layers.

Revised: Our network tends to generate much larger weights, roughly uniform in every NN layer.

Revised: We therefore scale our output such that it is close to the default initialization.

Revised: Concretely, we multiply for each layer the output of the generator by 2 / √ n , where n is the number of neurons in the previous layer.

Revised: Here we provide an ablation showing that this choice is crucial.

Revised: Figure 7 shows the importance of scaling the output of the generator in

Revised: Swimmer and Hopper.

Original: B
Revised: We compare this with and without weighted sampling from the replay buffer. We observe that in Swimmer, output scaling is very important, while in Hopper, most of the performance gain is due to weighted sampling. This choice of output scaling is rather heuristic and does not match the standard PyTorch initialization for all environments.

Revised: It might happen that a randomly initialized generator produces policies that are difficult to perturb.

Revised: This exploration issue seems to cause some difficulties for InvertedDoublePendulum, highlighting a possible limitation of our method.

Revised: [Figure]

Original: 3 Details on generated policy visualization
Revised: C.3. Details on generated policy visualization

Original: To create Figure 5, we perform Principal Component Analysis (PCA) on the probing actions of allpolicies in the buffer after training.
Revised: To create Figure 3, we perform Principal Component Analysis (PCA) on the probing actions of all policies in the buffer after training.

Original: The first two principal components indicate the positionof apolicy in our visualization.
Revised: The first two principal components indicate a policy’s position in our visualization.

Original: Using Delaunay triangulation, we assign an area to every policy andcolor it according to its achieved return.
Revised: Using Delaunay triangulation, we assign an area to every policy and color it according to its achieved return.

Original: We then take the generator at different stages of training (of the same run).
Revised: We then take the generator at different stages of training (of the same run).

Original: Each of these generators is given a set of 20 commands, evenly spaced acrossthe range of possible returns ( [ − 100 , 365] for Swimmer, [ − 100 , 3000] for Hopper, [0 , 1000] for 
Revised: Each of these generators is given a set of 20 commands, evenly spaced across the range of possible returns ( [ − 100 , 365] for Swimmer, [ − 100 , 3000] for Hopper, [0 , 1000] for InvertedPendulum and [ − 100 , 100] for

Original: InvertedPendulum and [ − 100 , 100] for MountainCarContinuous).
Revised: MountainCarContinuous).

Original: The resulting policies are plottedusing probing actions on the probing states of the fully trained value function V w and the same PCA.
Revised: The resulting policies are plotted using probing actions on the probing states of the fully trained value function V w (and the same PCA).

Original: B.4 Command strategies
Revised: C.4. Command strategies

Original: We tried an alternative approach using Importance Sampling (Hesterberg, 1988) estimators.
Revised: In early experiments, we tried an alternative approach using Importance Sampling (Hesterberg, 1988) estimators.

Original: Given amixture of weights β i ( θ ) , we considered estimators of the form ˆ J ( c ′ , w ′ )
Revised: Given a mixture of weights β i ( θ ) , we considered estimators of the form ˆ J ( c ′ , w ′ )

Original: =
Revised: =

Original: (cid:80) Ni
Revised: (cid:80) Ni

Original: =1 β
Revised: =1 β

Original: i ( θ i ) p ( θ i | c ′ ; w ′ )
Revised: i ( θ i ) p ( θ i | c ′ ; w ′ )

Original: p ( θ i
Revised: p ( θ i

Original: | c i ; w i ) r i ,which provides an unbiased estimate of the performance of a policy produced by a generator withparameters w ′ and command c ′ , using past data derived from old generators with different commands.
Revised: | c i ; w i ) r i , which provides an unbiased estimate of the performance of a policy produced by a generator with parameters w ′ and command c ′ , using past data derived from old generators with different commands.

Original: Maximizing ˆ J ( c ′ , w ′ ) with respect to the command c ′ should yield commands encouraging thegenerator to produce highly performant policies.
Revised: Maximizing ˆ J ( c ′ , w ′ ) with respect to the command c ′ should yield commands encouraging the generator to produce highly performant policies.

Original: We tested this using the BalanceHeuristic (Veach and
Revised: We tested this using the Balance

Original:  Guibas, 1995) estimator for β k , which is known to have small variance.
Revised: Heuristic (Veach & Guibas, 1995) estimator for β k , which is known to have small variance (Papini et al., 2019).

Original: However, in our experimentswe observed that generators using such command strategies did not significantly outperform thesimple strategy mentioned earlier.
Revised: However, in our experiments we observed that generators using such command strategies did not significantly outperform the simple strategy mentioned earlier.

Original: Ablation command Figure 8 shows that when choosing the command for exploration there is aslight advantage in asking to generate a policy whose return exceeds the best return so far by 20.
Revised: Ablation command Figure 8 shows that when choosing the command for exploration there is a slight advantage for asking the generator for a policy whose return exceeds the best return so far by 20.

Original: However, just asking for the maximum return (drive parameter = 0) is also competitive.
Revised: However, just asking for the maximum return (drive parameter = 0) is also competitive.

Original: [Figure]
Revised: [Figure]

Original: C Environment details
Revised: D. Environment details

Original: [Figure] MuJoCo (Todorov et al., 2012) is licensed under Apache 2.0.
Revised: MuJoCo (Todorov et al., 2012) is licensed under Apache 2.0.