P ARAMETER - BASED V ALUE F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Traditional off-policy actor-critic Reinforcement Learning (RL) algorithms learn value functions of a single target policy.
However, when value functions are updated to track the learned policy, they forget potentially useful information about old policies.
We introduce a class of value functions called Parameter-based Value
Functions (PVFs) whose inputs include the policy parameters.
They can generalize across different policies.
PVFs can evaluate the performance of any policy given a state, a state-action pair, or a distribution over the RL agent’s initial states.
First we show how PVFs yield novel off-policy policy gradient theorems.
Then we derive off-policy actor-critic algorithms based on PVFs trained by Monte Carlo or Temporal Difference methods.
We show how learned PVFs can zero-shot learn new policies that outperform any policy seen during training.
Finally our algorithms are evaluated on a selection of discrete and continuous control tasks using shallow policies and deep neural networks.
Their performance is comparable to state-of-the-art methods.
I NTRODUCTION
Value functions are central to Reinforcement Learning (RL).
For a given policy, they estimate the value of being in a speciﬁc state (or of choosing a particular action in a given state).
Many RL breakthroughs were achieved through improved estimates of such values, which can be used to ﬁnd optimal policies (Tesauro, 1995; Mnih et al., 2015).
However, learning value functions of arbitrary policies without observing their behavior in the environment is not trivial.
Such off-policy learning requires to correct the mismatch between the distribution of updates induced by the behavioral policy and the one we want to learn.
Common techniques include Importance Sampling (IS) (Hesterberg, 1988) and deterministic policy gradient methods (DPG) (Silver et al., 2014), which adopt the actorcritic architecture (Sutton, 1984; Konda & Tsitsiklis, 2001; Peters & Schaal, 2008).
Unfortunately, these approaches have limitations.
IS suffers from large variance (Cortes et al., 2010;
Metelli et al., 2018; Wang et al., 2016) while traditional off-policy actor-critic methods introduce off-policy objectives whose gradients are difﬁcult to follow since they involve the gradient of the action-value function with respect to the policy parameters ∇ θ Q π θ ( s, a ) (Degris et al., 2012; Silver et al., 2014). This term is usually ignored, resulting in biased gradients for the off-policy objective.
Furthermore, off-policy actor-critic algorithms learn value functions of a single target policy.
When value functions are updated to track the learned policy, the information about old policies is lost.
We address the problem of generalization across many value functions in the off-policy setting by introducing a class of parameter-based value functions (PVFs) deﬁned for any policy.
PVFs are value functions whose inputs include the policy parameters, the PSSVF V ( θ ) , PSVF V ( s, θ ) , and
PAVF Q ( s, a, θ ) .
PVFs can be learned using Monte Carlo (MC) (Metropolis & Ulam, 1949) or
Temporal Difference (TD) (Sutton, 1988) methods.
The PAVF Q ( s, a, θ ) leads to a novel stochastic and deterministic off-policy policy gradient theorem and, unlike previous approaches, can directly compute ∇ θ Q π θ ( s, a ) .
Based on these results, we develop off-policy actor-critic methods and compare our algorithms to two strong baselines, ARS and DDPG (Mania et al., 2018;
Lillicrap et al., 2015), outperforming them in some environments.
We make theoretical, algorithmic, and experimental contributions: Section 2 introduces the standard MDP setting; Section 3 formally presents PVFs and derive algorithms for V ( θ ) , V ( s, θ ) and
Q ( s, a, θ ) ; Section 4 describes the experimental evaluation using shallow and deep policies; Sections 5 and 6 discuss related and future work.
Proofs and derivations can be found in Appendix A.2.
B ACKGROUND
We consider a Markov Decision Process (MDP) (Stratonovich, 1960; Puterman, 2014)
M = ( S , A , P, R, γ, µ 0 ) where at each step an agent observes a state s ∈ S , chooses action a ∈ A , transitions into state s (cid:48) with probability P ( s (cid:48) | s, a ) and receives a reward R ( s, a ) .
The agent starts from an initial state, chosen with probability µ 0 ( s ) .
It is represented by a parametrized stochastic policy π θ : S → ∆( A ) , which provides the probability of performing action a in state s .
Θ is the space of policy parameters.
The policy is deterministic if for each state s there exists an action a such that π θ ( a | s )
= 1 .
The return R t is deﬁned as the cumulative discounted reward from time step t: R t = (cid:80) T − t − 1 k = γ k R ( s t + k +1 , a t + k +1 ) , where T denotes the time horizon and γ a realvalued discount factor.
The performance of the agent is measured by the cumulative discounted expected reward (expected return), deﬁned as J ( π θ ) =
E π θ [ R 0 ] .
Given a policy π θ , the state-value function V π θ ( s ) =
E π θ [ R t | s t = s ] is deﬁned as the expected return for being in a state s and following policy π θ .
By integrating over the state space S , we can express the maximization of the expected cumulative reward in terms of the state-value function J ( π θ ) =
(cid:82) S µ 0 ( s )
V π θ ( s ) d s .
The action-value function Q π θ ( s, a ) , which is deﬁned as the expected return for performing action a in state s , and following the policy π θ , is Q π θ ( s, a ) =
E π θ [ R t | s t = s, a t = a ] , and it is related to the state-value function by V π θ ( s ) = [Equation]
A π θ ( a | s ) Q [Equation] θ
[Equation]
s, a ) d a .
We deﬁne as d π θ ( s (cid:48) )
the discounted weighting of states encountered starting at s 0 ∼ µ 0 ( s ) and following the policy π θ : d π
[Equation]
=
[Equation]
S
[Equation]
∞
[Equation]
s
[Equation]
P [Equation] s → s (cid:48) , t, π θ ) d s , where P ( s → s (cid:48) , t, π θ ) is the probability of transitioning to s (cid:48) after t time steps, starting from s and following policy π θ .
Sutton et al.
(1999) showed that, for stochastic policies, the gradient of J ( π θ ) does not involve the derivative of d π θ ( s ) and can be expressed in a simple form:
[Equation]
Similarly, for deterministic policies Silver et al.
(2014) obtained the following:
[Equation]
Off-policy RL In off-policy policy optimization, we seek to ﬁnd the parameters of the policy maximizing a performance index J b ( π θ ) using data collected from a behavioral policy π b .
Here the objective function J b ( π θ ) is typically modiﬁed to be the value function of the target policy, integrated over d π b ∞ ( s ) =
lim t →∞ P ( s t =
s | s 0 , π b ) , the limiting distribution of states under π b (assuming it exists) (Degris et al., 2012; Imani et al., 2018;
Wang et al., 2016).
Throughout the paper we assume that the support of d π b ∞ includes the support of µ 0 so that the optimal solution for J b is also optimal for J .
Formally, we want to ﬁnd:
[Equation]
Unfortunately, in the off-policy setting
[Equation]
the states are obtained from d π b ∞ and not from d π θ ∞ , hence the gradients suffer from a distribution shift (Liu et al., 2019; Nachum et al., 2019).
Moreover, since we have no access to d π θ ∞ , a term in the policy gradient theorem corresponding to the gradient of the action value function with respect to the policy parameters needs to be estimated.
This term is usually ignored in traditional off-policy policy gradient theorems 1 .
In particular, when the policy is stochastic, Degris et al.
(2012) showed that:
[Equation]
Analogously [Equation] Silver et al.
(2014) provided the following approximation for deterministic policies 2 : [Equation]
[Equation]
Although the term ∇ θ Q π θ ( s, a ) is dropped, there might be advantages in using the approximate gradient of J b in order to ﬁnd the maximum of the original RL objective J .
Indeed, if we were on-policy, the approximated off-policy policy gradients by Degris et al.
(2012); Silver et al.
(2014) would revert to the on-policy policy gradients, while an exact gradient for J b would necessarily introduce a bias.
However, when we are off-policy, it is not clear whether this would be better than using the exact gradient of J b in order to maximize J .
In this work, we assume that J b can be considered a good objective for off-policy RL and we derive an exact gradient for it.
P ARAMETER - BASED V ALUE F UNCTIONS
In this section, we introduce our parameter-based value functions, the PSSVF V ( θ ) , PSVF V ( s, θ ) , and PAVF Q ( s, a, θ ) and their corresponding learning algorithms.
First, we augment the state and action-value functions, allowing them to receive as an input also the weights of a parametric policy.
The parameter-based state-value function (PSVF) V ( s, θ ) =
E [ R t | s t =
s, θ ] is deﬁned as the expected return for being in state s and following policy parameterized by θ .
Similarly, the parameter-based action-value function (PAVF) Q ( s, a, θ ) =
E [ R t | s t = s, a t = a, θ ] is deﬁned as the expected return for being in state s, taking action a and following policy parameterized by θ .
Using PVFs, the RL objective becomes: J ( π θ )
=
(cid:82) S µ 0 ( s ) V π
( s, θ ) d s .
Maximizing this objective leads to on-policy policy gradient theorems that are analogous to the traditional ones (Sutton et al., 1999; Silver et al., 2014):
Theorem 3.1.
Let π θ be stochastic.
For any Markov Decision Process, the following holds:
[Equation]
Theorem 3.2.
Let π θ be deterministic.
Under standard regularity assumptions (Silver et al., 2014), for any Markov Decision Process, the following holds:
[Equation]
Parameter-based value functions allow us also to learn a function of the policy parameters that directly approximates J ( π θ ) .
In particular, the parameter-based start-state-value function (PSSVF) is deﬁned as:
[Equation]
Off-policy RL In the off-policy setting, the objective to be maximized becomes:
[Equation]
By taking the gradient of the performance J b with respect to the policy parameters θ we obtain novel policy gradient theorems.
Since θ is continuous, we need to use function approximators V w ( θ ) ≈
V ( θ ) , V w ( s, θ )
≈
V ( s, θ ) and Q w ( s, a, θ )
≈ Q ( s, a, θ ) .
Compatible function approximations can be derived to ensure that the approximated value function is following the true gradient.
Like in previous approaches, this would result in linearity conditions.
However, here we consider nonlinear function approximation and we leave the convergence analysis of linear PVFs as future work.
In episodic settings, we do not have access to d π b ∞ , so in the algorithm derivations and in the experiments we approximate it by sampling trajectories generated by the behavioral policy.
In all cases, the policy improvement step can be very expensive, due to the computation of the arg max over a continuous space Θ .
Actor-critic methods can be derived to solve this optimization problem, where the critic (PVFs) can be learned using TD or MC methods, while the actor is updated following the gradient with respect to the critic.
Although our algorithms on PSSVF and PSVF can be used with both stochastic and deterministic policies, removing the stochasticity of the action-selection process might facilitate learning the value function.
All our algorithms make use of a replay buffer.
3. P ARAMETER - BASED S TART -S TATE -V ALUE F UNCTION V ( θ )
We ﬁrst derive the PSSVF V ( θ ) .
Given the original performance index J , and taking the gradient with respect to θ , we obtain:
[Equation]
In Algorithm 1, the critic V w ( θ ) is learned using MC to estimate the value of any policy θ .
The actor is then updated following the direction of improvement suggested by the critic.
Since the main application of PSSVF is in episodic tasks 3 , we optimize for the undiscounted objective.
Algorithm for many steps do : steps do : end
3. P ARAMETER - BASED S TATE -V ALUE F UNCTION V ( s, θ )
Learning the value function using MC approaches can be difﬁcult due to the high variance of the estimate.
Furthermore, episode-based algorithms like Algorithm 1 are unable to credit good actions in bad episodes.
Gradient methods based on TD updates provide a biased estimate of V ( s, θ ) with much lower variance and can credit actions at each time step.
Taking the gradient of J b ( π θ ) in the
PSVF formulation 4 , we obtain:
[Equation]
Algorithm 2 (Appendix) uses the actor-critic architecture, where the critic is learned via TD 5 .
3. P ARAMETER - BASED A CTION -V ALUE F UNCTION Q ( s, a, θ )
The introduction of the PAVF Q ( s, a, θ ) allows us to derive new policy gradients theorems when using a stochastic or deterministic policy.
Stochastic policy gradients We want to use data collected from some stochastic behavioral policy π b in order to learn the action-value of a target policy π θ .
Traditional off-policy actor-critic algorithms only approximate the gradient of J b , since they do not estimate the gradient of the actionvalue function with respect to the policy parameters ∇ θ Q π θ ( s, a ) (Degris et al., 2012; Silver et al., 2014).
With PVFs, we can directly compute this contribution to the gradient.
This yields an exact policy gradient theorem for J b :
Theorem 3.3.
For any Markov Decision Process, the following holds:
[Equation]
Algorithm 3 (Appendix) uses an actor-critic architecture and can be seen as an extension of OffPAC (Degris et al., 2012) to PAVF.
Deterministic policy gradients Estimating Q ( s, a, θ ) is in general a difﬁcult problem due to the stochasticity of the policy.
Deterministic policies of the form π : S
→ A can help improving the efﬁciency in learning value functions, since the expectation over the action space is no longer required.
Using PVFs, we can write the performance of a policy π θ as:
[Equation]
Taking the gradient with respect to θ we obtain a deterministic policy gradient theorem:
Theorem 3.4.
Under standard regularity assumptions (Silver et al., 2014), for any Markov Decision
Process, the following holds:
[Equation]
[Equation] Algorithm 4 (Appendix) uses an actor-critic architecture and can be seen as an extension of
DPG (Silver et al., 2014) to PAVF.
Despite the novel formulation of algorithm 3, we decided to avoid the stochasticity of the policy and to implement and analyze only the deterministic PAVF.
E XPERIMENTS
Applying algorithms 1, 2 and 4 directly can lead to convergence to local optima, due to the lack of exploration.
In practice, like in standard deterministic actor-critic algorithms, we use a noisy version of the current learned policy in order to act in the environment and collect data to encourage exploration.
More precisely, at each episode we use π ˜ θ with ˜ θ
= θ + (cid:15), (cid:15) ∼ N (0 , σ 2 I ) instead of π θ and then store ˜ θ in the replay buffer.
In our experiments, we report both for our methods as well as the baselines the performance of the policy without parameter noise.
Source code will be made available with the publication.
4. V ISUALIZING PVF S USING LQR S
We start with an illustrative example that allows us to visualize how PVFs are learning to estimate the expected return over the parameter space.
For this purpose, we use an instance of the 1D Linear
Quadratic Regulator (LQR) problem and a linear deterministic policy with bias.
In ﬁgure 1, we plot the episodic J ( θ ) , the cumulative return that an agent would obtain by acting in the environment using policy π θ for a single episode, and the cumulative return predicted by the PSSVF V ( θ ) for two different times during learning.
At the beginning of the learning process, the PSSVF is able to provide just a local estimation of the performance of the agent, since only few data have been observed.
However, after 1000 episodes, it is able to provide a more accurate global estimate over the parameter space.
Appendix A.4.1 contains a similar visualization for PSVF and PAVF, environment details and hyperparameters used.
4. M AIN RESULTS
Given the similarities between our PAVF and DPG, Deep Deterministic Policy Gradients (DDPG) is a natural choice for the baseline.
Additionally, the PSSVF V ( θ ) resembles evolutionary methods as the critic can be interpreted as a global ﬁtness function.
Therefore, we decided to include in the comparison Augmented Random Search (ARS) which is known for its state-of-the-art performance using only linear policies in continuous control tasks.
For the policy, we use a 2-layer MLP (64,64) with tanh activations and a linear policy followed by a tanh nonlinearity.
Figure 2 shows results for deterministic policies with both architectures.
In all the tasks the PSSVF is able to achieve at least the same performance compared to ARS, often outperforming it.
In the Inverted Pendulum environment, PSVF and PAVF with deep policy are very slow to converge, but they excel in the Swimmer task and MountainCarContinuous.
In Reacher, all PVFs fail to learn the task, while DDPG converges quickly to the optimal policy.
We conjecture that for this task it is difﬁcult to perform a search in parameter space.
On the other hand, in MountainCarContinuous, the reward is more sparse and DDPG only rarely observes positive reward when exploring in action space.
In Appendix A. we include additional results for PSSVF and PSVF with stochastic policies and hyperparameters.
We analyze the sensitivity of the algorithms on the choice of hyperparameters in Appendix A.4.4.
[Figure]
60 episodes
[Figure]
4. Z ERO - SHOT LEARNING
In order to test whether PVFs are generalizing across the policy space [Figure] we perform the following experiment with shallow deterministic policies: while learning using algorithm 1, we stop training and randomly initialize 5 policies.
Then, without interacting with the environment, we train these policies ofﬂine, in a zero-shot manner, following only the direction of improvement suggested by ∇ θ V w ( θ ) , whose weights w remain frozen.
We observe that shallow policies can be effectively trained from scratch.
Results for PSSVFs in Swimmer-v3 are displayed in ﬁgure 3.
In particular, we compare the performance of the policy learned, the best perturbed policy for exploration seen during training and ﬁve policies learned from scratch at three different stages in training.
We note that after the PSSVF has been trained for 100,000 time steps interactions with the environment (ﬁrst snapshot), these policies are already able to outperform both the current policy and any policy seen while training the PSSVF.
They achieve an average return of 297, while the best observed return was 225.
We include additional results for PSVF and PAVF in different environments, using shallow and deep policies in Appendix A.4.2.
When using deep policies, we obtain similar results only for the simplest environments.
For this task, we use the same hyperparameters as in ﬁgure 2.
4. O FFLINE LEARNING WITH FRAGMENTED BEHAVIORS
In our last experiment, we investigate how PSVFs are able to learn in a completely ofﬂine setting.
The goal is to learn a good policy in Swimmer-v3 given a ﬁxed dataset containing 100,000 transitions, without additional environment interactions.
Furthermore, the policy generating the data is perturbed every 200 time steps, for a total of 5 policies per episode.
Observing only incomplete trajectories for each policy parameter makes TD bootstrapping harder:
In order to learn, the PSVF needs to generalize across both the state and the parameter space.
Given the ﬁxed dataset, we ﬁrst train the PSVF, minimizing the TD error.
Then, at different stages during learning, we trainnew shallow deterministic policies.
Figure 4 describes this process.
We note that at the beginning of training, when the PSVF V ( s, θ ) has a larger TD error, these policies have poor performance.
However, after 7000 gradient updates, they are able to achieve a reward of 237, before eventually degrading to 167.
They outperform the best policy in the dataset used to train the PSVF, whose return is only of 58.
[Figure]
R ELATED WORK
There are two main classes of similar algorithms performing search in policy parameter space.
Evolutionary algorithms (Wierstra et al., 2014; Salimans et al., 2017; Mania et al., 2018) iteratively estimate a ﬁtness function evaluating the performance of a population of policies and then perform gradient ascent in parameter space, often estimating the gradient using ﬁnite difference approximation.
By replacing the performance of a population through a likelihood estimation, evolutionary algorithms become a form of Parameter Exploring Policy Gradients (Sehnke et al., 2008; 2010).
Our methods are similar to evolution since our value function can be seen as a ﬁtness.
Unlike evolution, however, our approach allows for obtaining the ﬁtness gradient directly and is more suitable for reusing past data.
While direct V ( θ ) optimization is strongly related to evolution, our more informed algorithms optimize V ( s, θ ) and Q ( s, a, θ ) .
That is, ours both perform a search in policy parameter space AND train the value function and the policy online, without having to wait for the ends of trials or episodes.
The second related class of methods involves surrogate functions (Box & Wilson, 1951;
Booker et al., 1998; Moore & Schneider, 1996).
They often use local optimizers for generalizing across ﬁtness functions.
In particular, Bayesian Optimization (BO) (Snoek et al., 2012; 2015) uses a surrogate function to evaluate the performance of a model over a set of hyperparameters and follows the uncertainty on the surrogate to query the new data to sample.
Unlike BO, we do not build a probabilistic model and we use the gradient of the value function instead of a sample from the posterior to decide which policy parameters to use next in the policy improvement step.
The possibility of augmenting the value functions with auxiliary parameters was already
[Figure] [Figure]
considered in work on General Value Functions
(Sutton et al., 2011), where the return is deﬁned with respect to an arbitrary reward function.
Universal Value Function Approximators (Schaul et al., 2015) extended this approach to learn a single value function V π θ ( s, g ) , representing the value, given possible agent goals g .
In particular, they learn different embeddings for states and goals, exploiting their common structure, and they show generalization to new unseen goals.
Similarly, our PSVF V ( s, θ ) is able to generalize to unseen policies, observing data for only a few ( s, θ ) pairs.
General and Universal Value Functions have not been applied to learn a single value function for every possible policy.
Policy Evaluation Networks (PENs) (Harb et al., 2020) are closely related to our work and share the same motivation.
PENs focus on the simplest PSSVF V ( θ ) trained without an actor-critic architecture.
Like in some of our experiments, the authors show how following the direction of improvement suggested by V ( θ ) leads to an increase in policy performance.
They also suggest to explore in future work a more complex setting where a PSVF V ( s, θ ) is learned using an actor-critic architecture.
Our work directly introduces the PSVF V ( s, θ ) and PAVF Q ( s, a, θ ) and presents novel policy gradient theorems for PAVFs when stochastic or deterministic policies are used.
There are many differences between our approach to learning V ( θ ) and theirs.
For example, we do not use a ﬁngerprint mechanism (Harb et al., 2020) for embedding the weights of complex policies.
Instead, we simply parse all the policy weights as inputs to the value function, even in the nonlinear case.
Fingerprinting may be important for representing nonlinear policies without losing information about their structure and for saving memory required to store the weights. Harb et al.
(2020) focus on the ofﬂine setting.
They ﬁrst use randomly initialized policies to perform rollouts and collect reward from the environment.
Then, once V ( θ ) is trained using the data collected, many gradient ascent steps through V yield new, unseen, randomly initialized policies in a zero-shot manner, exhibiting improved performance. They train their value function using small nonlinear policies of one hidden layer and 30 neurons on Swimmer-v3. They evaluate 2000 deterministic policies on 500 episodes each (1 million policy evaluations), achieving a ﬁnal expected return of ≈ 180 on new policies trained from scratch through V. On the other hand, in our zero-shot learning experiment using a linear PSSVF, after only 100 policy evaluations, we obtain a return of 297. In our main experiments, we showed that a ﬁngerprint mechanism is not necessary for the tasks we analyzed: even when using a much bigger 2-layers MLP policy, we are able to outperform the results in PEN.
Although Harb et al.
(2020) use Swimmer-v3 “to scale up their experiments”, our results suggest that Swimmer-v3 does not conclusively demonstrate possible beneﬁts of their policy embedding.
Gradient Temporal Difference (Sutton et al., 2009a;b; Maei et al., 2009; 2010; Maei, 2011) and Emphatic Temporal Difference methods (Sutton et al., 2016) were developed to address convergence under on-policy and off-policy
(Precup et al., 2001) learning with function approximation.
The ﬁrst attempt to obtain a stable off-policy actor-critic algorithm under linear function approximation was called Off-PAC (Degris et al., 2012), where the critic is updated using GTD( λ ) (Maei, 2011) to estimate the state-value function.
This algorithm converges when using tabular policies.
However, in general, the actor does not follow the true gradient direction for
J b . A paper on DPG (Silver et al., 2014) extended the Off-PAC policy gradient theorem (Degris et al., 2012) to deterministic policies.
This was coupled with a deep neural network to solve continuous control tasks through Deep Deterministic Policy Gradients (Lillicrap et al., 2015).
Imani et al.
(2018) used emphatic weights to derive an exact off-policy policy gradient theorem for J b .
Differently from Off-PAC, they do not ignore the gradient of the action-value function with respect to the policy, which is incorporated in the emphatic weighting: a vector that needs to be estimated.
Our off-policy policy gradients provide an alternative approach that does not need emphatic weights.
The widely used off-policy objective function J b suffers the distribution shift problem.
Liu et al.
(2019) provided an off-policy policy gradient theorem which is unbiased for the true RL objective J ( π θ ) , introducing a term d π θ ∞ /d π b ∞ that corrects the mismatch between the states distributions.
Despite their sound off-policy formulation, estimating the state weighting ratio remains challenging.
All our algorithms are based on the off-policy actor-critic architecture.
The two algorithms based on Q ( s, a, θ ) can be viewed as analogous to Off-PAC and DPG where the critic is deﬁned for all policies and the actor is updated following the true gradient with respect to the critic.
L IMITATIONS AND FUTURE WORK
We introduced PVFs, a novel class of value functions which receive as input the parameters of a policy and can be used for off-policy learning.
We have shown that PVFs are competitive to ARS and DDPG (Mania et al., 2018;
Lillicrap et al., 2015) while generalizing across policies and allowing for zero-shot training in an ofﬂine setting.
Despite their positive results on shallow and deep policies,
PVFs suffer the curse of dimensionality when the number of policy parameters is high.
Embeddings similar to those used in PENs (Harb et al., 2020) may be useful not only for saving memory and computational time, but also for facilitating search in parameter space.
We intend to evaluate the beneﬁts of such embeddings and other dimensionality reduction techniques.
We derived off-policy policy gradient theorems, showing how PVFs follow the true gradient of the performance J b .
With these results, we plan to analyze the convergence of our algorithms using stochastic approximation techniques (Borkar, 2009) and test them on environments where traditional methods are known to diverge (Baird, 1995).
Finally, we want to investigate how PVFs applied to supervised learning tasks or POMDPs, can avoid BPTT by mapping the weights of an RNN to its loss.
A A PPENDIX I NDEX OF THE APPENDIX
In the following, we brieﬂy recap the contents of the appendix.
• Appendix A.1 contains additional related works • Appendix A.2 reports all proofs and derivations.
• Appendix A.3 illustrates implementation details and pseudocode.
• Appendix A.4 provides the hyperparameters used in the experiments and further results.
A. A DDITIONAL RELATED WORKS
Recent work (Unterthiner et al., 2020) shows how to map the weights of a trained Convolutional
Neural Network to its accuracy.
Experiments show how these predictions allow for performance rankings of neural networks on new unseen tasks.
These maps are either learned by taking the ﬂattened weights as input or using simple statistics.
However, these predictions do not guide the training process of CNNs.
In 1990, adaptive critics trained by TD were used to predict the gradients of an RNN from its activations (Schmidhuber, 1990), avoiding backpropagation through time (BPTT) (Werbos, 1990). This idea was later used to update the weights of a neural network asynchronously (Jaderberg et al., 2017).
In our work, the critic is predicting errors instead of gradients.
If applied to POMDPs, or supervised learning tasks involving long time lags between relevant events, the PSSVF could avoid BPTT by viewing the parameters of an RNN as a static object and mapping them to their loss (negative reward).
Additional differences between our work and Policy Evaluation Networks (PENs) (Harb et al., 2020) concern the optimization problem: we do not predict a bucket index for discretized reward, but perform a regression task.
Therefore our loss is simply the mean squared error between the prediction of V ( θ ) and the reward obtained by π θ , while their loss (Harb et al., 2020) is the KL divergence between the predicted and target distributions.
Both approaches optimize the undiscounted objective when learning V ( θ ) .
A. P ROOFS AND DERIVATIONS
Theorem 3.1.
Let π θ be stochastic.
For any Markov Decision Process, the following holds:
[Equation]
Proof.
The proof follows the standard approach by Sutton et al.
and we report it for completeness.
We start by deriving an expression for ∇ θ V ( s, θ ) :
[Equation]
Taking the expectation with respect to s 0 ∼ µ 0 ( s ) we have:
[Equation]
Theorem
[Equation]
Let π θ be deterministic.
Under standard regularity assumptions (Silver et al., 2014), for any Markov Decision Process, the following holds:
[Equation]
Proof.
The proof follows the standard approach by Silver et al.
and we report it for completeness.
We start by deriving an expression for ∇ θ V ( s, θ ) :
[Equation]
Taking the expectation with respect to s 0 ∼ µ 0 ( s ) we have:
[Equation]
Theorem 3.3.
For any Markov Decision Process, the following holds:
[Equation]
Proof.
[Equation]
Theorem
[Equation]
Under standard regularity assumptions (Silver et al., 2014)
[Equation]
for any Markov Decision
Process
[Equation]
the following holds:
[Equation]
Proof.
[Equation]
A. I MPLEMENTATION DETAILS A.3.
In this appendix, we report the implementation details for PSSVF, PSVF, PAVF and the baselines.
We specify for each hyperparameter, which algorithms and tasks are sharing them.
Shared hyperparameters:
• Deterministic policy architecture (continuous control tasks):
We use three different deterministic policies: a linear mapping between states and actions; a single-layer MLP with 32 neurons and tanh activation; a 2-layers MLP (64,64) with tanh activations.
All policies contain a bias term and are followed by a tanh nonlinearity in order to bound the action.
• Deterministic policy architecture (discrete control tasks):
We use three different deterministic policies: a linear mapping between states and a probability distribution over actions; a single-layer MLP with 32 neurons and tanh activation; a 2-layers MLP (64,64) with tanh activations.
The deterministic action a is obtained choosing a = arg max π θ ( a | s ) .
All policies contain a bias term.
• Stochastic policy architecture (continuous control tasks):
We use three different stochastic policies: a linear mapping; a single-layer MLP with 32 neurons and tanh activation; a 2layers MLP (64,64) with tanh activations all mapping from states to the mean of a Normal distribution.
The variance is state-independent and parametrized as e 2Ω with diagonal Ω .
All policies contain a bias term.
Actions sampled are given as input to a tanh nonlinearity in order to bound them in the action space.
• Stochastic policy architecture (discrete control tasks): We use three different deterministic policies: a linear mapping between states and a probability distribution over actions; a single-layer MLP with 32 neurons and tanh activation; a 2-layers MLP (64,64) with tanh activations. All policies contain a bias term. • Policy initialization: all weights and biases are initialized using the default Pytorch initialization for PVFs and DDPG and are set to zero for ARS.
• Critic architecture: 2-layers MLP (512,512) with bias and ReLU activation functions for PSVF, PAVF; 2-layers MLP (256,256) with bias and ReLU activation functions for DDPG.
• Critic initialization: all weights and biases are initialized using the default Pytorch initialization for PVFs and DDPG.
• Batch size: 128 for DDPG, PSVF, PAVF; 16 for PSSVF.
• Actor’s frequency of updates: every episode for PSSVF; every batch of episodes for ARS; every 50 time steps for DDPG, PSVF, PAVF.
• Critic’s frequency of updates: every episode for PSSVF; every 50 time steps for DDPG, PSVF, PAVF.
• Replay buffer: the size is 100k; data are sampled uniformly.
Optimizer: Adam for PVFs and DDPG.
Tuned hyperparameters:
• Number of directions and elite directions for ARS ([directions, elite directions]): tuned with values in [[1 , 1] , [4 , 1] , [4 , 4] , [16 , 1] , [16 , 4] , [16 , 16]] .
• Policy’s learning rate: tuned with values in [1 e − 2 , 1 e − 3 , 1 e − 4] .
• Critic’s learning rate: tuned with values in [1 e − 2 , 1 e − 3 , 1 e − 4] .
• Noise for exploration: the perturbation for the action (DDPG) or the parameter is sampled from N (0 , σI ) with σ tuned with values in [1 , 1 e − 1] for PSSVF, PSVF, PAVF; [1 e − 1 , 1 e − 2] for DDPG; [1 , 1 e − 1 , 1 e − 2 , 1 e − 3] for ARS. For stochastic PSSVF and PSVF we include also the value σ = 0 , although it almost never results optimal.
Environment hyperparameters:
• Environment interactions: 1M time steps for Swimmer-v3 and Hopper-v3; 100k time steps for all other environments.
• Discount factor for TD algorithms: 0.999 for Swimmer; 0.99 for all other environments.
• Survival reward in Hopper: True for DDPG, PSVF, PAVF; False for ARS, PSSVF.
-speciﬁc hyperparameters
• Critic’s number of updates: 50 for DDPG, 5 for PSVF and PAVF
• Actor’s number of updates: 50 for DDPG, 1 for PSVF and PAVF; 10 for PSSVF.
• Observation normalization: False for DDPG; True for all other algorithms.
• Starting steps in DDPG (random actions and no training): ﬁrst 1% .
• Polyak parameter in DDPG: 0.995.
PAVF ∇ θ Q ( s, a, θ ) ablation
We investigate the effect of the term ∇ θ Q ( s, a, θ ) in the off-policy policy gradient theorem for deterministic PAVF.
We follow the same methodology as in our main experiments to ﬁnd the optimal hyperparameters when updating using the now biased gradient:
[Equation]
which corresponds to the gradient that DDPG is following.
Figure 5 reports the results for Hopper and Swimmer using shallow and deep policies.
We observe a signiﬁcant drop in performance in
Swimmer when removing part of the gradient.
In Hopper the loss of performance is less signiﬁcant, possibly because both algorithms tend to converge to the same sub-optimal behavior.
[Figure]
r e t u r n
Hopper
[Figure]
PAVFPAVF
[Figure]
biased
[Table]
Noise for exploration
[Table]
ARS For ARS, we used the ofﬁcial implementation provided by the authors and we modiﬁed it in order to use nonlinear policies.
More precisely, we used the implementation of ARSv2-t (Mania et al., 2018) [Table] which uses observation normalization, elite directions and an adaptive learning rate based on the standard deviation of the return collected
To avoid divisions by zero, which may happen if all data sampled have the same return, we perform the standardization only in case the standard deviation is not zero.
In the original implementation of ARS (Mania et al., 2018), the survival bonus for the reward in the Hopper environment is removed to avoid local minima.
Since we wanted our PSSVF to be close to their setting, we also applied this modiﬁcation.
We did not remove the survival bonus from all TD algorithms and we did not investigate how this could affect their performance.
We provide a comparison of the performance of PSSVF with and without the bonus in ﬁgure 6 using deterministic policies. [Figure]
DDPG For DDPG, we used the Spinning Up implementation provided by OpenAI (Achiam, 2018), which includes target networks for the actor and the critic and no learning for a ﬁxed set of time steps, called starting steps.
We did not include target networks and starting steps in our
PVFs, although they could potentially help stabilizing training.
The implementation of DDPG that we used (Achiam, 2018) does not use observation normalization.
In preliminary experiments we observed that it failed to signiﬁcantly increase or decrease performance, hence we did not use it.
Another difference between our TD algorithms and DDPG consists in the number of updates of the actor and the critic.
Since DDPG’s critic needs to keep track of the current policy, the critic and the actor are updated in a nested form, with the ﬁrst’s update depending on the latter and vice versa.
Our PSVF and PAVF do not need to track the policy learned, hence, when it is time to update, we need only to train once the critic for many gradient steps and then train the actor for many gradient steps.
This requires less compute.
On the other hand, when using nonlinear policies, our PVFs suffer the curse of dimensionality.
For this reason, we proﬁted from using a bigger critic.
In preliminary experiments, we observed that DDPG’s performance did not change signiﬁcantly through a bigger critic.
We show differences in performance for our methods when removing observation normalization and when using a smaller critic (MLP(256,256)) in ﬁgure 7.
We observe that the performance is decreasing if observation normalization is removed.
However, only for shallow policies in Swimmer and deep policies in Hopper there seems to be a signiﬁcant beneﬁt.
Future work will assess when bigger critics help
[Figure]
for different environments and policies removing observation normalization and using a smaller critic.
We use the hyperparameters maximizing the average return.
Learning curves are averaged over 5 seeds.
For this ablation we use deterministic policies.
Discounting in Swimmer For TD algorithms, we chose a ﬁxed discount factor γ = 0 .
99 for all environments but Swimmer-v3.
This environment is known to be challenging for TD based algorithms because discounting causes the agents to become too short-sighted.
We observed that, with the standard discounting, DDPG, PSVF and PAVF were not able to learn the task.
However, making the algorithms more far-sighted greatly improved their performance.
In ﬁgure 8 we report the return obtained by DDPG, PSVF and PAVF for different values of the discount factor in Swimmer when using deterministic policies
[Figure]
Input : Differentiable critic V w : S × Θ → R with parameters w ; deterministic or stochastic actor π θ with parameters θ ; empty replay buffer D
Output : Learned V w ≈ V
( s, θ ) , learned π θ ≈ π θ ∗
Initialize critic and actor weights w , θ repeat :
Observe state s, take action a
= π θ ( s ) , observe reward r and next state s (cid:48)
Store ( s, θ, r, s (cid:48) ) in the replay buffer D if it’s time to update then : for many steps do :
Sample a batch B 1 = { ( s, ˜ θ, r, s (cid:48) ) } from D
Update critic by stochastic gradient descent: ∇
[Equation]
]end for for many steps do :
Sample a batch B 2 = { ( s ) } from D Update actor by stochastic gradient ascent:
[Equation]
] end for end if until convergence
[Table]
Initialize critic and actor weights w , θ repeat :
Observe state s, take action a
= π θ ( s ) , observe reward r and next state s (cid:48)
Store ( s, a, θ, r, s (cid:48) ) in the replay buffer D if it’s time to update then : for many steps do :
Sample a batch B 1 = { ( s, a, ˜ θ, r, s (cid:48) ) } from D
Update critic by stochastic gradient descent: ∇
[Table]
end for for many steps do :
Sample a batch B 2 = { ( s, a, ˜ θ ) } from D Update actor by stochastic gradient ascent:
[Table]
end for end if until convergence
[Table]
[Table] [Table]
repeat :
[Table]
state
[Table]
, observe reward r and next state s
Store replay buffer D if it’s time to update then : for many steps do :
Sample
[Equation]
[Equation]
for for many steps do :
Sample
[Equation] [Equation]
[Equation]
[Equation]
for end if until convergence
A. E XPERIMENTAL DETAILS A.4. LQR
For our visualization experiment, we employ an instance of the Linear Quadratic Regulator.
Here, the agent observes a 1-D state, corresponding to its position and chooses a 1-D action.
The transitions are s (cid:48)
=
s + a
and there is a quadratic negative term for the reward: R ( s, a )
= − s 2 − a 2 .
The agent starts in state s 0 = 1 and acts in the environment for 50 time steps.
The state space is bounded in [-2,2].
The goal of the agent is to reach and remain in the origin.
The agent is expected to perform small steps towards the origin when it uses the optimal policy.
For this task, we use a deterministic policy without tanh nonlinearity and we do not use observation normalization.
Below additional details and plots for different algorithms.
PSSVF We use a learning rate of 1 e − 3 for the policy and 1 e − 2 for the PSSVF.
Weights are perturbed every episode using σ = 0 .
The policy is initialized with weight 3 . 2 and bias − 3 .
All the other hyperparameters are set to their default.
The true episodic J ( θ ) is computed by running 10,000 policies in the environment with parameters in [ − 5 , 5] ×
[ − 5 , 5] .
V w ( θ ) is computed by measuring the output of the PSSVF on the same set of policies.
Each red arrow in ﬁgure 1 represents 200 update steps of the policy.
PSVF ad PAVF Using the exact same setting, we run PSVF and PAVF in LQR environment and we compare learned V ( s 0 , θ ) and Q ( s 0 , π θ ( s 0 ) , θ ) with the true PSVF and PAVF over the parameter space. Computing the value of the true PSVF and PAVF requires computing the inﬁnite sum of discounted reward obtained by the policy. Here we approximate it by running 10,000 policies in the environment with parameters in [ − 5 , − 5] × [ − 5 , 5] for 500 time steps. This, setting γ = 0 . 99 , provides a good approximation of their true values, since further steps in the environment result in almost zero discounted reward from s 0 .
We use a learning rate of 1 e − 2 for the policy and 1 e −for the PSVF and PAVF.
Weights are perturbed every episode using σ = 0 .
The policy is updated every 10 time steps using 2 gradient steps; the PSVF and PAVF are updated every 10 time steps using 10 gradient updates.
The critic is a 1-layer MLP with 64 neurons and tanh nonlinearity.
In Figures 9 and 10 we report J ( θ ) , the cumulative discounted reward that an agent would obtain by acting in the environment for inﬁnite time steps using policy π θ and the cumulative return predicted by the PSVF and PAVF for two different times during learning.
Like in the
PSSVF experiment, the critic is able improve its predictions over the parameter space.
Since in the plots V ( s, θ ) and Q ( s, π θ ( s ) , θ ) are evaluated only in s 0 , the results show that PVFs are able to effectively bootstrap the values of future states.
Each red arrow in Figures 9 and 10 representsupdate steps of the policy.
Optimization after 15 episodes
−
[Equation]
1 −
[Table]
−
[Table]
−
[Equation]
[Table]
[Table]
[Table]
Optimization after 100 episodes
[Equation]
θ
J
[Equation]
θ )
[Table]
−
[Table]
−
[Table]
−
[Table]
[Table]
[Figure]
[Figure]
A.4. O FFLINE EXPERIMENTS
Zero-shot learning We evaluate the performance of the policies learned from scratch evaluating them with 5 test trajectories every 5 gradient steps.
In addition to the results in the main paper, we report in Figures 11 and 12 a comparison of zero-shot performance between PSSVF, PSVF and PAVF in three different environments using deterministic shallow and deep policies (2-layers
MLP(64,64)). In this task we use the same hyperparameters found in tables 4, 6 and 8.
One additional hyperparameter needs to be considered: the learning rate of the policies trained from scratch.
In Figure 3 of the main paper, we use a tuned learning rate of 0.02 that we found working particularly well for PSSVF in the Swimmer environment.
In the additional experiments in Figures 11 and 12, we use a learning rate of 0 . 05 that we found working well across all policies, environments and algorithms when learning zero-shot. We observe that, using shallow policies, PVFs can effectively zero-shot learn policies with performance comparable to the policy learned in the environment without additional tuning for the learning rate. We note the regular presence of a spike in performance followed by a decline due to the policy going to regions of the parameter space never observed.
This suggests that there is a trade-off between exploiting the generalization of the critic and remaining in the part of the parameter space where the critic is accurate.
Measuring the width of these spikes can be useful for determining the number of ofﬂine gradient steps to perform in the general algorithm. When using deep policies the results become much worse and zero-shot learned policies can recover the performance of the main policy being learned only in simple environments and at beginning of training (eg. MountainCarContinuous). We observe that, when the critic is trained (last column), the replay buffer contains policies that are very distant to policies randomly initialized. This might explain why the zero-shot performance is better sometimes at the beginning of training (eg. second column).
However, since PVFs in practice perform mostly local off-policy evaluation around the learned policy, this problem is less prone to arise in our main experiments.
Ofﬂine learning with fragmented behaviors In this task, data are generated by perturbing a randomly initialized deterministic policy every 200 time steps and using it to act in the environment. [Figure]
We use σ = 0 .
5 for the perturbations.
After the dataset is collected, the PSVF is trained using a learning rate of 1 e − 3 with a batch size of 128 .
When the policy is learned, we use a learning rate of 0.02.
All other hyperparameters are set to default values.
A.4. F ULL EXPERIMENTAL RESULTS
Methodology
In order to ensure a fair comparison of our methods and the baselines, we adopt the following procedure.
For each hyperparameter conﬁguration, for each environment and policy architecture, we run 5 instances of the learning algorithm using different seeds.
We measure the learning progress by running 100 evaluations while learning the deterministic policy (without action or parameter noise) using 10 test trajectories.
We use two metrics to determine the best hyperparameters: the average return over policy evaluations during the whole training process and the average return over policy evaluations during the last 20% time steps.
For each algorithm, environment and policy architecture, we choose the two hyperparameter conﬁgurations maximizing the performance of the two metrics and test them on 20 new seeds, reporting average and ﬁnal performance in tableand 3 respectively.
Figures 13 and 14 report all the learning curves from the main paper and for a small non linear policy with 32 hidden neurons.
Stochastic policies We include some results for stochastic policies when using PSSVF and PSVF.
Figures 15 and 16 show a comparison with the baselines when using shallow and deep policies respectively.
We observe results sometimes comparable, but often inferior with respect to deterministic policies.
In particular, when using shallow policies, PVFs are able to outperform the baselines in the MountainCar environment, while obtaining comparable performance in CartPole and InvertedPendulum. Like in previous experiments, PVFs fail to learn a good policy in Reacher. When using deep policies, the results are slightly different: PVFs outperform ARS and DDPG in Swimmer, but fail to learn InvertedPendulum. Although the use of stochastic policies can help smoothing the ob-
[Figure]
jective function and allows the agent exploring in action space, we believe that the lower variance provided by deterministic policies can facilitate learning PVFs.
A.4. S ENSITIVITY ANALYSIS
In the following, we report the sensitivity plots for all algorithms, for all deterministic policy architectures and environments.
In particular, ﬁgure 17, 18, 19, 20 and 21 show the performance of each algorithm given different hyperparameters tried during training.
We observe that in general deep policies are more sensitive and, apart for DDPG, achieve often a better performance than smaller policies.
The higher sensitivity displayed by ARS is in part caused by the higher number of hyperparameters we tried when tuning the algorithm.
A.4. T ABLE OF BEST HYPERPARAMETERS
We report for each algorithm, environment, and policy architecture the best hyperparameters found when optimizing for average return or ﬁnal return in tables 4, 5, 6, 7, 8 and 9.
[Table] [Table]
[Figure] [Figure] [Figure]
arspsvfpssvfddpgpavf
[Figure]
We use the best hyperparameters found while maximizing the average reward for each task
[Figure]
[Figure] [Figure]
[Figure]
u r
[Figure] [Figure]
[Figure] [Figure]
[Figure]
u r
[Figure]
[Table]
[Table]
1e-
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e [Table] exploration Acrobot
[Table]
1e
[Table]
1e1e- 1e
[Table]
MountainCarContinuous
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
CartPole
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
1e
[Table] [Table]
1e
[Table]
1e1e- 1e1e- 1eInvertedPendulum-v 1e1e- 1e1e- 1e1e-
Reacher-v 1e1e- 1e1e- 1e1e-
Hopper-v 1e1e- 1e1e- 1e1e-
Number of directions and elite directions
Acrobot-v (4,4) (4,4) (1,1) (1,1) (1,1) (1,1)
MountainCarContinuous-v (1,1) (1,1) (1,1) (16,4) (1,1) (1,1)
CartPole-v (4,4) (4,4) (1,1) (1,1) (4,1) (4,1)
Swimmer-v (1,1) (1,1) (1,1) (4,1) (1,1) (1,1)
InvertedPendulum-v (4,4) (4,4) (1,1) (4,4) (4,1) (16,1)
Reacher-v (16,16) (16,16) (1,1) (16,4) (1,1) (1,1)
Hopper-v (4,1) (4,1) (1,1) (1,1) (1,1) (1,1)
Noise for exploration
Acrobot-v 1e1e- 1e1e- 1e1e-
MountainCarContinuous-v 1e1e- 1e1e- 1e1e-
CartPole-v 1e1e- 1e1e- 1e1e-
Swimmer-v 1e1e- 1e1e- 1e1e-
InvertedPendulum-v 1e1e- 1e1e- 1e1e-
Reacher-v 1e1e- 1e1e- 1e1e-
Hopper-v 1e1e- 1e1e- 1e1e-
[Table]
Reacher-v 1e1e- 1e1e- 1e1e- Hopper-v 1e1e- 1e1e- 1e1e-
Noise for exploration
Acrobot-v 1.
1e1e- 1e1e-
MountainCarContinuous-v 1.
1e1e- 1.
1e1e-
CartPole-v 1.
1e1e- 1eSwimmer-v 1.
InvertedPendulum-v 1.
1e1e- 1e1e-
Reacher-v 1.
1e1e-
Hopper-v 1.
1e1e- 1e1.
[Table]
Policy: [] [64,64] [] [
[Table]
Metric:
[Table]
Acrobot-v 1e1e- 1e1e- MountainCarContinuous-v 1e1e- 1e1e-
CartPole-v 1e1e- 1e1e-
Swimmer-v 1e1e- 1e1e-
InvertedPendulum-v 1e1e- 1e1e-
Reacher-v 1e1e- 1e1e-
Hopper-v 1e1e- 1e1e-
Learning rate critic
Acrobot-v 1e1e- 1e1e- MountainCarContinuous-v 1e1e- 1e1e-
CartPole-v 1e1e- 1e1e-
Swimmer-v 1e1e- 1e1e-
InvertedPendulum-v 1e1e- 1e1e-
Reacher-v 1e1e- 1e1e-
Hopper-v 1e1e- 1e1e-
Noise for exploration
Acrobot-v 1. MountainCarContinuous-v 1.
1e1. 1eCartPole-v 1. 1eSwimmer-v 1. 1e1.
1eInvertedPendulum-v 1.
1eReacher-v 1e0.
Hopper-v 1.
1e1. 1e-
[Table]
-v 1. 1e1e- 1e1e- Reacher-v 1e1e- 1e1.
Hopper-v 1.
1e1e- 1e1.
[Table]
MountainCarContinuous-v 1e1e- 1e1e- 1e1e-
Swimmer-v 1e1e- 1e1e- 1e1e-
InvertedPendulum-v 1e1e- 1e1e- 1e1e-
Reacher-v 1e1e- 1e1e- 1e1e-
Hopper-v 1e1e- 1e1e- 1e1e-
[Table]
MountainCarContinuous-v 1e1e- 1e1e- 1e1e-
Swimmer-v 1e1e- 1e1e- 1e1e-
InvertedPendulum-v 1e1e- 1e1e- 1e1e-
Reacher-v 1e1e- 1e1e- 1e1e-
Hopper-v 1e1e- 1e1e- 1e1e-
Noise for exploration
MountainCarContinuous-v 1e1e- 1e1e- 1e1e-
Swimmer-v 1e1e- 1e1e- 1e1e-
InvertedPendulum-v 1e1e- 1e1e- 1e1e-
Reacher-v 1e1e- 1e1e- 1e1e-
Hopper-v 1e1e- 1e1e- 1e1e-