S AMPLES A RE U SEFUL ? N OT A LWAYS : DENOISING POLICY GRADIENT UPDATES USING VARIANCE EXPLAINED
Anonymous authors
Paper under double-blind review
A BSTRACT
Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efﬁciently sampling an environment.
However, while most sampling procedures are based on sampling the agent’s policy directly, self-performance measures could be used to improve sampling before each policy update.
Following this line of thoughts, we propose SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise.
The criterion, V ex , is the fraction of variance explained by the value function V : a measure of the discrepancy between V and the returns.
In this work, V ex is used to estimate the impact transitions will have on learning: it reﬁnes sampling by simplifying the underlying state space and improves policy gradient methods.
In this paper: (a) We introduce V ex , the criterion used for denoising policy gradient updates.
We refer to this procedure as transition dropout and explore its implications on learning.
(b) We conduct experiments across a variety of benchmark environments, including continuous control problems.
SAUNA clearly improves performance.
(c) We investigate how V ex reliably selects samples with most positive impact on learning.
(d) We study how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.
I NTRODUCTION
Learning to control agents in simulated environments has been a challenge for decades in reinforcement learning (RL) (Nguyen & Widrow, 1990; Werbos, 1989; Schmidhuber & Huber, 1991;
Robinson & Fallside, 1989) and has recently led to a lot of research efforts in this direction (Mnih et al., 2013; Burda et al., 2019; Ha & Schmidhuber, 2018; Silver et al., 2016; Espeholt et al., 2018), notably in policy gradient methods (Schulman et al., 2016; Silver et al., 2014; Lillicrap et al., 2016;
Mnih et al., 2016).
Despite the deﬁnite progress made, policy gradient algorithms still heavily suffer from sample inefﬁciency (Kakade, 2003; Wu et al., 2017; Schulman et al., 2017; Wang et al., 2017).
In particular, many policy gradient methods are subject to use as much experience as possible in the most efﬁcient way.
However, the quality of the sampling procedure also determines the ﬁnal performance of the agent.
We make the hypothesis that not all experiences are worth to use in the gradient update.
Indeed, while perhaps trajectory simulations should be as rich as possible, some transitions may instead add noise to the gradient update, diluting relevant signals and hindering learning.
The central idea of SAUNA is to reject transitions that are not informative for the particular task at hand.
For that purpose, we use a measure of the discrepancy between the estimated state value and the observed returns.
This discrepancy is formalized with the notion of the fraction of variance explained V ex (Kv˚alseth, 1985).
Transitions for which V ex is close to zero are those for which the correlation between the value function V and the observed returns is also close to zero.
SAUNA keeps transitions where there is either strong correlation or lack of ﬁt between V and the returns, while avoiding the learning signals to be diluted by the dropped out samples.
We will examine the impact of transition dropout (inspired by Srivastava et al.
(2014) and Freeman et al.
and its theoretical implications.
We consider on-policy methods for their unbiasedness and stability compared to off-policy methods (Nachum et al., 2017).
However, our method can be applied to off-policy methods as well, and we leave this investigation open for future work.
The contributions of this paper are summarized as follows:
We propose to move from a simple policy-based sampling procedure based to one taking into account the agent’s ability in an environment measured by V ex .
We explore how the use of V ex can drive the alignment between the samples used to update the policy and the agent’s progress, while simplifying the underlying state space with transition dropout.
We provide a method that transforms policy gradient algorithms by assuming that not all samples are useful for learning and that these disturbing samples should, therefore, be rejected.
While our method is a simple extension of policy gradient algorithms, it adds a variance criterion to the optimization problem and introduces a novel rejection sampling procedure.
By combining (1) and (2), we obtain a learning algorithm that is empirically effective in learning neural network policies for challenging control tasks.
Our results extend the stateof-the-art in using reinforcement learning for high-dimensional continuous control.
We also evaluate the theoretical implications of our method.
P RELIMINARIES
We consider a Markov Decision Process (MDP) with states s ∈ S , actions a ∈ A , transition distribution s t +1 ∼ P ( s t , a t ) and reward function r ( s, a ) .
Let π = { π ( a | s ) , s ∈ S , a ∈ A} denote a stochastic policy and let the objective function be the traditional expected discounted reward:
[Equation]
where γ ∈ [0 , 1) is a discount factor (Puterman, 1994) and τ = ( s 0 , a 0 , s 1 , . .
is a trajectory sampled from the environment.
Policy gradient methods aim at modelling and optimizing the policy directly (Williams, 1992).
The policy π is generally implemented with a function parameterized by θ .
In the sequel, we will use θ to denote the parameters as well as the policy.
In deep reinforcement learning, the policy is represented in a neural network called the policy network and is assumed to be continuously differentiable with respect to its parameters θ .
2. F RACTION OF VARIANCE EXPLAINED : V ex
The fraction of variance that the value function explains about the returns corresponds to the proportion of the variance in the dependent variable V that is predictable from the independent variable s t .
We deﬁne V exτ as the fraction of variance explained for a trajectory τ :
[Equation]
where ˆ R t and V ( s t ) are respectively the return and the expected return from state s t ∈ τ , and R is the mean of all returns in trajectory τ .
In statistics, this quantity is also known as the coefﬁcient of determination R 2 and it should be noted that this criterion may be negative for non-linear models (Kv˚alseth, 1985), indicating a severe lack of ﬁt of the corresponding function:
• V exτ = 1 : V perfectly explains the returns - V and the returns are correlated ; • V exτ =
corresponds to a simple average prediction - V and the returns are not correlated ; • V exτ < 0 : V provides a worse ﬁt to the outcomes than the mean of the returns.
One can have the intuition that V exτ close to 1 is interesting because it gives samples from an exercised behavior while V exτ < 0 corresponds to a high mean-squared error for the value function, meaning the agent will learn from the corresponding samples.
On the other hand, V exτ close todoes not provide any valuable information for a correct ﬁtting of the state value function.
We will demonstrate that V ex is a relevant indicator for assessing self-performance in RL.
2. P OLICY GRADIENT METHODS : PPO AND A2C
We use PPO (Schulman et al., 2017) throughout this work.
We also use A2C, a synchronous variant of Mnih et al.
to demonstrate SAUNA’s performance.
Below we provide some details regarding PPO, as we use it generously in this work.
In previous work, PPO has been tested on a set of benchmark tasks and has produced impressive results in many cases despite a relatively simple implementation.
At each iteration, the new policy θ new is obtained from the old policy θ old :
[Equation]
We use the clipped version of PPO whose objective function is:
[Equation]
where
[Equation]
A is the advantage function, A ( s, a ) (cid:44) Q ( s, a ) − V
( s ) .
The expected advantage function A π θold is estimated by an old policy and then re-calibrated using the probability ratio between the new and the old policy.
In Eq.
4, this ratio is constrained to stay within a small interval around 1, making the training updates more stable.
2. R ELATED W ORK
Our method incorporates three key ideas: (a) function approximation with a neural network combining or separating the actor and the critic with an on-policy setting, (b) transition dropout reducing signal dilution in gradient updates while simplifying the underling MDP and (c) using V exτ as a measure of correlation between the value function and the returns to allow for better sampling and more efﬁcient learning.
Below, we consider previous work building on some of these approaches.
Actor-critic algorithms essentially use the value function to alternate between policy evaluation and policy improvement (Sutton & Barto, 1998; Barto et al., 1983).
In order to update the actor, many methods adopt the on-policy formulation (Peters & Schaal, 2008; Mnih et al., 2016; Schulman et al., 2017).
However, despite their important successes, these methods suffer from sample complexity.
In the literature, research has also been conducted in prioritization sampling.
While Schaul et al.
(2016) makes the learning from experience replay more efﬁcient by using the TD error as a measure of these priorities in an off-policy setting, our method directly selects the samples on-policy.
Schmidhuber (1991) is related to our method in that it calculates the expected improvement in prediction error, but with the objective to maximize the intrinsic reward through artiﬁcial curiosity.
Instead, our method estimates the expected fraction of variance explained and ﬁlters out some of the samples to improve the learning efﬁciency.
Finally, motion control in physics-based environments is a long-standing and active research ﬁeld.
In particular, there are many prior work on continuous action spaces (Schulman et al., 2016; Levine & Abbeel, 2014; Lillicrap et al., 2016; Heess et al., 2015) that demonstrate how locomotion behavior and other skilled movements can emerge as the outcome of optimization problems.
SAUNA: A DYNAMIC TRANSITION DROPOUT METHOD 3. E STIMATING V ex
While sampling the environment, SAUNA rejects samples where V ( s t ) is not correlated with the return from s t .
Therefore, V exτ must be estimated at each timestep so we deﬁne V exθ ( s t ) as the prediction of V exτ under parameters θ at state s t ∈ τ .
In addition, for shared parameters conﬁgurations, an error term on the value estimation is added to the PPO objective.
The ﬁnal objective becomes:
[Equation]
t
[Equation]
t
[Equation]
where c 1 and c 2 are respectively the coefﬁcient for the squared-error loss of the value function and of the fraction of variance explained function.
For cases where the network is not shared between the policy and the value function, V exτ is added to the value function network.
Appendix A illustrates how V exθ ( s t ) is embedded in the original architecture.
The rest of the network is unchanged, making it very easy to use SAUNA without altering the complexity of existing policy gradient methods.
U SING V ex FOR DYNAMIC TEMPORAL ABSTRACTION
Mechanistically, SAUNA results in dropping out several transitions from a trajectory τ , before each gradient update, and as a function of the starting state s t of those transitions.
The mechanism is analogous to the method of dropout in deep learning but articulated here by a dropout in the state space of the MDP.
To evaluate the theoretical implications of the transition dropout, our method can be formulated using the Options framework (Sutton et al., 1999; Precup, 2000) thoroughly applied to policy gradient methods in Smith et al.
Previous work shows that by reasoning at several levels of abstraction, reinforcement learning agents are able to infer, learn, and plan more effectively.
We detail below how SAUNA can be theoretically understood using the Options framework.
In this work, we take semi-Markov options that may depend on the entire preceding sequence.
The agent is given access to a set of options, indexed by σ .
Interestingly, the framework can be reduced as follows: all options share the same policy, π σ ( a t
| s t )
=
π θ ( a t | s t ) , introduced in section 2, the same initiation set and the same termination condition which we parameterize as the function β θ ( s t ) .
β θ ( s t ) represents the probability of terminating an option and is deﬁned as follows:
[Equation]
, (7) where for simplicity we rewrite V exθ ( s t ) as V ext , ˜ V ex 0: t − 1 is the median of V exθ between timesteps 0 and t − 1 , (cid:15)
= 10 − 8 is to avoid division by zero and r is the dropout parameter.
One may legitimately ask why not use directly b θ ( s t ) =
|V ext | .
The rationale is practical: the ratio becomes a standardized measure as the agent learns, stabilized by the median, more robust to outliers than the mean.
a a
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
ext
[Figure]
(b)
Fig.
1a illustrates an example of transition dropout dynamics in SAUNA while Fig.
1b depicts the behavior of Eq.
The termination function β θ ( s t ) dynamically selects the transitions for which V ext is either high or low, but not in between.
Those transitions should have a strong impact on learning: a high absolute score means that the samples correspond to a state s t for which the value function is highly correlated with the returns or does not ﬁt them properly.
With this modiﬁcation, the resulting gradients should be affected accordingly, which we investigate in section 4.2.2.
3. P UTTING IT ALL TOGETHER : SAUNA SAMPLING PROCEDURE
In this work, we hypothesize that low correlated transitions dilute the valuable signal information of the transitions with high absolute correlation.
The temporal abstraction provided by the Options framework is leveled dynamically thanks to a termination function exploiting the self-performance measure V ext .
Instead of keeping these noisy transitions, the sampling procedure drops them until the trajectory is T -steps long.
Algorithm 1 illustrates how learning is achieved with SAUNA applied to PPO, in particular, the ﬁtting of the V ex function in Eq.
10 and the transition dropout in the if statement.
We choose to depict a conﬁguration where the parameters between the policy network, the value function and the V ex function are not shared, since from this conﬁguration the shared parameter case is direct.
Algorithm
Initialize policy parameters θ
Initialize value function parameters φ
and V ex function parameters ψfor k = 0 , 1 , 2 , . . .
do (cid:46)
For each update k
Initialize trajectory τ to capacity T while size( τ ) ≤ T do (cid:46)
For each timestep t a
[Equation]
∼
[Equation]
execute action a t and observe reward r t +1 and next state s t + if |V ext | |
[Equation]
V
[Equation]
≥ r then collect transition ( s t , a t , r t , v t , s t +1 , V ext ) in τ elsecontinue without collecting the transition
Gradient Update
[Equation]
PPO sampling
SAUNA dropout
H OW DOES SAUNA LEARN ?
For ease of reproducibility and sharing, we have forked the original baselines repository from OpenAI and modiﬁed the code to incorporate our method 1 .
The complete list of hyperparameters and details of our implementation are given in Appendix E and F respectively.
A discussion about additional experiments whose results are non-positive, but which we think contribute positively to this paper, can be found in Appendix B.
4. SAUNA IN THE CONTINUOUS DOMAIN : M U J O C O AND R OBOSCHOOL
To verify the generalizability of our method, we study SAUNA against PPO and A2C.
We compare
SAUNA (PPO+ V ex in red) with its natural baseline PPO (PPO in blue).
We use 6 simulated robotic tasks from OpenAI Gym (Brockman et al., 2016) using MuJoCo (Todorov et al., 2012).
The two hyperparameters required by our method ( r = 0 .
3 from Eq.
2 and c 2 = 0 .
5 from Eq. 6) and all the others (identical to those in Schulman et al.
(2017)) are exactly the same for all tasks.
We made this choice within a clear and objective framework of comparison between the two methods.
Thus, we have not optimized the rest of the hyperparameters for SAUNA, and its reported performance is not necessarily the best that could be obtained with more intensive tuning.
[Figure]
From the graphs reported in Fig.
2, we see that our method outperforms all continuous control tasks.
We also present in Table 1 the scores obtained for each task on the experiments with PPO and A2C.
The graphs for A2C are reported in Appendix C.1.
[Table]
We then experiment with the more difﬁcult, high-dimensional continuous domain environment of
Roboschool (Klimov & Schulman, 2017) with different neural network capacities for the model.
When resources are limited in terms of number of parameters, it seems natural that dropping out samples based on their predicted learning impact allows to reduce noise in the gradient update and accelerate learning.
And as a result, our method is faster and more efﬁcient than the baseline.
When resources are not limited, the gap closes towards the end of the training and our method performs as well as the baseline.
4. U NDERSTANDING THE IMPACT OF SAUNA ON LEARNING 4.2. W HAT ADVANTAGE DOES DROPOUT GIVE ?
We further study the impact of dropping out noisy samples by conducting additional experiments in predicting V ex while omitting the ﬁltering step before the gradient update: the if statement in Algorithm 1 is removed and all transitions are collected in τ .
Indeed, the SAUNA algorithm could improve the agent’s performance by simply training the shared network to optimize the variance explained head.
Fig.
3 (full results are provided in Appendix D) demonstrates the positive effects of dropping out the samples.
In addition, we studied the number of sample dropouts per task and their evolution throughout the training.
On average, SAUNA rejects 5-10% of samples at the beginning of training which reduces to 2-6% at the end.
[Figure]
4.2. H OW DOES SAUNA IMPACT THE GRADIENTS ?
Prior to the gradient updates
[Figure]
SAUNA removes the transitions for which the state value is poorly correlated with the returns.
By doing so
[Figure]
we hypothesized that information signals from samples with signiﬁcant V ex would be less diluted by dropped out samples.
One can easily see by the graphs that the norm of the gradients is affected by such a change in the learning procedure.
Fig.
4 shows that SAUNA dropout generates larger gradients.
Hence policy updates have bigger steps, which eventually results in better performance
[Figure]
One may wonder quite rightly why performance is not damaged, since larger gradients could hinder learning.
The performance results suggest that the gradients contain more useful information from each of the transitions that passed SAUNA sampling.
In other terms, the relevant information for the task at hand is less diluted, the gradients are more qualitative and have been partially denoised.
4.2. H ALF C HEETAH : A QUALITATIVE STUDY
Under review as a conference paper at ICLR
4.3 R OBOSCHOOL
We then experimented with the more difﬁcult, high-dimensional continuous domain environment of Roboschool: RoboschoolHumanoidFlagrunHarder-v1 .
Thepurposeofthistaskistoallowtheagent to run towards a ﬂag whose position varies randomly over time.
It is left to fall and is continuously bombarded by white cubes that push it out of its path.
[Figure]
In Fig.
4a, the same fully-connected network as for the MuJoCo experiments (2 hidden layers each with 64 neurons) is used.
In Fig.
4b, the network is composed of 3 hidden layers with 512,and 128 neurons.
We trained those agents with 32 parallel actors.
In both experiments, our method performsbetterandfasteratthebeginning.
Then, whenthepolicyandvaluefunctionsbeneﬁtfroma larger network, the gap closes, and our contribution does as well as its baseline.
When resources are limited parameter-wise, it seems natural that ﬁltering out samples based on their predicted training impact allows to remove noise from the gradient update and accelerate learning.
Finally, we investigated further and conducted the same experiment as with the larger network (3 hidden layers with 512, 256 and 128 neurons), but with 128 actors in parallel instead of 32.
The total number of timesteps is unchanged and is still equal to 100M. Results are reported in Fig.
4c: our method is still faster and achieves better performance than the baseline
4.4 V ARIANCE EXPLAINED V ex : CASE STUDY
While studying HalfCheetah-v2 , we observed that for a number of seeds, PPO was converging to a localminimumforcingtheagenttomoveforwardonitsback.
Thisisawell-knownbehavior(Lapan, 2018).
However, we observed that our method made it possible to leave from, or at least to avoid theselocalminima.
Thoseparticulardeterministicenvironmentscanbegeneratedreproduciblywith speciﬁc seeds.
This is illustrated in Fig.
5 where we can look at still frames of two agents trained for 10 6 timesteps on identically seeded environments.
[Figure]
In HalfCheetah , a well-known behavior (Lapan, 2018) is that for multiple seeds PPO is converging to a local minimum forcing the agent to move forward on its back.
However, we observed that SAUNA made it possible to leave from, or at least to avoid these local minima.
This is illustrated in Fig.
5a where we can see still frames of two agents trained with PPO and SAUNA for 10 6 timesteps on identically seeded environments.
Their behavior is entirely different.
Looking at V ex in Fig.
5b, we can see that the graphs differ quite interestingly.
The orange agent seems to ﬁnd very quickly a local minimum on its back while the green agent’s V ex varies much more.
This seems to allow the latter to explore more states than the former and ﬁnally ﬁnd the fastest way forward.
Supported by the previous study, we can infer that SAUNA agents are better able to explore interesting states while exploiting with conﬁdence the value given to the states observed so far.
D ISCUSSION
Intuitively, for the policy update, our method will only use qualitative samples that provide the agent with (a) reliable and exercised behavior (high V ex ) and (b) challenging states from the point of view of correctly predicting their value (low V ex ).
The SAUNA algorithm keeps samples with high learning impact, rejecting other noisy samples from the gradient update.
5. D ENOISING POLICY GRADIENT UPDATES AND THE POLICY GRADIENT THEOREM
Policy gradient algorithms are backed by the policy gradient theorem (Sutton et al., 2000):
[Equation]
As long as the asymptotic stationary regime is not reached, it is not reasonable to assume the sampled states to be independent and identically distributed (i.i.d.).
Hence, it seems intuitively better to ignore some of the samples for a certain period, to allow the most efﬁcient use of information.
One can understand SAUNA as making gradient updates more robust through dropout, especially when the update is low and the noise can be dominant.
Besides, not taking all samples reduces the bias in the state distribution d π .
Therefore, it now seems more reasonable to consider the sampled states i.i.d., which we theoretically need for the policy gradient theorem.
5. I MPACT OF V ex ON THE SHARED NETWORK PARAMETERS
The shared network predicts V ex in conjunction with the value function and the policy.
Therefore, as its parameters are updated through gradient descent, they converge to one of the objective function minima (hopefully, a global minimum).
This parameter conﬁguration integrates V ex , predicting how much the value function has ﬁt the observed samples, or informally speaking how well the value function is doing for state s t .
This new head tends to lead the network to adjust predicting a quantity relevant for the task.
Instead of using domain knowledge for the task, the method rather introduces problem knowledge by constraining the parameters directly.
C ONCLUSION
Policy gradient methods optimize the policy directly through gradient ascent with the objective to maximize the expected return.
We have introduced a new, lightweight and agnostic method technically applicable to any policy gradient method using a neural network as function approximation.
The central idea of this paper is that samples that are uncorrelated with the return are dropped out and not considered for the policy update.
Those low correlated samples are ignored by SAUNA, the mechanism is reﬂected in a dynamic temporal abstraction controlled by the estimated variance explained of each state.
The relevant signal information being less diluted, this results in a denoising effect on the gradients, ultimately leading to improved performance.
We demonstrated the effectiveness of our method when applied to two policy gradient methods on several standard benchmark environments.
We also established that samples can be removed from the gradient update without hindering learning but can, on the opposite, improve it.
We further studied the impact that such a modiﬁcation in the sampling procedure has on learning.
Several open topics warrant further study.
The inﬂuence of SAUNA on the distribution of states could, for instance, be theoretically investigated.
We also ﬁnd the effort (Cobbe et al., 2019; Zhang et al., 2018) to go further towards generalization in RL very promising, and we think SAUNA could be useful in these problems as a way to regularize policy gradient methods.
A I LLUSTRATION OF THE SAUNA ARCHITECTURE
Shared network v t π t a t ∼ π t s t
PPO (shared network)
Policy network π t a t ∼ π t s t
Value network s t v t
PPO (separate network)
Shared network v t π t a t ∼ π t 𝒱 ext s t
SAUNA (shared network)
Policy network π t a t ∼ π t s t
Value network s t v t
SAUNA (separate network)
𝒱 ext
In Fig.
6 we rewrite V exθ ( s t ) as V ext .
On the right side of the ﬁgure is illustrated the V ext head that is added to the shared or the separate network conﬁgurations.
Note that even if V exτ is deﬁned for a sampled trajectory τ , the model predicts its value at each state s t ∈ τ .
B A DDITIONAL EXPERIMENTS WITH NON - POSITIVE RESULTS
Atari domain.
We tested our method on the Atari 2600 domain (Bellemare et al., 2013) without observing any improvement in learning.
By comparing the two algorithms where the method of ﬁltering the samples is used or not, we could not observe any difference, as some tasks were better performed by one method and others by the other.
Mean of V ex .
Although ˜ V ex , the median of V ex , is more expensive to calculate, we observe that it gives much better results than if we use its mean in Eq. 7.
Using the median helps (Kv˚alseth, 1985) because the distribution of V ex is not normal and includes outliers that will potentially produce misleading results.
Non-empirical V ex .
We also experimented with using the real values of V ex in Eq.
7 when calculating ˜ V ex 0: t − 1 , instead of the predicted ones.
This has yielded less positive results, and it is likely that this is due to the difference between the predicted and actual values at the beginning of learning, which has the effect of distorting the ratio in Eq.
Adjusting state count.
In order to stay in line with the policy gradient theorem (Sutton et al., 2000), we have worked to adjust the distribution of states d π to what it really is, since some states that the agent has visited are not included in the gradient update.
We adjusted it using the ratio between the number of states visited and the actual number of transitions used in the gradient update, but this did not improve the learning, and instead, we observed a decrease in performance.
Random dropout.
We experimented with dropping out at random, and before each gradient update, a number of samples corresponding to the same approximate number of samples that SAUNA drops.
This resulted in a decrease in performance compared to PPO, as one can expect.
C A DDITIONAL EXPERIMENTS WITH SAUNA IN THE CONTINUOUS DOMAIN C. M U J O C O : C OMPARISON OF A2C+SAUNA AND A2C
[Figure]
In Fig.
7 we compare A2C+SAUNA (A2C+ V ex in red) with A2C (A2C in green) on again 6 simulated robotic tasks from OpenAI Gym using the MuJoCo physics engine.
C.
R OBOSCHOOL : C OMPARISON OF PPO+ V ex AND PPO
We experiment here with the more difﬁcult, high-dimensional continuous domain environment of
Roboschool (Klimov & Schulman, 2017): RoboschoolHumanoidFlagrunHarder-v1 .
The purpose of this task is to allow the agent to run towards a ﬂag whose position varies randomly over time.
It is continuously bombarded by white cubes that pushes it out of its path, and if it does not hold itself up it is left to fall.
[Figure]
In Fig.
8a, the same fully-connected network as for the MuJoCo experiments (2 hidden layers each with 64 neurons) is used.
In Fig.
8b, the network is composed of a larger 3 hidden layers with 512, 256 and 128 neurons.
We trained those agents with 32 parallel actors.
In both experiments, SAUNA performs better and learns faster at the beginning.
Then, only when the policy and value functions beneﬁt from a larger network, the gap closes, and our method does as well as the baseline.
When resources are limited in terms of the number of parameters, it seems natural that ﬁltering out samples based on their predicted training impact helps to reduce noise in the gradient update and accelerate learning.
Finally, we investigated further and conducted the same experiment with the larger network (3 hidden layers with 512, 256 and 128 neurons), but with 128 actors in parallel instead of 32.
Results are reported in Fig.
8c: our method still learns faster and achieves better performance than the baseline
[Figure]
D A DDITIONAL EXPERIMENTS ON THE ADVANTAGE OF TRANSITION DROPOUT
In order to identify the effects of the training of the V ex head and the ﬁltering out of sample
[Figure]
we verify the hypothesis that ﬁltering out noisy samples does improves the performance.
To do so, in Section 4.2.
[Figure]
we conduct experiments where the network predicts V ex but where the noisy samples are not ﬁltered out: the if statement in Algorithm 1 is removed and all transitions are collected in τ .
In Fig.
[Figure]
we see that when the noisy samples are not ﬁltered out the performance is worst than the baseline, conﬁrming the positive denoising impact of ﬁltering out the variance-selected samples
[Figure]
E H YPERPARAMETERS
We have tuned the hyperparameters of our method by performing a grid search and selecting the best combinations by considering those with the largest consensus.
[Table] [Table]
F I MPLEMENTATION DETAILS
Unless otherwise stated, the policy network used for MuJoCo and Roboschool tasks is a fullyconnected multi-layer perceptron with 2 hidden layers of 64 units.
For Atari, the network is shared between the policy and the value function and is the same as in Mnih et al.
The architecture for the V ex function head is the same as for the value function head.
G C LIPPED SURROGATE OBJECTIVE DETAILS
In Eq.
4, we use the following standard deﬁnitions for the advantage function:
[Equation]
(12) where
Q
[Equation]
(cid:88) l = γ l r t + l (cid:35) and V π ( s t ) =
[Equation] [Equation]
t
[Equation]
1 : ∞ (cid:34)
[Equation]
(cid:88) l = γ l r t + l (cid:35) .
H A N ANALOGY WITH SAUNAS
Saunas originated in Northern Europe and are thought to date back to 7000 BC.
Their use helps to release impurities [ﬁltered out noisy samples] and improves the regeneration of cells [improved policy gradient updates].
Their temperatures could be fatal if not regulated by humidity [ V ex criterion].