Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
Abstract
Improving the sample efﬁciency of reinforcement learning algorithms requires effective exploration.
Following the principle of optimism in the face of uncertainty (OFU), we train a separate exploration policy to maximize the approximate upper conﬁdence bound of the critics in an off-policy actor-critic framework.
However, this introduces extra differences between the replay buffer and the target policy regarding their stationary state-action distributions.
To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy RL training.
In particular, we correct the training distribution for both policies and critics.
Empirically, we evaluate our proposed method in several challenging continuous control tasks and show superior performance compared to state-of-the-art methods.
We also conduct extensive ablation studies to demonstrate the effectiveness and rationality of the proposed method.
Introduction
Incorporating high capacity function approximators (e g., neural network) with actor-critic algorithms has shown great promise in many areas [Mnih et al., 2013, Silver et al., 2016, Kalashnikov et al., 2018, Shani et al., 2005, Li et al., 2019].
However, its training instability and poor sample efﬁciency hinder its wider deployment in the real world.
While these issues have been studied under different settings, one of the most powerful and practical frameworks is the off-policy actor-critic algorithm [Degris et al., 2012, Haarnoja et al., 2018, Fujimoto et al., 2018, Lillicrap et al., 2015], where a replay buffer is maintained to store previously collected transitions and both the policy and the critic are trained by sampling transitions from it.
There are two major obstacles that a successful off-policy actor-critic algorithm has to overcome.
Firstly, a carefully-designed exploration strategy is required to learn efﬁciently.
Secondly, off-policy training is unstable when combing with function approximation and bootstrapping, known as the deadly triad [Sutton and Barto, 2018, Van Hasselt et al., 2018].
To stabilize training, state-of-the-art (SOTA) methods, e.g., SAC [Haarnoja et al., 2018] and TD3 [Fujimoto et al., 2018], use the minimum value between a pair of critics as an approximate lower bound to alleviate the overestimation of Q -values.
To facilitate exploration, TD3 injects Gaussian white noise to the deterministic action while SAC explicitly trains its policy to have a target entropy depending on the dimensionality of the action space.
Given the success of the OFU principle [Auer et al., 2002, Jin et al., 2020], OAC proposes to maintain a separate exploration policy by maximizing an approximate upper bound of the critics and empirically showing that such an exploration strategy leads to more efﬁcient exploration.
To limit the amount of off-policyness and stabilize training, OAC further constrains the KL divergence between the exploration and target policies.
While the intuition of OAC to leverage an “optimistic" exploration policy could be beneﬁcial for the efﬁcient exploration, we argue that an important aspect of off-policy training has not been taken into account in all the above methods, i.e., a faithful policy gradient computation requires samples from the on-policy distribution which is in general not satisﬁed by directly sampling from the relay buffer in off-policy training.
This issue becomes even more severe when a separate exploration policy is adopted.
As a trade-off, OAC imposes a KL constraint between the exploration and the target policy, limiting the potential of deploying a more effective exploration policy to execute informative actions spuriously underestimated by the target policy. However, it still suffers from the distribution shift during off-policy training of the target policy.
In this work, we propose directly applying a distribution correction scheme in off-policy RL where a separate optimistic exploration policy can be trained solely on executing the most informative actions without worrying about its “dissimilarity" to the target policy.
Estimating the distribution correction ratio, however, is non-trivial.
Traditional methods based on importance sampling suffer from high variance [Precup et al., 2001].
The recently proposed Distribution Correction Estimation (DICE) family [Nachum et al., 2019a,b, Zhang et al., 2020a,b, Uehara et al., 2020, Yang et al., 2020] provides us with a way to estimate the distribution correction ratio in a low-variance manner by solving a linear program (LP).
However, these methods are designed for the off-policy evaluation (OPE) task, where both the target policy and replay buffer are ﬁxed.
It is unclear how to incorporate them into the more sophisticated off-policy RL.
In this work, we successfully integrate the DICE optimization scheme into the off-policy actor-critic framework and stabilize the DICE-related training process by modifying the original optimization problem.
Compared with OAC, our distribution-corrected training procedure enables learning a more effective exploration policy without enforcing any distance constraint between the exploration and the target policy.
Empirically, we show that our methods achieve a higher sample efﬁciency in challenging continuous control tasks than SOTA off-policy RL algorithms, demonstrating the power of the proposed distribution-corrected off-policy RL framework in allowing for a more effective exploration strategy design.
Moreover, we examine the quality of our learned correction ratio by using it to construct the dual estimator proposed in [Yang et al., 2020] to estimate the average per-step rewards (on-policy rewards) of the RL target policy.
We highlight that this is harder than the standard OPE setting, as our target policy and replay buffer are changing during RL training.
Surprisingly, the dual estimator gives a decent estimation by matching the on-policy rewards, demonstrating the effectiveness of our distribution correction scheme.
To the best of our knowledge, this is the ﬁrst work to improve the exploration efﬁciency of RL algorithms by explicitly correcting the training distribution rather than imposing divergence constraints, opening up a direction of potentially more profound exploration strategy design.
Preliminaries 2.1 Reinforcement Learning
We consider the standard Reinforcement Learning (RL) setup that maximizes the expected discounted cumulative rewards in an inﬁnite horizon Markov Decision Process (MDP).
Formally, a MDP is deﬁned by a tuple ( S , A , R , T , γ ), with state space S , action space A , reward function R ( s, a ) , transition function T ( s (cid:48) | s, a ) , and discount factor γ ∈ (0 , 1) .
At timestep t , the agent selects an action a t ∼ π ( ·| s t ) for current state s t by policy π ( ·| s t ) , receives a reward r t :=
R ( s t , a t ) and transits to the next state s t +1 ∼ T ( ·| s t , a t ) .
We deﬁne the action value function of policy π as Q π ( s, a ) =
E τ ∼ π : s
= s,a 0 = a [Σ ∞ t =
γ t r t ] , where τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . . )
is a trajectory generated by rolling out in the environment following π .
Thus, the standard RL objective can be expressed as J = E s 0 ∼ ρ 0 ( s ) ,a ∈ π [ Q π ( s 0 , a )] , where ρ 0 is the initial state distribution.
We further denote the discounted marginal state distribution of π as d π ( s ) and the corresponding state-action marginal as d π ( s, a ) =
d π ( s ) π ( a | s ) .
2.2 Soft Actor-Critic (SAC)
To encourage exploration, SAC [Haarnoja et al., 2018] optimizes the maximum entropy objective J = E τ ∼ π : s 0 ∼ ρ 0 ( s )
[Σ ∞ t =
γ t ( r t + α H ( ·| π ( s t ))] , where H is the entropy function and α ∈ R + .
The resulting target policy π has higher entropy, thus generating diverse actions during rollout.
SAC learns the Q value function as critics from the collected experience and uses it to guide the policy (actor) update.
To stabilize training, SAC estimates lower conﬁdence bound of the true Q value by maintaining a pair of Q networks, ˆ Q 1 and ˆ Q 2 , and taking their minimum, i.e., ˆ
Q LB ( s, a ) =
min i ∈{ 1 , 2 } ˆ Q i
s, a ) .
The two Q networks are structured identically and trained in the same way but initialized differently.
The Q value function is trained via value-based methods.
The objective function minimizes the expected single-step bellman error .
It further keeps ˆ Q (cid:48) 1 and ˆ Q (cid:48) 2 as the exponential moving average of ˆ Q 1 and ˆ Q 2 to provide a stable training target.
To this end, the target Q value is computed by
[Equation]
where a (cid:48) ∼ π ( · | s (cid:48) ) , and the training loss for Q functions are
[Equation]
As for the actor, SAC reparameterizes [Kingma and Welling, 2013] the stochastic policy π ( ·| s ) as f θ ( s, ε ) , where ε ∼ N (0 , I ) , and derives the policy gradient w.r.t its parameter θ , which is given by
[Equation]
) where φ ( ε ) denotes the density function of ε .
As we do not have access to the ground truth d π ( s ) during training, SAC approximates d π ( s ) with the state distribution d D ( s ) of the replay buffer, and applies the following Monte-Carlo estimator of (2):
[Equation]
However, when d π ( s ) exhibits large difference from d D ( s ) , the bias introduced by (3) cannot be ignored.
As we will show, correcting such a bias leads to large performance gain.
2.3 Optimistic Actor-Critic (OAC)
OAC [Ciosek et al., 2019] is built on top of SAC [Haarnoja et al., 2018], and proposes to train a separate optimistic exploration policy π E in addition to the target policy π T for deployment.
π E is designed to maximize an approximate upper conﬁdence bound ˆ Q UB of the true Q value, while π T is trained to maximize an approximate lower bound ˆ Q LB .
OAC shows that exploration using π E avoids pessimistic under-exploration and directional uniformedness [Ciosek et al., 2019], achieving higher sample efﬁciency over SAC.
To obtain the approximate upper bound ˆ Q UB , OAC ﬁrst computes the mean and variance of the Q estimates from ˆ Q 1 and ˆ Q 2 ,
[Equation]
(4) then deﬁne ˆ Q UB = µ Q + β UB σ Q , where β UB ∈ R + controls the level of optimism.
Note that the previous approximate lower bound ˆ Q LB ( s, a ) =
min i ∈{ 1 , 2 } ˆ Q i
s, a ) can be expressed as ˆ Q LB = µ Q − β LB σ Q with β LB = 1 .
OAC derives a closed-form solution of the Gaussian exploration policy π E =
( µ E , Σ E ) , by solving an optimization problem that maximizes the expectation (w.r.t.
π E ) of a linear approximation of ˆ Q UB and constraints the maximum KL Divergence between π E and π T .
Given the trained target policy π T = ( µ T , Σ T ) , the closed-form solution of π E ends up having a µ E from a directionally informed shifts of µ T , and a Σ E equal to Σ T .
Although the KL constraint enables a closed-form solution of π E and stabilizes the training, we argue that such a constraint limits the potential of π E in executing more informative actions that may better correct the spurious estimates of critics, as π E is constrained to generate actions that are not far from those generated by π T which is trained conservatively from the critics.
In our framework, we remove the KL constraint to release the full potential of an optimistic exploration policy while addressing the training stability by explicitly correcting the biased gradient estimator in policy training.
training phase buffer distributioncorrection ! "
s, r a deployment ! " !
# s, r a rollout(s) !
# {( $ % ,' % ,$ %( ,) % )} critics optimistic conservative
2.4 Distribution Correction Estimation
Distribution Correction Estimation (DICE) family [Nachum et al., 2019a, Zhang et al., 2020a,b,
Uehara et al., 2020, Nachum and Dai, 2020, Nachum et al., 2019b, Yang et al., 2020] is proposed to solve the OPE task, which seeks an estimator of the policy value, i.e., the normalized expected per-step reward, from a static replay buffer D .
DICE obtains an unbiased estimator by estimating the distribution correction ratio ζ ∗ ( s, a ) =
d π ( s,a ) d D ( s,a ) , where d D ( s, a ) is the state-action distribution of D .
We adapt the DICE optimization scheme to the off-policy actor-critic framework, and use it to estimate the distribution correction ratio for our training.
Optimistic Exploration with Explicit Distribution Correction
Following the philosophy of OAC and recent advances in the off-policy RL [Lee et al., 2021], we propose to simultaneously train a conservative target policy π T for deployment, and an optimistic exploration policy π E for executing the most informative actions w.r.t the current Q value estimation.
Figure 1 provides an overview of our framework.
We apply explicit distribution corrections to train both policies and critics, and no constraint is imposed on π E in terms of its divergence from π T .
In the rest of this section, we ﬁrst introduce our off-policy training objectives and theoretically show that it forms an unbiased policy gradient estimator for π T with the distribution correction ratio.
We then elaborate on how to estimate the distribution correction ratio by adapting the DICE optimization scheme.
We close this section by summarizing our algorithm with pseudo-codes.
3.1 Off-policy Training Objectives
π T and π E are parameterized by θ T and θ E respectively.
We start from the training objective for the target policy π T .
To prevent from overestimation, we follow previous work [Haarnoja et al., 2018, Fujimoto et al., 2018, Ciosek et al., 2019] in using the approximate lower bound ˆ Q LB as the critic, and correct the sample distribution with d πT ( s,a ) d D ( s,a ) ,
[Equation]
The following proposition (proved in the Appendix B) shows that the gradient of training objective (5) provides an unbiased estimation of the policy gradient (2).
This contrasts with previous off-policy actor-critic algorithms that directly apply the biased estimator (3) in training the target policy.
Proposition 3.1.
∇ θ T ˆ J π T gives an unbiased estimate of the policy gradient (2) with π = π T .
Regarding the exploration policy π E , there are two considerations.
Firstly, it should explore optimistically w.r.t.
the estimated Q values, in order to collect informative experience to efﬁciently correct spurious Q estimates.
Therefore, we follow OAC to use the approximate upper bound ˆ Q UB as critic in the objective.
Secondly, the ultimate goal of π E and better Q value estimation is to facilitate a more accurate gradient estimate for π T .
Therefore, the training sample distribution for π E ’s objective should be consistent with that for π T ’s. As a result, we apply the same correction ratio as in (5) to get the following objective for π E ,
[Equation]
again, this is different from all existing works that adopt a separate exploration policy.
As for training the critics, while the best sample distribution for performing off-policy Q learning is still an open problem [Kumar et al., 2020], there exists some empirical work[Sinha et al., 2020, Nachum et al., 2019b] showing that using samples from the same distribution as that for policy training beneﬁts the training of Q functions.
Therefore, we also apply the same correction ratio to Q function objectives (1) and obtain the corrected objectives:
[Equation]
3.2 Estimating Distribution Correction Ratio via DICE
We estimate the distribution correction ratio d πT ( s,a ) d D ( s,a ) by adapting the DICE optimization scheme [Yang et al., 2020].
Yang et al.
[2020] shows that existing DICE estimators can be formulated as the following min-max distributional linear program with different regularizations,
[Equation]
where g 1 and g 2 are some convex and lower-semicontinuous regularization functions, α ν and α ζ control the corresponding regularization strength.
By slightly abusing the notation, we deﬁne
B π T ν ( s, a ) =
α R · R ( s, a ) + γν ( s (cid:48) , a
[Equation]
) , a (cid:48) ∼ π T ( s (cid:48) ) with α R scaling the reward.
Solving the optimization problem (8) for ζ gives us ζ ∗ ( s, a ) =
d πT ( s,a ) d D ( s,a ) , which is exactly the distribution correction ratio.
Note that the formulation supports both discounted and undiscounted return cases.
We now provide intuitions for (8).
For simplicity, we refer to RHS as the RHS of (8).
As ζ ∗ ( s, a ) is the distribution correction ratio, λ is the Lagrangian coefﬁcient to ensure the expectation of ζ ∗ ( s, a ) to be 1 .
ν can be viewed as a synonym to the Q , with the ﬁrst line of the RHS giving the normalized expected per-step reward of policy π T .
Line 3 and 4 together can be viewed as the expected singlestep bellman error , which becomes clear if referring to the Fenchel Duality and setting g 2 ( ζ ) =
12 ζ 2 .
We refer interested readers to the original paper [Yang et al., 2020] for more details.
Directly applying the above DICE algorithms to the off-policy RL settings cast great optimization difﬁculties.
While OPE assumes a ﬁxed target policy and a static replay buffer with sufﬁcient stateaction space coverage [Nachum et al., 2019a], in RL, both the target policy and the replay buffer change during training.
We thus make three modiﬁcations to (8) to overcome the challenges.
Our ﬁrst modiﬁcation to (8) is removing the E s 0 ∈ ρ 0 ( s ) ,a 0 ∼ π ( s 0 )
[ ν ( s 0 , a 0 )] term from the objective.
In the initial training stage, the state-action pairs stored in the replay buffer only achieve little coverage of the entire state-action space, especially for high dimensional continuous control tasks.
As a result, the target policy π T may sample some a 0 that is very different from any collected experience in the replay buffer.
ν ( s 0 , a 0 ) will then be minimized pathologically low and thus incur a huge error.
Note that similar phenomena also appear in the ofﬂine RL [Fujimoto et al., 2019, Kumar et al., 2019].
Our second modiﬁcation is to add an absolute value operator | · | to the single step bellman error
B π ν ( s, a ) − ν ( s, a ) .
We empirically ﬁnd that it increases the training stability.
The intuition comes from considering solving for L ζ deﬁned by (11), since when g 2 is square function then the absolute value automatically helps the ζ function lie on the positive region without any hard projection.
Our third modiﬁcation to the original problem (8) is to set both α ν and α ζ greater than zero.
This is different from existing DICE schemes [Yang et al., 2020] that set either α ν or α ζ to be zero.
Though such a setting introduces bias and has never been applied in existing DICE algorithms, it dramatically
Input : target and explore policy parameters θ T , θ E ; ˆ Q 1 , ˆ Q 2 parameters ω 1 , ω 2 ; ˆ Q (cid:48) 1 , ˆ Q (cid:48) 2 parameters ω (cid:48) 1 , ω (cid:48) 2 ; ν , ζ parameters η ν , η ζ , ν (cid:48) parameters η ν (cid:48) , and λ ; replay buffer D 1: w (cid:48)
1 ← w 1 , w (cid:48) 2 ← w 2 , η ν (cid:48) ← η ν , D ← ∅ 2: repeat 3: for each environment step do 4:
Sample action a t ∼ π E ( ·| s t ) 5:
Observe s t +1 ∼ T ( ·| s t , a T ) and r ( s t , a t ) 6:
D ← D ∪ { ( s t , a t , r ( s t , a t ) , s t +1 ) } 7: end for 8: for each training step do 9:
Sample minibatch B = { ( s, a, r, s (cid:48) ) } from D 10: # DICE training 11:
Update η ν , η ζ , λ via ∇ η ν L ν , ∇ η ζ L ζ , ∇ ψ λ L λ 12:
Calculate ˜ ζ with (12) 13: # RL training 14:
Update θ T with ∇ θ T ˆ J π T 15:
Update θ E with ∇ θ E ˆ J π E 16:
Update w i with ∇ w i ˆ L iQ , i ∈ { 1 , 2 } 17: # Soft update 18: w (cid:48)
i ← τw i + (1 − τ ) w (cid:48)
i , i ∈ { 1 , 2 } ; η ν (cid:48) ← τη ν + (1 − τ ) η ν (cid:48) 19: end for 20: until convergence stabilizes the training in practice.
Moreover, adding two regularizations on a min-max program has been widely applied for faster convergent in optimization literature [Allen-Zhu et al., 2016].
To this end, we obtain as follows the loss functions for each of ν , ζ , and λ respectively,
[Equation]
Both ζ and ν are parameterized by neural networks with parameters η ζ and η ν respectively.
To stabilize the training, we maintain a ν (cid:48) as the exponential moving average of ν to compute B π T ν ( s, a ) .
When applying the learned ζ ( s, a ) as a surrogate of d πT ( s,a ) d D ( s,a ) in the RL training objectives, we need to tackle the issue of ﬁnite samples since our training is in minibatch.
Inspired by [Sinha et al., 2020], we apply self-normalization to ζ ( s, a ) with temperature T , and use the ˜ ζ ( s, a ) deﬁned below to construct the RL objectives (5), (6) and (7).
[Equation]
3.3 Summary of Algorithm
To this end, we summarize our overall algorithm in Algorithm 1.
Like conventional off-policy actor-critic algorithms, we alternatively perform environment interactions following π E and update of all trainable parameters.
In each training step, our algorithm ﬁrst updates the ν , ζ , and λ related to DICE, using SGD w.r.t.
losses (10), (11), and (9) respectively, and computes the distribution correction ratio ˜ ζ from the updated ζ .
With ˜ ζ , our algorithm then performs RL training to update π T , π E , ˆ Q 1 , and ˆ Q 2 .
Finally, soft update of target critics ˆ Q (cid:48) 1 , ˆ Q (cid:48) 2 , and ν (cid:48) is performed
Experimental Results
We validate the proposed algorithm from three perspectives.
First, we compare our algorithm with
SOTA off-policy actor-critic algorithms on challenging control tasks.
Second, we examine the quality of our learned correction ratio by evaluating whether it helps to provide a reasonable estimate of the on-policy rewards.
Third, we ablate key components of our algorithms to evaluate their contributions to the overall performance.
We evaluate on 5 continuous control tasks from the MuJoCo
[Todorov et al., 2012] benchmark and compare it with SAC [Haarnoja et al., 2018] and OAC [Ciosek et al., 2019].
We further implement a strong baseline OAC + DICE by integrating our DICE correction into OAC, i.e., similar to (5) and (7), using the learned distribution correction ratio to correct the sample distribution for both the target policy training and the critics training.
For each task, we run 5 different random seeds for each algorithm.
We evaluate each algorithm every 1000 training steps.
Our algorithm introduces additional hyper-parameters (HP), including β LB and β UB that control the extent of conservativeness and optimism respectively, the temperature T from (7), and the set of hyper-parameters from the DICE optimization (9), (10), (11).
We provide detailed hyper-parameter tuning procedures in the Appendix C. Note that the implementation of our algorithms are based on OAC’s ofﬁcially released codes 1 , which themselves are based on the
SAC implementation in RLkit 2 .
While we are aware of different implementations of SAC with performance variance on different tasks, given the difﬁculty in reproducing the best RL results in practice [Engstrom et al., 2020, Pineau et al., 2021], we have tried our best to make all comparisons fair and reproducible by implementing all methods and variants in a uniﬁed and credible codebase.
4.1 Evaluation on Continuous Control Tasks
As shown in Figure 2, our method consistently outperforms both OAC and SAC on 4 out of thetasks, while being on par with OAC on the less challenging HalfCheetah.
The superior performance of our method vs OAC and SAC demonstrates the effectiveness of the proposed framework and the beneﬁts of both the optimistic exploration and distribution correction for off-policy RL.
We would also like to highlight the performance of our speciﬁc baseline OAC + DICE.
While it outperforms OAC by a large margin on Hopper, there are only slight improvements over OAC on
Humanoid and Ant, even with no performance gain on Walker2d and HalfCheetah.
These results demonstrate that the proposed distribution correction scheme at least does not harm the performance and often leads to improvements for off-policy actor-critic algorithms beyond ours.
Moreover, the performance of our method vs. OAC + DICE on the more challenging Humanoid, Ant, and Walker2d demonstrates that our exploration policy design is superior to that of OAC when paired with the
DICE distribution correction.
This is not surprising.
Due to the use of KL constraint in OAC, the actions generated by π E can not differ much from those generated by π T for most of the states.
As a result, the potential of π E in collecting the most informative experience for correcting the critics has been largely constrained.
Our method, on the other hand, does not impose any constraints on the distance between π E and π T , and therefore learns a more effective exploration policy for collecting the informative experience for off-policy training.
Similarly, we implement SAC + DICE by integrating our DICE correction ratio into SAC without any HP tuning.
As shown in Figure 3, SAC + DICE outperforms SAC on the challenging Humanoid and Walker2d by a clear margin while being on par with SAC on the Ant.
There is also a slight performance gain on the Hopper.
It is worth noting that integrating our learned DICE correction ratio into NO DICE (a variant of ours), OAC, and SAC all results in a performance gain.
We thus highlight that our proposed DICE correction scheme effectively beneﬁts general off-policy actor-critic training.
4.2 Examine the Correction Ratio
To examine the quality of the learned distribution correction ratio, we propose to tackle a novel
OPE setting that is more challenging than the traditional counterpart, where both the target policy and the replay buffer are changing.
Following our Algorithm 1 and the same experiment settings in Section 4.1, we now use the following dual estimator deﬁned in [Yang et al., 2020] to estimate the on-policy average per-step rewards r π T on (on-policy rewards) of π T , [Equation]
[Equation]
(13) where ζ is the distribution correction ratio estimated in Algorithm 1.
We use the metric | ˆ µ ζ ( π T ) − r π T on | to evaluate the OPE performance, which directly reﬂects the quality of ζ ( s, a ) .
If | ˆ µ ζ ( π ) − r π T on | < | r D − r π T on | , where r D is the mean reward in the current replay buffer D , correcting the training distribution of RL using the learned ζ can at least provide positive momentum to the overall training.
Figure 4 shows the performance of two OPE methods, i.e., “Our dual estimate" (13) and the naive “Batch rewards" baseline, both are estimated using a batch of transitions.
We also plot the “on policy rewards" r π T on as a reference which averages over the rewards collected by the target policy during evaluation.
As shown in Figure 4, our OPE performance is surprisingly decent.
The ˆ µ ζ ( π ) learned by our method nicely matches the on-policy reward estimate on Humanoid, Ant, and HalfCheetah, while staying close to the on-policy reward estimate on Hopper and Walker2d.
These experimental results demonstrate the effectiveness of our adaptation of the DICE scheme from standard OPE to the off-policy RL setting, which also explains the success of our proposed algorithm in experiments of
Section 4.1 where the power of this adapted DICE scheme has been leveraged.
4.3 Ablation Study
We emphasize the key of our framework is to explicitly correct the RL training distribution while training an explore policy solely to correct the ﬂawed Q value that undermines the target policy’s gradient estimate.
In Section 4.1, we already show that having such an exploration is beneﬁcial.
We now show by ablation that such an aggressive training strategy is indeed backed by our distribution correction techniques.
The ﬁrst ablated version of our method, No DICE , does not have the DICE component, i.e., its training procedures can be summarized as Algorithm 1 without Line 10-12, and always set ˜ ζ
( s, a )
= 1 .
As shown in Figure 5, the performance drops substantially on Humanoid,
Walker2d, and Hopper, and somewhat on Ant and HalfCheetah, demonstrating the necessity of performing an effective distribution correction when applying an aggressive exploration strategy.
The second ablated version is to check the necessity of correcting the state-action distribution w.r.t the Q function loss.
We term the version of our method without applying the correction ratio to the
Q function objective as Only weight policies .
From Figure 5, we can see the performance dropped dramatically on Ant and slightly on the other tasks.
We include more ablation studies in the Appendix, showing how different HP inﬂuences the performance of our method.
Conclusion
In this paper, we proposed a novel off-policy RL paradigm that enables the training of an exploration policy that solely focuses on executing the most informative actions without constraining its divergence from the target policy.
We support such an efﬁcient exploration by learning the distribution correction ratio to correct the training distribution, which is realized by successfully adapting the DICE [Yang et al., 2020] optimization schemes into the off-policy actor-critic framework.
By evaluating in challenging continuous control tasks, we demonstrate the effectiveness of our method.
Appendix Outline of the Appendix
In this Appendix, we organize the content in the following ways:
• In Appendix A, we additional related works.
• In Appendix B, we present the proof of Proposition 3.1.
• In Appendix C, we present experiment details and hyper-parameter settings • In Appendix D, we present additional ablation studies to examine how the hyper-parameters β UB , β LB and T impacts the performance.
We also show the necessity of correcting the distribution of both the target and exploration policy objectives via comparing with another variant of our approach that only weights the Q function training.
A Additional Related Work
Incorporating the DICE family [Nachum et al., 2019a, Zhang et al., 2020a,b, Uehara et al., 2020, Nachum and Dai, 2020, Nachum et al., 2019b, Yang et al., 2020] into the training process of offpolicy actor-critic is never easy, as the DICE optimization itself already poses great optimization difﬁculties.
Moreover, in RL, the target policy is always changing and remains unstable.
When tackling complicated continuous control tasks with high dimensional state-action space, the instability gets even exacerbated by the limited state-action coverage at the initial training stage [Nachum et al., 2019a].
We tackle such difﬁculties by directly making modiﬁcations to the objectives and successfully stabilize the training.
Though suffering from bias, experimental results show that our optimization objective actually leads to surprisingly good OPE performance.
In our work, by explicitly correcting the training distribution, we open up the direction to a potentially more profound exploration strategy design.
There are a bunch of literature investigate to design better exploration objective [Brafman and Tennenholtz, 2002, Kearns and Koller, 1999, Sorg et al., 2012, Lopes et al., 2012, Araya et al., 2012, Doshi-Velez et al., 2010].
While most of the pioneering works mainly tackle the problem in discrete or small state-action space via directly operate on the transition matrix, which cannot be directly extended to complicated continuous control tasks.
Noise-based methods [Fortunato et al., 2017, Plappert et al., 2017] improve exploration by perturbing the observation, state, or the parameters in the function approximator for obtaining diverse actions.
Intrinsic motivation has also been introduced to encourage the agent to discover novel states [Pathak et al., 2017, Burda et al., 2018a, Stadie et al., 2015, Tang et al., 2017, Pathak et al., 2019] or better understand the environment [Houthooft et al., 2016, Zhao and Tresp, 2019, Burda et al., 2018b, Savinov et al., 2018, Choshen et al., 2018, Denil et al., 2016].
Inspired by Thompson sampling, Osband et al.
[Osband et al., 2016] propose to improve exploration of DQN [Mnih et al., 2013] by modeling uncertainty.
Eysenbach et al.
[Eysenbach et al., 2018] propose to encourage exploration through maximizing the diversity of the learned behavior.
SAC [Haarnoja et al., 2018] tries to improve exploration through adding extra regularization on the entropy term, while OAC [Ciosek et al., 2019] derives an upper conﬁdence bound on the state-action value function for facilitating exploration.
Although these methods demonstrate that good exploration will beneﬁt the learning process, these works neglect the issue of distribution shift and instability, which is also proven to be a non-trival problem in reinforcement learning [Kumar et al., 2020, Sinha et al., 2020].
B Proof of Proposition 3.
Proposition 3.1.
∇ θ T ˆ J π T gives an unbiased estimate of the policy gradient (2) with π = π T .
Proof.
Recall that
[Equation]
Therefore
[Equation]
∇ θ T
ˆ J π T is an unbiased estimator for
[Equation]
dads
[Equation]
The implementation of our method is based on OAC’s ofﬁcially released codes 3 , which themselves are based on the SAC implementation in RLkit 4 .
We are aware of the fact that OAC’s ofﬁcially released codes 3 use a set of HP that is slightly different from its original paper.
Speciﬁcally, they set β LB = 1 . 0 instead of 3 .
65 as per (Table 1, Appendix E) of OAC’s original paper [Ciosek et al., 2019].
The implementation to solve the DICE optimization is based on the ofﬁcial dice_rl repo 5 .
We set the regularization function g 1 ( x ) =
g 2 ( x ) = 1 m | x | m in Equation (10) and (11), where we treat the exponential p as a hyper-parameter.
All of our results are obtained by averaging across 5 ﬁxed random seeds as per Appendix D of OAC’s original paper [Ciosek et al., 2019].
We list the HP speciﬁc to our method and those different from SAC and OAC in Table 1.
For completeness, we include the HP for SAC and OAC 3 in Table 2.
We set both α ν = 1 and α ζ =as we found this is the only setting to stabilize the DICE optimization in our HP space.
We set the temperature T = 3 .
as our preliminary experiment ﬁnds the value of T from our HP space will not signiﬁcantly impact the performance, which is proved in our ablation study (Appendix D.1).
As for β UB and β LB , we tune their values simultaneously.
However, we do not conduct the full sweep in the
HP space due to limited computational resources.
We empirically ﬁnd it beneﬁcial to set a similar value for β UB and β LB (Appendix D.2). [Table] [Table] Does
[Table]
tune
[Table]
[Table]
[Table]
[Table]
[Table]
Does
[Table]
tune
[Table]
Does
[Table]
tune
[Table]
[Table]
Does
[Table]
tune
[Table]
[Kingma and Ba, 2014]
Does
[Table] tune [Table] ReLU
Does
[Table]
tune
[Table]
https https
[Table]
[Table]
D Additional Ablation Study D.1 Ablation study of the temperature T
In this section, we examine how the temperature T inﬂuences the performance.
Note that we keep the other HP unchanged as per Table 1.
For T ∈ { 2 . 0 , 3 . 0 , 5 . 0 } , we present the performance of Our
Method on 4 MuJoCo tasks as shown in Figure 6 .
A low value of T will introduce instability to the training process, especially for Ant and Walker2d, while the performance on Humanoid remains stable w.r.t different T .
With T ≥ 3 . 0 , we can achieve a stable training process and decent ﬁnal performance in all 4 tasks.
In this section, we examine how different values of β UB and β LB inﬂuence the performance of our method.
We empirically ﬁnd that setting similar values to β UB and β LB generally lead to good performance.
Note that the performance of β UB = 1 . 5 , β LB = 1 . 5 , and T = 2 . 5 is to show that Our Method is robust to a wide range of β UB and β LB beyonds { 2 . 0 , 2 . 5 } .
Similar to Only weight policies , we construct the ablated version of our method Only weight Q by removing the distribution correction applied to the target policy’s objective (5) and exploration policy’s objective (6).
As shown in Figure 8, ablating the distribution correction for both target and exploration policies of Our Method leads to a clear performance loss on Humanoid.
And the ablated version of Our Method also converges slower on the Hopper.
Therefore, we can conclude that it is necessary to correct the training distributions of the target and exploration policy.