OVD-E XPLORER : A G ENERAL I NFORMATIONTHEORETIC E XPLORATION A PPROACH FOR R EINFORCEMENT L EARNING
A BSTRACT
Many exploration strategies are built upon the optimism in the face of the uncertainty (OFU) principle for reinforcement learning.
However, without considering the aleatoric uncertainty, existing methods may over-explore the state-action pairs with large randomness and hence are non-robust.
In this paper, we explicitly capture the aleatoric uncertainty from a distributional perspective and propose an information-theoretic exploration method named Optimistic Value Distribution
Explorer (OVD-Explorer).
OVD-Explorer follows the OFU principle, but more importantly, it avoids exploring the areas with high aleatoric uncertainty through maximizing the mutual information between policy and the upper bounds of policy
’s returns.
Furthermore, to make OVD-Explorer tractable for continuous RL, we derive a closed form solution, and integrate it with SAC, which, to our knowledge, for the first time alleviates the negative impact on exploration caused by aleatoric uncertainty for continuous RL.
Empirical evaluations on the commonly used Mujoco benchmark and a novel GridChaos task demonstrate that OVDExplorer can alleviate over-exploration and outperform state-of-the-art methods.
I NTRODUCTION
The exploration and exploitation trade-off is critical in reinforcement learning (Thompson, 1933;
Sutton & Barto, 2018).
Many exploration strategies have been proposed in the literature (Osband et al., 2016;
Lillicrap et al., 2016; Martin et al., 2017; Pathak et al., 2017; Ciosek et al., 2019).
Among these strategies, those following the Optimism in the Face of Uncertainty (OFU) principle (Auer et al., 2002) provide efficient guidance for exploration (Chen et al., 2017; Ciosek et al., 2019).
Generally, OFU-based methods regard the uncertainty as the result of insufficient exploration for the state-action pair, and refer to such uncertainty as epistemic uncertainty .
Besides, there is another uncertainty called aleatoric uncertainty , which captures environmental stochasticity:
• Epistemic uncertainty (a.k.a.
parametric uncertainty) represents the ambiguity of the model arisen from insufficient knowledge, and is high at those state-action pairs seldom visited (Dearden et al., 1998;
Osband et al., 2016; Moerland et al., 2017).
• Aleatoric uncertainty (a.k.a. intrinsic uncertainty) is the variation arisen from environment randomness, caused by the stochasticity of policy, reward and/or transition probability, and is characterized by return distribution.
(Bellemare et al., 2017; Moerland et al., 2017).
In the heteroscedastic stochastic tasks where the environment randomness of different state-action pairs differs, those OFU-based methods may be inefficient without properly tackling aleatoric uncertainty (Nikolov et al., 2019).
As shown in Figure 1, if the epistemic uncertainty estimation is disturbed due to the volatility caused by aleatoric uncertainty, the exploration strategy (i.e., policy
A) optimistic about such estimated uncertainty may lead to explore areas with high aleatoric uncertainty.
Visiting such areas results in unstable and risky transitions.
This issue, to explore overly the state-action pairs visited frequently but with high aleatoric uncertainty, is referred to as the overexploration issue .
Intuitively, such issue could be solved by avoiding exploring optimistically about aleatoric uncertainty (i.e., policy B in Figure 1).
[Figure]
The similar concern has been raised in the discrete
RL, where Nikolov et al.
(2019) proposes to use the Information-Directed Sampling (IDS) to avoid the over-exploration issue in the environments with heteroscedastic noise.
However, IDS needs to calculate information-regret ratio for each action, thus applying IDS for continuous RL with explosive or even infinite action space could be non-trivial and ineffective.
Meanwhile, many advanced continuous RL approaches suffer from the over-exploration issue.
Soft
Actor-Critic (SAC) is a well performed continuous RL algorithm, but does not account for efficient exploration beyond maximizing policy entropy (Haarnoja et al., 2018).
OAC improves the exploration following OFU principle, but it ignores to avoid exploration towards higher aleatoric uncertainty (Ciosek et al., 2019).
Therefore, an effective exploration method to address the over-exploration issue for continuous RL is of urgent need.
To address the over-exploration issue, we propose that the characterization of aleatoric uncertainty and a new exploration principle are necessary.
The new principle need to properly trade-off between the epistemic and aleatoric uncertainties, so as to explore seldom visited state-action pairs while avoiding trapped in areas with high aleatoric uncertainty.
In this paper, we propose a general information-theoretic exploration approach OVD-Explorer, which enriches the OFU exploration principle with a complement: “avoid areas with high aleatoric uncertainty”.
Furthermore, to guide the exploration following the new principle, OVD-Explorer maximizes the mutual information between policy and corresponding upper bounds.
Notable, the upper bounds represent the best return that policy can reach, formulated using Optimistic Value Distributions (OVD), and the value distribution characterises the aleatoric uncertainty.
From the theoretical derivation, we show that maximizing such mutual information urges the OVD-Explorer to guide exploration towards the areas with higher epistemic uncertainty and alleviate over-exploration issue by avoiding the areas with lower information gain, i.e., the areas that have higher aleatoric uncertainty but low epistemic uncertainty.
To make OVD-Explorer tractable for continuous action space, we derive its closed form, and propose a scheme to incorporate OVD-Explorer with any policy-based RL algorithm.
Practically, we demonstrate the exploration benefits based on SAC.
Evaluations on many challenging continuous
RL tasks, including a toy task GridChaos, tasks in Mujoco as well as their stochastic variants are conducted, and the results verify our analysis.
To the best of our knowledge, OVD-Explorer firstly addresses the negative impact of aleatoric uncertainty for exploration in continuous RL.
P RELIMINARIES 2. D ISTRIBUTIONAL RL
To model the randomness in the observed long-term return, which characterises aleatoric uncertainty, distributional RL methods (Bellemare et al., 2017; Dabney et al., 2018b;a) are used to estimate Q -value distribution rather than expected Q -value
(Mnih et al., 2015).
In our paper, we focus on quantile regression used in QR-DQN (Dabney et al., 2018b), where the distribution of Q -value is represented by the quantile random variable Z .
Z maps the state-action pair to a uniform probability distribution supported on the values at all corresponding quantile fractions.
Based on value distribution estimator parameterized as θ , we denote the value at quantile fraction τ i as ˆ Z i ( s, a, θ ) given state-action pair ( s, a ) .
Similar to the Bellman operator in the traditional Q-Learning (Watkins & Dayan, 1992), the distributional Bellman operator (Bellemare et al., 2017) T πD under policy π is given as:
[Equation]
Notice that T πD operates on random variables, D = denotes that distributions on both sides have equal probability laws.
Based on operator T πD , Dabney et al.
(2018b) propose QR-DQN to train quantile estimations via the quantile regression loss (Koenker & Hallock, 2001), which is denoted as:
[Equation]
where TD error δ i,j = R ( s t , a t )
+ γ
ˆ Z i ( s t +1 , a t +1 ; ¯ θ ) − ˆ Z j ( s t , a t ; θ ) , the quantile Huber loss ρ τ ( u ) =
u ∗ ( τ − 1 u< 0 ) , and ˆ τ i means the quantile midpoints, which is defined as ˆ τ i
=
τ i +1 + τ i.
2. D ISTRIBUTIONAL S OFT A CTOR -C RITIC METHOD
Distributional Soft Actor-Critic (DSAC)
(Ma et al., 2020) seamlessly integrates distributional RL with Soft Actor-Critic (SAC) (Haarnoja et al., 2018).
Basically, based on the Equation 1, the distributional soft Bellman operator T πDS is defined considering the maximum entropy RL as follows:
[Equation]
where a t +1 ∼ π ( ·| s t +1 ) , s t +1 ∼ P ( ·| s t , a t ) .
Then, the quantile regression loss differs from Equation 2 on δ i,j , by extending clipped double Q-Learning (Fujimoto et al., 2018) based on the maximum entropy RL framework, to overcome overestimation of Q value estimation:
[Equation]
where ¯ θ and ¯ ϕ represents their target networks respectively.
The actor is optimized by minimizing the following actor loss, the same as SAC,
[Equation]
where D is the replay buffer, f ( s, ϵ ; ϕ ) means sampling action with re-parameterized policy and ϵ is a noise vector sampled from any fixed distribution, like standard spherical Gaussian, and Q value is the minimum value of the expectation on certain distributions, as
[Equation]
M ETHODOLOGY
We introduce a general exploration method, named OVD-Explorer, to achieve efficient and robust exploration for continuous RL.
Overall, OVD-Explorer estimates the upper bounds of the policy’s return, and uses the bounds as criteria to find a behavior policy for online exploration from the information-theoretic perspective.
Innovatively, to avoid over-exploration, OVD-Explorer particularly takes the aleatoric uncertainty into account, leveraging the optimistic value distribution (OVD) to formulate the upper bounds of policy’s return.
Then, OVD-Explorer derives the exploration policy via maximizing the mutual information between policy and corresponding upper bounds for an optimistic exploration.
Meanwhile, the derived behavior policy avoids being optimistic to aleatoric uncertainty, thus avoiding over-exploration issue.
In the following, we first give the intuition together with the theoretical derivation of OVD-Explorer.
Then we practically describe how we formulate OVD based on uncertainty estimation.
Lastly, analysis are given to further illustrate why OVD-Explorer can explore effectively and robustly.
3. O PTIMISTIC VALUE DISTRIBUTION GUIDED EXPLORATION
Most OFU-based algorithms follow the exploration principle: “select the action that leads to areas with high uncertainty”.
However, this principle is incomplete as the high uncertainty may also be caused by the aleatoric uncertainty, which misleads the exploration direction.
Hence, we complement the principle by introducing a constrain: “Not only select the action that leads to areas with high uncertainty, but also avoid the ones that only lead to the area with high aleatoric uncertainty” .
Notable, this new principle intuitively describes a good exploration ability, and we will illustrate how to quantitatively measure a policy’s exploration ability in the later section.
By following this principle, OVD-Explorer derives the behavior policy for exploration.
Intuitively, each policy π in the policy space Π has different exploration ability (at state s ), denoted by F π ( s ) .
OVD-Explorer aims at finding the exploration policy π E with the best exploration ability by solving the following optimization problem:
[Equation]
We propose to quantitatively measure the policy’s exploration ability from an information-theoretic perspective by measuring the mutual-information of multi-variables as follows:
[Equation]
where π ( ·| s ) is the action random variable, and
¯ Z π ( s, a i ) the random variable describing the upper bounds of return that action a i can reach under policy π at state s .
Here, a i ∈
A denotes any legal action, thus k could be infinite in continuous action space.
The higher F π ( s ) is, the better ability that policy π has in exploring towards higher epistemic uncertainty while simultaneously avoiding higher aleatoric uncertainty, which satisfies the principle we raised.
(see analysis in Section 3.3.)
Now we state the Theorem 1 to measure the mutual information above in continuous action space.
Theorem 1.
The mutual information in Equation 8 at state s can be approximated as:
[Equation]
Φ x ( · ) is the cumulative distribution function (CDF) of random variable x , ¯ z ( s, a ) is the sampled upper bound of return from its distribution ¯ Z π ( s, a ) following policy π , Z π describes the current return distribution of the policy π , and C is a constant (See proof in Appendix B.1).
Theorem 1 reveals that F π ( s ) is only proportional to Φ Z π (¯ z ( s, a )) , by maximizing which the policy’s exploration ability can be improved.
Concretely, given any policy π θ (parameterized by θ ), the derivative ∇ θ Φ Z πθ (¯ z ( s, a )) can be measured, thus gradient ascent can be iteratively performed to derive a better exploration policy.
(The complete closed form solution is given in later Section 4)
Note that, to perform above optimization procedure, we need to formulate two critical components in Φ Z π (¯ z ( s, a )) :
1 the return of the policy Z π ( s, a ) and 2 upper bound of the return of the policy ¯
Z π ( s, a ) .
Therefore, we first details the formulations in Section 3.2, and then analyze why maximizing Φ Z π (¯ z ( s, a )) can derive a better exploration in Section 3.3.
3. F ORMULATING THE RETURN OF POLICY AND CORRESPONDING UPPER BOUND
Now, we introduce the formulation of the return of policy Z π ( s, a ) and corresponding upper bound ¯
Z π ( s, a ) .
As mentioned before, most OFU-based methods neglect the aleatoric uncertainty when formulating the return and upper bound of return (Mavrin et al., 2019;
Chen et al., 2017), resulting in the over-exploration issue.
Therefore, OVD-Explorer particularly takes into account the aleatoric uncertainty, leveraging distributional RL paradigm to characterize aleatoric uncertainty (Bellemare et al., 2017; Dabney et al., 2018b;a).
OVD-Explorer uses two value distribution estimators ˆ Z ( s, a ; θ 1 ) and ˆ Z ( s, a ; θ 2 ) parameterized by θ 1 and θ 2 , as ensemble estimators to formulate ¯ Z π and Z π in different ways.
Unless stated otherwise, the ( s, a ) is omitted hereafter to ease notation.
Formulation of ¯ Z π .
The ¯ Z π denotes the upper bounds of value that policy π can reach via different actions.
We propose Gaussian distribution with optimistic mean value for formulation as follows, and accordingly refer to it as Optimistic Value Distribution (OVD):
[Equation]
where µ ¯ Z ( s, a ) and σ 2 aleatoric ( s, a ) is mean value and variance, respectively.
Notable, Chen et al.
(2017) discovers the optimisticity is beneficial for better estimating the upper bound, which motivates us to optimistically estimate the µ ¯ Z ( s, a ) by considering epistemic uncertainty as follows:
[Equation] [Equation]
[Table]
where µ ( s, a ) represents the Q -value estimation, σ epistemic ( s, a ) is the epistemic uncertainty, β is a hyper-parameter controlling the magnitude of epistemic uncertainty, U is uniform distribution, N is the number of quantiles, and ˆ Z i ( s, a ; θ k ) is the value of the i -th quantile drawn from ˆ Z ( s, a ; θ k ) .
Moreover, the aleatoric uncertainty should be considered as it will affect the policy’s performance, thus we propose to model it as the variance of ¯ Z π .
In practice, the aleatoric uncertainty σ aleatoric ( s, a ) can be captured by value distribution estimators (Clements et al., 2019) as follows:
[Equation]
Leveraging optimistic value estimations (via epistemic uncertainty) together with explicitly modeling aleatoric uncertainty, the upper bound of the value ¯ Z π can be effectively formulated as an optimistic Gaussian distribution.
Such an upper bound value estimation provides a useful guidance for OVD-Explorer to derive the behavior policy for online exploration.
Formulation of Z π .
Z π estimates the value obtained following policy π .
Inspired by TD3 (Fujimoto et al., 2018), to alleviate overestimation, we propose to formulate Z π in a pessimistic way.
In practice, Z π can be measured in two ways.
First, similar to formulating ¯ Z π in Equation 10, Z π can also be formulated as Gaussian distribution as follows:
[Equation]
where µ ( s, a ) , σ aleatoric ( s, a ) and σ epistemic ( s, a ) are the same defined in Equation 11.
Differently, σ epistemic ( s, a ) is subtracted from µ ( s, a ) , which reveals the pessimistic value estimation of Z π .
Another way is to formulate Z π as multivariate uniform distribution as follows:
[Equation]
where each quantile value z πi ( s, a ; θ ) equals to the minimum estimated value among ensemble estimators (i.e., z i ( s, a ; θ k ).
As such, the estimated value of Z π is relative pessimistic.
Adopting pessimistic value estimation avoids overestimation issue, thereby the value distribution
Z π can be more accurately formulated.
Additional, Gaussian distribution helps more when the environment randomness follows a unimodal distribution, and multivariate uniform distribution can be more flexible and suitable for scenarios with multi-modal distributions.
(see more in Section 5.3.)
3. A NALYSIS OF OVD-E XPLORER
This section analyzes how OVD-Explorer optimistically explores the informative areas and avoids the over-exploration issue.
According to Theorem 1, the exploration policy OVD-Explorer derives can maximize F π ( s ) , which is proportional to CDF value Φ Z π (¯ z ( s, a )) .
In the following, Figure 2(a) and 2(b) illustrate such CDF values for different actions (shaded area), and Table 1 shows how uncertainties affect the exploration.
In these cases, the value distribution Z π ( s, a ) is specified as a Gaussian distribution (in Equation 13), and the sampled optimistic value ¯ z ( s, a ) is specified as the mean of OVD µ ¯ Z ( s, a ) (in Equation 11).
Specifically, at state s , we assume that the means of Z π at actions a 1 and a 2 are the same for ease of clarification.
Basically, Figure 2(a) shows OVD-Explorer can achieve more optimistic exploration.
We assume the aleatoric uncertainty at a 1 and a 2 are the same, but epistemic uncertainty is higher at a 1 , causing µ ¯ Z ( s, a 1 ) > µ ¯ Z ( s, a 2 ) .
Thus the CDF value is larger at a 1 , which means the action with higher epistemic uncertainty is preferred.
P r ob a b ilit y D
e
n s it y π 1 ( , )
Z s a π 2 ( , ) Z s a 1 ( , )  Z s a 2 ( , )  Z s a ( , ) zs a (a)
P r ob a b ilit y D
e
n s it y (b)
More crucially, Figure 2(b) shows that OVDExplorer could guide to avoid the area with higher aleatoric uncertainty given the equal optimism.
We assume that the epistemic uncertainty at a 1 and a 2 are the same, causing their optimistic value estimation to be equal to µ ¯ Z ( s, a ) , but aleatoric uncertainty is lower at a 1 , i.e., PDF curve of Z π ( s, a 1 ) is ”thinner and taller”.
Thus the CDF value is larger at a 1 , meaning that the action with lower aleatoric uncertainty is preferred.
In the limit of t → ∞ , the epistemic uncertainty tend to be 0 , and the aleatoric uncertainty estimated in Equation 12 converges to represent the true environment randomness.
This is comparable to the second case (Figure 2(b)), and OVD-Explorer tends to choose the one with lower aleatoric uncertainty, which is a great advantage of our method over other OFU methods.
To summarize, the exploration guided by OVD-Explorer trades off between two criteria: exploring the areas with higher epistemic uncertainty, to ensure exploring optimistically, and avoiding the areas with higher aleatoric uncertainty, to avoid over-exploration caused by high environment randomness.
OVD-E XPLORER FOR MODERN RL ALGORITHMS
For continuous action space, the argmax operator in Equation 7 is intractable.
To address that, in this section, we derive the behavior policy π E in closed form and state the scheme to incorporate it with existing policy-based algorithms.
First, we define the target policy π T as the policy learned by any policy-based algorithm.
To avoid the gap between training data and the policy, we need to constrain the difference between behavior policy π E and target policy π T , thus we derive π E in the vicinity of π T .
Second, to derive exploration π E for modern RL algorithms with stochastic policy based on OVDExplorer, where both the exploration π E = N ( µ E , Σ E ) and target policy π T = N ( µ T , Σ T ) are
Gaussian distributions, we introduce the following proposition:
Proposition 1.
The OVD-Explorer behavior policy π E = N ( µ E , Σ E ) is as follows:
[Equation]
and
[Equation]
In specific, α is the step size controlling the exploration level and m = log Φ Zπ
[Equation]
(see proof in Appendix B.2).
The expectation E ¯ Z π can be estimated by sampling K samples, then Equation 15 is simplifies as:
[Equation]
Algorithm 1 summarizes the overall procedure of OVD-Explorer, including the formulation of ¯ Z π and Z π (Line 2 and line 3) and behavior policy generation (Line 4).
The generated behavior policy can be integrated with any modern policy-based RL algorithms to render the stable and wellperformed algorithm.
Please see Appendix C for the entire pseudo-code and details about how the
OVD-Explorer-based behavior policy is incorporated with DSAC.
E XPERIMENTS
To reveal the consistency between our theoretical analysis and the performance of OVD-Explorer algorithm, we conduct experiments to address the following questions:
5. E XPERIMENT S ETUP
Baselines include SAC (Haarnoja et al., 2018), DSAC (Ma et al., 2020), and DOAC, which is the distributional variant of OAC (Ciosek et al., 2019).
We implement Gaussian and quantile formulations of Z π as in Equation 13 and 14, which are denoted in the following as OVDE G and OVDE Q, respectively.
We test OVD-Explorer on a novel task GridChaos and several tasks in Mujoco (Todorov et al., 2012) including the stochastic variants.
The appendix gives more details, including environment settings, implementation details, hyperparameters settings, and computing infrastructure used, as well as more experiment evaluations.
5. E XPLORATION IN G RID C HAOS (RQ1)
To illustrate the exploration pattern of OVD-Explorer and show the advantage of OVD-Explorer over DSAC and DOAC, we evaluate OVD-Explorer on a novel continuous and stochastic control task called GridChaos.
[Table]
Figure 3(a) shows the map of GridChaos, in which we control the cyan triangle (agent) aiming to reach the fixed dark blue goal.
The state is the current coordinate, and the action is a two-dimensional vector including the movement angle and distance.
One episode terminates when the agent reaches the goal or maximum steps (i.e., MAX STEP, typically 100).
Also, it receives a +100 reward when reaching the goal otherwise 0.
The reason why it is chaos is that the randomness of the transition is heterogeneous in different parts of this environment.
Table 2 shows the environment settings.
Moreover, Figure 3(b) shows that OVDExplorer can reach the goal faster, with more efficient exploration.
Figure 3(c) shows the values of uncertainty estimation and exploration objective (mutual information) taken four different actions at the state shown in Figure 3(a) and at training epoch 1249.
Basically, Figure 3(c) illustrates that the estimated aleatoric uncertainty of left is higher than that of right, indicating that OVD-Explorer models aleatoric uncertainty properly.
Further, OVD-explorer encourages to explore the right side at that time, since the value of exploration objective (in green) is highest.
It implies that OVD-explorer tends to explore areas with higher epistemic uncertainty and avoiding higher aleatoric uncertainty, which is in accordance with our exploration principle.
On the other hand, if high epistemic uncertainty is considered only like in OAC(Ciosek et al., 2019), the agent would be guided to the left.
Then the agent may be trapped in the left side due to the high aleatoric uncertainty, and it is the possible reason about why OAC fails tackling such heteroscedastic
[Figure]
stochastic task.
In appendix, we show more analysis about the exploration pattern, please refer to
Appendix E.5.
Besides, we conduct the evaluation when the randomness is high around the goal, as well as many other noise settings, the results enhance the ability of OVD-Explorer.
More experiments that evaluate the performance on several other noise settings, as well as more detailed value histogram on different epoch in training process can be found in Appendix.
5. P ERFORMANCE ON M UJOCO TASKS (RQ2)
To demonstrate the performance of OVD-Explorer more generally, we evaluate it on several Mujoco tasks.
For 5 standard Mujoco tasks 1 , the transition is deterministic and the randomness is only from the stochastic policy.
For 5 noisy tasks, the heteroscedastic Gaussian noise is added in Mujoco tasks.
Specifically, in each state transition on noisy tasks, Gaussian noise of different scales is randomly injected following a certain probability.
Overall, we have the following findings.
Basically, OVD-Explorer can perform stably in the standard tasks with slight or little randomness, guiding efficient exploration.
From the results 2 in standard Mujoco tasks in table 3, it is shown that in the relatively easy tasks, i.e., Hopper-v2, Reacher-v2, and HalCheetah-v2, OVD-Explorer does not obtain much gain beyond the baselines, which seems that these tasks are not profoundly demanding for exploration.
In the high-dimension tasks, i.e., Ant-v2 as shown in Figure 4(a), OVD-Explorer significantly outperforms DSAC and DOAC.
It means that the aleatoric uncertainty caused by policy indeed degrades the performance of DOAC, where the aleatoric and epistemic uncertainties are not distinguished.
Besides, OVD-Explorer can avoid the impact of heteroscedastic aleatoric uncertainty on exploration and thus improve robustness.
As the results of noisy Mujoco tasks shown in Table 3, DOAC is worse than DSAC in most cases, which means that heteroscedastic aleatoric uncertainty causes significant degrades of DOAC.
Simultaneously, OVD-Explorer significantly outperforms DOAC and DSAC, especially in Noisy Ant-v2 as shown in Figure 4(b).
Furthermore, OVDE G takes the Gaussian prior and concretely obtains better estimation of critic in the noisy tasks, while OVDE Q performs better in the standard tasks due to the flexibility of quantile distribution.
Generally, the difference between OVDE G and OVDE Q is that they use different formulations of Z π ( s, a ) .
Theoretically, the value function distribution in OVDE Q is more flexible and should perform better than OVDE G, as the reported results on five standard Mujoco tasks show.
On the other hand, OVDE G performs better in the noisy tasks, and it is because that the transition probability is Gaussian in the stochastic Mujoco tasks.
Here, the reported results may be slightly different from previously reported results, partly due to the statistic approach, and partly due to the implementation.
Nevertheless, the patterns of these baselines (e.g.,
DSAC outperforms SAC in the vast majority of cases) are consistent with previous results.
The details about the implementation of
[Table]
[Figure]
S ENSITIVITY TO α (RQ3)
The step size α in Equation 15 controls how much the behavior policy derived from OVD-Explorer, i.e. π E , is far away from the target policy π T , which can be essential for exploration benefit.
To investigate the appropriate range of it, we test several α value on Noisy Ant-v2 task using OVDE G, and the result is shown in Figure 4(c).
If α is quite small, OVD-Explorer degenerates to DSAC and implies little exploration.
In contrast, if α is larger, the performance becomes worse because of the huge gap between behavior policy and target policy.
The result demonstrates that α should be taken in a suitable range to facilitate more adequate exploration.
In our experiments, we uniformly use α to be equal to 0.05 to show the performance of the OVD-Explorer.
This paper proposes an information-theoretic exploration method OVD-Explorer, which introduces a novel measurement of exploration ability, i.e., the mutual information between the policy and the upper bounds of return. By maximizing the mutual information, OVD-Explorer is able to derive the behavior policy, that follows the OFU principle, and further avoids exploring the areas with high aleatoric uncertainty. Integrated with SAC, OVD-Explorer addresses the negative impact of aleatoric uncertainty for exploration in continuous RL for the first time.
The work is supported by the National Natural Science Foundation of China (Grant No. 62106172). In addition, we thank Matthew E. Taylor for reviewing an early draft of the paper A R ELATED W ORK
In this work, we consider the exploration strategy under the principle of Optimism in the Face of Uncertainty (OFU) (Auer et al., 2002), especially in the heteroscedastic stochastic environment. We aim to improve exploration efficiency, and alleviate the over exploration issue caused by aleatoric uncertainty.
Basic exploration strategies, like ϵ -greedy (Sutton & Barto, 2018), noise perturbation (Lillicrap et al., 2016), entropy regularization (Mnih et al., 2016) and stochastic policy (Haarnoja et al., 2018), lead to undirected exploration through random perturbations. With the increasing emphasis on exploration efficiency in RL, various exploration methods have been developed. One kind of methods uses intrinsic motivation to stimulate agent to explore, such as count-based novelty (Martin et al., 2017; Ostrovski et al., 2017; Bellemare et al., 2016; Tang et al., 2017; Fox et al., 2018), prediction error (Pathak et al., 2017), reachability (Savinov et al., 2019) and information gain on environment dynamics (Houthooft et al., 2016). Some recent methods, originating from tracking uncertainty, guide efficient exploration under the principle of OFU, such as Thompson Sampling (Thompson, 1933; Osband et al., 2016),
IDS (Nikolov et al., 2019; Clements et al., 2019) and other customized methods (Moerland et al., 2017; Pathak et al., 2019).
The base of OFU methods is to model epistemic and aleatoric uncertainties in RL.
Bootstrapped
DQN (Osband et al., 2016) has become the well-used approach for capturing epistemic uncertainty (Kirschner & Krause, 2018; Ciosek et al., 2019), and distributional RL methods (Bellemare et al., 2017; Zhou et al., 2020; Dabney et al., 2018a;b) are used for capturing aleatoric uncertainty. However, most traditional OFU methods do not distinguish the two types of uncertainty, which can easily lead the naive solution to favor actions with higher variances in stochastic tasks, i.e., over-exploration issue.
To address that, Mavrin et al.
(2019) study how to take advantage of value distribution for efficient exploration under both types of uncertainty, proposing Decaying Left Truncated Variance (DLTV) based on QR-DQN.
Besides, Nikolov et al.
(2019) and Clements et al.
(2019) propose to use Information Direct Sampling (Kirschner & Krause, 2018) for efficient exploration in RL, which formulates epistemic and heteroscedastic aleatoric uncertainty and maximizes information gain on globally optimal action to explore informative state-action pairs.
However, such methods are complicated when deriving a behavior policy and is limited to discrete control.
Meanwhile, there is not any strategy that can help the well-performed continuous RL algorithms (Haarnoja et al., 2018; Ciosek et al., 2019; Ma et al., 2020) to address aleatoric uncertainty when exploration.
OAC (Ciosek et al., 2019) proposes exploration bonus guided by the upper bound of Q estimation to facilitate exploration based on Soft Actor-Critic (SAC) (Haarnoja et al., 2018).
Nevertheless, OAC ignores the potential impact of the aleatoric uncertainty, which may cause misleading exploration.
Our proposed OVD-Explorer is a novel exploration strategy, which can guide agent to explore towards higher epistemic uncertainty, and also avoid the areas with high aleatoric uncertainty, improving the robustness of exploration especially facing heteroscedastic aleatoric uncertainty.
To capture aleatoric uncertainty, OVD-Explorer models the value distribution and uses mutual information to guide exploration following the principle of OFU, measuring the correlations between the policy distribution and upper bounds distribution of return.
There are some other informationtheoretic exploration strategies using mutual information, such as VIME (Houthooft et al., 2016), which measures the information gain on environment dynamics, and EMI (Kim et al., 2019), which generates intrinsic reward using prediction error of representation learned by mutual information.
Those methods can solve sparse reward problem very well by using intrinsic reward.
Nevertheless, those exploration methods use mutual information neither on the value distribution, nor for OFUbased exploration. Besides, unlike the mechanisms used in measuring mutual information, such as variational inference (Hinton & van Camp, 1993; Houthooft et al., 2016) and f-divergence (Nowozin et al., 2016; Kim et al., 2019), we find the correlation between policy and upper bounds of return through uncertainty as shown in Theorem 1, thus we can directly derive the close form exploration policy. B P ROOFS B. P ROOF OF T HEOREM In order to prove the Theorem 1, we first propose the following lemma about F π ( s ) .
Lemma 1.
The mutual information of ¯ Z π ( s, a 0 ) , · · · , ¯ Z π ( s, a k − 1 ) and π ( ·| s ) at state s is:
[Equation]
where p ( a | ¯ z
( s, a ) , s ) represents the posterior probability distribution of policy given current state s and the sampled upper bound of return
¯ z ( s, a ) .
Proof.
For simplicity, we assume that the size of action space is k = 2 , and the actions are denoted as a 0 and a 1 , a 0 ̸ = a 1 .
Then we derive the mutual information among three random variables
MI ( ¯ Z π ( s, a 0 ) , ¯ Z π ( s, a 1 ) , π ( ·| s )) , where the action sampled from π ( ·| s ) is either a 0 or a 1 .
Considering the formula for the mutual information, F π ( s ) is derived as follows:
[Equation]
where the posterior distribution p ( a | ¯ z
( s, a 0 ) , ¯ z ( s, a 1 )) is the probability of choosing action a on the condition of the samples from upper bounds of action a 0 and a 1 .
Considering that in the decision-making process, the probability of action a 0 is independent to the upper bound of other actions, such as ¯ z ( s, a 1 ) , which means that p ( a 0 | ¯ z
( s, a 0 ) , ¯ z ( s, a 1 )) =
p ( a 0 | ¯ z
( s, a 0 )) .
Therefore, the above equation can be further reduced as follows.
[Equation]
It is easy to extend to infinite-action case.
Based on that for any action a i ∈ [ a 0 , a k − 1 ] and k → ∞ , the conditional probability p ( a i | ¯ z
( s, a 0 ) , ¯ z ( s, a 1 ) , ..., ¯ z
( s, a k − 1 ))
= p ( a i | ¯ z
( s, a i )) , F π ( ·| s ) can be simplified as following,
[Equation]
Lemma 1 tells that the mutual information F π ( s t ) is in direct proportion to p ( a |
¯ z ( s, a ) , s ) , which measures how much it is worth acting under the current policy π ( a | s ) when the upper bound is known.
Next, to measure the posterior probability p ( a | ¯ z ( s, a ) , s ) , we use a general and practically effective approach (Wang & Jegelka, 2017; Belakaria et al., 2020; Perrone et al., 2019;
Li et al., 2020) of approximating the posterior probability given upper bound value.
Specifically, we approximate p ( a | ¯
z
( s, a ) , s ) using the prior that z π ( s, a ) ≤ ¯ z ( s, a ) with given policy π ( s, a ) , since ¯ z ( s, a ) is the upper bound of z π ( s, a ) .
Hence, we use the indicator function 1 z π ( s,a ) ≤ ¯ z
( s,a ) to truncate the policy π ( s, a ) , and utilize the constant C to normalize the probability, as is shown in the following equation.
[Equation]
Here, E z π ( s,a ) ∼ Z
[Equation]
( s,
[Equation]
(cid:2) 1 z π ( s,a ) ≤ ¯ z
( s,
[Equation]
) (cid:3)
=
Φ Z π ( s
[Equation]
) (¯ z ( s, a )) , where Φ x is the cumulative distribution function (CDF) of x , ¯ Z π
and Z π are the random variables, whose distributions describe the randomness of the returns, and ¯ z ( s, a ) is the value of random variable ¯ Z π .
Therefore, the posterior probability can be measured as follows,
[Equation]
In our method, we do not use the commonly used mechanisms about mutual information such as neural network estimation (Belghazi et al., 2018) and upper bound estimation (Cheng et al., 2020).
Instead, we can find the correlation between random variables as shown in Equation 20, which helps to derive mutual information directly
[Equation]
According to Lemma 1 and Equation
[Equation]
we can give the proof of Theorem 1 in the following.
Proof.
By Combining Lemma 1 and Equation 20, F π ( s ) can be further derived as follows.
[Equation]
Here
[Equation]
the last equality follows from Theorem
[Equation]
B. P ROOF OF PROPOSITION
Proof.
Similar to (Ciosek et al., 2019), we set the covariance matrix of behavior policy π E is that of target policy π T , i.e., Σ E = Σ T .
Hence, the OVD-Explorer problem is simplified as:
[Equation]
To ensure that the behavior policy samples actions around the target policy, we derive the π E upon mean µ T of target policy π T .
In specific, we firstly obtain the gradient of ˆF ( s, µ ) at π T , which is given as follows:
[Equation]
1: Initialise: Value networks θ 1 , θ 2 , policy network ϕ and their networks ¯ θ 1 , ¯ θ number N, smoothing coefficient
D 2: for each iteration do 3: for each environmental step do 4: a t ∼ π E ( a t , s t ) according to Algorithm5:
D ← D ∪ { ( s t , a t , r ( s t , a t ) , s t +1 ) } 6: end for 7: for each training step do 8: for i = 1 to N do 9: for j = 1 to N do 10: calculate δ ki,j , k = 1 , 2 , following Eq.11: end for 12: end for 13:
Calculate L QR ( θ k ) , k = 1 , 2 using δ ki,j following Eq.14:
Update θ k with ∇L QR ( θ k ) 15:
Calculate J π ( ϕ ) , following Eq.16:
Update ϕ with ∇J π ( ϕ ) 17: end for 18:
Update target value network with ¯ θ k ← τθ k + (1 − τ )¯ θ k , k = 1 ,19:
Update target policy network with ¯ ϕ ← τϕ + (1 − τ )¯ ϕ 20: end for where ˆ m is given as:
[Equation]
and ϕ ( x ) is the probability distribution function (pdf).
Hence, the µ E is given as follows:
[Equation]
where α is the step size controlling exploration level and m = log Φ Zπ ( s,µT
[Equation]
(¯ z ( s,µ T )) C
+ 1 .
C A LGORITHM 2: OVD-E XPLORER FOR DSAC
In this section, we show the whole algorithm of our implementation of OVD-Explorer based on
DSAC in Algorithm 2.
All the code can be found in the supplementary material.
D M ORE DETAILS ABOUT THE EXPERIMENTS D. G RID C HAOS
GridChaos is an environment built on OpenAI’s
Gym toolkit, whose map is shown as in Figure 3(a) and Figure 5.
In this section we illustrate more details in addition to Section 5.2.
The movable cyan triangle and the fixed symmetric dark blue goal are two parts split from square, and the goal is to make the triangle embedded in the goal to recover the original square, which is to say that it is an isosceles triangle whose base side is equal to the height.
The triangle is always initialised randomly in the cyan rectangle, and the black line in the map represents the wall, where the triangle will be
[Table]
adsorbed once hits the wall.
The state transition is stochastic, and we add Gaussian noise to the action resulting in Gaussian transition probability.
To represent the location of the triangle, we establish a Cartesian coordinate system using the centroid of the map as the origin, as shown in Figure 5.
Then the coordinates of the triangle are represented by the midpoint of the altitude of the triangle which is shown as the red point in the triangle.
In the case shown in Figure 5, the initial coordinate of the triangle (agent) is in the negative half of the x-axis and the task target is in the first quadrant.
D. B ASELINES
Ma et al.
(2020) shows the performance of DSAC and TD4, which is the distributional extension of TD3 (Fujimoto et al., 2018), and can also be used to capture epistemic and aleatoric uncertainty.
Moreover, DSAC outperforms TD4 on Mujoco tasks as shown in Ma et al.
(2020), so we evaluate only on SAC and DSAC, and further implement OVD-Explorer based on DSAC to develop the exploration ability.
SAC.
The SAC (Haarnoja et al., 2018) implementation is mainly based on OAC repository, and the results in Ant-v2 and Hopper-v2 are similar to reported results by OAC.
Our SAC report a better result than OAC’s implementation for SAC on HalfCheetah-v2, which is because the high variance of this environment as explained as in OAC.
DSAC.
The DSAC (Ma et al., 2020) implementation is based on our implementation of SAC, except that the distributional Q function in the DSAC repository is used instead of the traditional Q function in SAC.
As it is based on SAC, we set the hyper-parameters of DSAC to be consistent with SAC to ensure the fair comparison, which also results in the different reported results from original paper of
DSAC.
In our results, DSAC can guarantee an absolute advantage over SAC in most cases, which is consistent with the previous conclusion.
DOAC.
The DOAC implementation is mainly based on our implementation of DSAC as well as the open source code of OAC.
As DSAC shows great advantage due to the distributional value estimation, to ensure a fair comparison, we extend OAC (Ciosek et al., 2019) to its distributional version, i.e., DOAC, by replacing the exploration process of DSAC by the behavior policy derived by OAC.
We set the hyper-parameters the same as used by OAC in Mujoco 3 , and our results of
DOAC on Ant-v2 and HalfCheetah-v2 are significantly better than that OAC reported.
D. I MPLEMENTATION
Our implementation of OVD-Explorer is based on the open source code of OAC 4 , also refer to the code of DSAC 5 as well as softlearning 6 .
{+All experiments are performed on NVIDIA GeForce RTX 
+}
Ti 11GB graphics card.
The training process of OVD-Explorer and DOAC are the same as in DSAC, except for the different behavior policy used, while OVD-Explorer and DOAC enrich the experience replay with the data using the the derived exploration policies, respectively.
To ensure the fair comparison, the hyperparameters for training process of baselines and OVD-Explorer are the same.
Besides, we have three hyper-parameters associated with OVD-Explorer as mentioned before, including α that controls the exploration level, β that determines the magnitude of uncertainty we use, as well as C that is the normalization factor.
The hyper-parameters in our experiments are shown in Table 4.
E M ORE E XPERIMENT S TUDY E. R UNTIME ANALYSIS
[Figure]
Figure 6 shows the time consumption of algorithms relative to SAC.
As can be seen, the distributional value estimation used in
DSAC, DOAC and our methods introduces extra time consumption distinctly.
Nevertheless, the relative time consumption of OVDE G and
OVDE Q to SAC is 1.21 and 1.17, respectively, which means that OVD-Explorer spends about 20% more time than SAC to achieve up to nearly 100% performance gain as shown in Figure 6(b).
This demonstrates the extra time consumption is well worth it.
Besides, the time consumption of OVDE Q is close to that of
DSAC, only with a larger variance, which indicates that the additional time consumption of
OVDE Q is minimal while performing better exploration.
E. E VALUATION ON SEVERAL OTHER NOISE SCALE IN G RID C HAOS
OVD-Explorer can explore efficiently in heteroscedastic stochastic environment by considering differently about epistemic and aleatoric uncertainty for exploration as shown in Section 5.2.
To further empirically prove its strength, we test OVD-Explorer in GridChaos with different noise scales in four quadrants, and the result is shown as Figure 7 and Table 5.
We can find that OVD-Explorer can perform well in all those different noise injection of environment.
Concretely, from the results, the following observations deserve to be noticed.
First, OVD-Explorer can significantly achieve better average return in all those settings, especially when the noise is set high, and can learn to reach the goal faster (see column about FRG).
It shows the ability of OVDExplorer to guide agent explore against higher aleatoric uncertainty on the left side (the second and third quadrants).
Second, for the task without noise as shown in row E , which means the state transition is deterministic, OVD-Explorer still learns quickly.
The results in row E show the inherently high difficulty of this task, not only because of the very sparse reward, but also the gate leading to the goal is set very small (the width of the gate is only 30% of the length of agent, i.e., the base of the isosceles triangle, which means that at the doorway the agent can only move a very small distance horizontally, otherwise it would be adsorbed to the wall and immobile).
[Figure]
[Figure]
[Table]
In the previous experiments in GridChaos [Table] the noise (i
[Table]
e., aleatoric uncertainty) near the goal is set lower.
In such situation
[Table]
OVD-Explorer, which follows the principle of OFU and further avoids exploring areas with higher aleatoric uncertainty, could bring significant advantage.
Such setup of heterogeneous noise is reasonable, because in real life, the goal or optimal policy is always not expected to be highly stochastic.
Nevertheless, the evaluation about tasks with the existence of high randomness in the target region is valuable, so we conducted the following experiment in GridChaos, where the environment randomness in the right half (first and fourth quadrants), where the target is located, was set larger.
The results are shown in Table 6.
Note that we use OVDE(P) to denote the usual implementation that pessimistically estimates the value distribution (i.e., using Equation 13).
Besides, OVDE(M) denotes the implementation that does not pessimistically estimate the value distribution (i.e., we modify the mean of Gaussian distribution Z π in Equation 13 from the lower bound to expected value of the Q estimation µ ( s, a ) as in Equation 11.).
Our experimental findings are mainly the following three aspects.
Firstly, when facing extremely high aleatoric uncertainty around the goal (see row F), which causes the interaction around goal to be very unstable, chaotic and disorder, OVD-Explorer would strongly discourage exploring such a area, and thus performance would be damaged.
In contrast, DSAC and
DOAC have no restriction on aleatoric uncertainty, and high randomness may instead increase the probability of achieving the goal.
Second, in most cases (see G, H, I, J), OVD-Explorer always can guide better exploration and achieve better performance than DSAC and DOAC, especially when the noise is negligible (see row
J).
This reflects the fact that our exploration objective (the mutual information shown in Equation 8)
[Table]
makes great sense, achieving an appropriate trade-off between avoiding high aleatoric uncertainty and being optimistic about high epistemic uncertainty.
Third, an interesting finding is that OVD-Explorer may perform better by turning off the pessimistic estimation facing higher aleatoric uncertainty around the goal (see column OVDE(M)).
This suggests that excessive pessimism is unnecessary if there is a need to explore areas with high aleatoric uncertainty.
Overall, from the results in Table 5 and Table
[Table]
OVD-Explorer is able to tackle most of the cases quite well.
When there is high randomness around the goal, OVD-Explorer has a shortcoming that it will inevitably slow down the efficiency of reaching the goal, because it limit the exploration towards such area.
Fortunately, this shortcoming can be mitigated by turning off the pessimistic estimation.
E. A BLATION STUDY ON VALUE DISTRIBUTION ESTIMATION
As mentioned in Section 3.2, we estimate Z π pessimistically to alleviate over-estimation.
Also, as mentioned in Appendix E.3, the pessimism is unnecessary if there is a need to explore areas with high aleatoric uncertainty.
In the following, to investigate the benefit of pessimistic estimation in general case, we compare the performance of OVD-Explorer to the modified versions that use normally estimated Z π .
Our results show that pessimistic estimation can mostly be better than that using normal estimation.
[Figure]
For OVDE G (mean), we modified the mean of Gaussian distribution Z π in Equation 13 from the lower bound to average value of the Q estimation µ ( s, a ) as in Equation 11.
For OVDE Q (mean), we modified the z
πi ( s, a ) in Equation 14 from the minimum value to the average value, i.e., z πi ( s, a ; θ ) =
E k =1 , 2 ˆ Z i ( s, a ; θ k ) .
As shown in Figure 8, both OVDE G (mean) and OVDE Q (mean) perform worse than the pessimistic version.
To draw a conclusion, pessimistic estimate is indeed required in general cases.
Only when there is a need to explore areas with high aleatoric uncertainty, is such pessimistic estimation worth being turned off.
E. A NALYSIS ABOUT EXPLORATION PROCESS OF OVD-E XPLORER
In the following, we further verify from statistical analysis that OVD-Explorer is in compliance with our raised exploration principle, i.e., OVD-Explorer can achieve better trade-off between optimistic exploration and effectively avoiding exploring the areas with high aleatoric uncertainty .
We show the values of uncertainty estimations and our exploration objective (mutual information) at different stages during the training processes of two trials with different noise settings in figures.
In Figure 9, the environment noise is set lower around the goal, noting that the darker background color in the map represents higher aleatoric uncertainty, and the red dot represents the coordinate of the current state.
The performance under this trail is given and the agent hardly ever reaches the goal before the 1000th epoch.
Therefore, in the early stage, the aleatoric uncertainty is inaccurate and remains very low, as the value distribution shows little divergence.
The figure also shows that our exploration objective (in green) is high when the epistemic uncertainty is high.
So before the 1000th epoch, the exploration is guided by epistemic uncertainty, which follows the OFU principle.
Later, once the goal has been explored, the aleatoric uncertainty is properly modelled, i.e., the aleatoric uncertainty towards left is larger than right at current state (see epoch 1240).
Then the mutual information value towards left is lower than right, although the epistemic towards left is higher.
It indicates that OVD-Explorer can property balance epistemic uncertainty and aleatoric uncertainty, and effectively avoid to explore the areas with higher aleatoric uncertainty.
In Figure 10, the environment noise is set higher around the goal.
At the stage before the 1000th epoch, the exploration guidance is similar to Figure 9.
The agent would hardly estimate the accurate aleatoric uncertainty without obtaining any reward.
In the later stage, at the 1240th epoch, OVDExplorer suggests exploring to the right, even though it has been recognized that the environmental uncertainty on the right is high.
This is because the epistemic unertainty dominates under the mutual information.
In contrast, at the 1249th epoch, when the action towards right has been explored much, the significant higher aleatoric uncertainty towards right dominates.
Therefore, following the mutual information, the action towards left is preferred.
This demonstrates the trade off that OVD-Explorer make, which satisfies our raised exploration principle.
[Figure]
7\SH (SLVWHPLFXQFHUWDLQW\
$OHDWRULFXQFHUWDLQW\ 0XWXDOLQIRUPDWLRQ
[Figure]
[Figure]