OVD-E XPLORER : A G ENERAL I NFORMATIONTHEORETIC E XPLORATION A PPROACH FOR R EIN -
FORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many exploration strategies are built upon the optimism in the face of the uncertainty (OFU) principle for reinforcement learning.
However, without considering the aleatoric uncertainty, existing methods may over-explore the state-action pairs with large randomness and hence are non-robust.
In this paper, we explicitly capture the aleatoric uncertainty from a distributional perspective and propose an information-theoretic exploration method named Optimistic Value Distribution
Explorer (OVD-Explorer).
OVD-Explorer follows the OFU principle, but more importantly, it avoids exploring the areas with high aleatoric uncertainty through maximizing the mutual information between policy and the upper bounds of policy
’s returns.
Furthermore, to make OVD-Explorer tractable for continuous RL, we derive a closed form solution, and integrate it with SAC, which, to our knowledge, for the ﬁrst time alleviates the negative impact on exploration caused by aleatoric uncertainty for continuous RL.
Empirical evaluations on the commonly used Mujoco benchmark and a novel GridChaos task demonstrate that OVDExplorer can alleviate over-exploration and outperform state-of-the-art methods.
I NTRODUCTION
The exploration and exploitation trade-off is critical in reinforcement learning (Thompson, 1933;
Sutton & Barto, 2018).
Many exploration strategies have been proposed in the literature (Osband et al., 2016;
Lillicrap et al., 2016; Martin et al., 2017; Pathak et al., 2017; Ciosek et al., 2019).
Among these strategies, those following the Optimism in the Face of Uncertainty (OFU) principle (Auer et al., 2002) provide efﬁcient guidance for exploration (Chen et al., 2017; Ciosek et al., 2019).
Generally, OFU-based methods regard the uncertainty as the result of insufﬁcient exploration for the state-action pair, and refer to such uncertainty as epistemic uncertainty .
Besides, there is another uncertainty called aleatoric uncertainty , which captures environmental stochasticity:
• Epistemic uncertainty (a.k.a.
parametric uncertainty) represents the ambiguity of the model arisen from insufﬁcient knowledge, and is high at those state-action pairs seldom visited (Dearden et al., 1998;
Osband et al., 2016; Moerland et al., 2017).
• Aleatoric uncertainty (a.k.a. intrinsic uncertainty) is the variation arisen from environment randomness, caused by the stochasticity of policy, reward and/or transition probability, and is characterized by return distribution.
(Bellemare et al., 2017; Moerland et al., 2017).
In the heteroscedastic stochastic tasks where the environment randomness of different state-action pairs differs, those OFU-based methods may be inefﬁcient without properly tackling aleatoric uncertainty (Nikolov et al., 2019).
As shown in Figure 1, if the epistemic uncertainty estimation is disturbed due to the volatility caused by aleatoric uncertainty, the exploration strategy (i.e., policy
A) optimistic about such estimated uncertainty may lead to explore areas with high aleatoric uncertainty.
Visiting such areas results in unstable and risky transitions.
In the real world, the aleatoric uncertainty can be caused easily, for example, unpredictable wind can shift the trajectory of an robot’s action.
If such aleatoric uncertainty is not modelled, the RL agent may be trapped because the state transitions in such area can be wrongly considered novel and worth exploring due to the high uncertainty.
This issue, to explore overly the state-action pairs visited frequently but with high aleatoric uncertainty, is referred to as the over-exploration issue .
Intuitively, such issue could be solved by avoiding exploring optimistically about aleatoric uncertainty (i.e., policy B in Figure 1).
[Figure]
The similar concern has been raised in the discrete
RL, where Nikolov et al.
(2019) proposes to use the Information-Directed Sampling (IDS) to avoid the over-exploration issue in the environments with heteroscedastic noise.
However, IDS needs to calculate information-regret ratio for each action, thus applying IDS for continuous RL with explosive or even inﬁnite action space could be non-trivial and ineffective.
Meanwhile, many advanced continuous RL approaches suffer from the over-exploration issue.
Soft
Actor-Critic (SAC) is a well performed continuous RL algorithm, but does not account for efﬁcient exploration beyond maximizing policy entropy (Haarnoja et al., 2018).
OAC improves the exploration following OFU principle, but it ignores to avoid exploration towards higher aleatoric uncertainty (Ciosek et al., 2019).
Therefore, an effective exploration method to address the over-exploration issue for continuous RL is of urgent need.
To address the over-exploration issue, we propose that the characterization of aleatoric uncertainty and a new exploration principle are necessary.
The new principle need to properly trade-off between the epistemic and aleatoric uncertainties, so as to explore seldom visited state-action pairs while avoiding trapped in areas with high aleatoric uncertainty.
In this paper, we propose a general information-theoretic exploration approach OVD-Explorer, which enriches the OFU exploration principle with a complement: “avoid areas with high aleatoric uncertainty”.
Furthermore, to guide the exploration following the new principle, OVD-Explorer maximizes the mutual information between policy and corresponding upper bounds.
Notable, the upper bounds represent the best return that policy can reach, formulated using Optimistic Value Distributions (OVD), and the value distribution characterises the aleatoric uncertainty.
From the theoretical derivation, we show that maximizing such mutual information urges the OVD-Explorer to guide exploration towards the areas with higher epistemic uncertainty and alleviate over-exploration issue by avoiding the areas with lower information gain, i.e., the areas that have higher aleatoric uncertainty but low epistemic uncertainty.
To make OVD-Explorer tractable for continuous action space, we derive its closed form, and propose a scheme to incorporate OVD-Explorer with any policy-based RL algorithm.
Practically, we demonstrate the exploration beneﬁts based on SAC.
Evaluations on many challenging continuous
RL tasks, including a toy task GridChaos, tasks in Mujoco as well as their stochastic variants are conducted, and the results verify our analysis.
To the best of our knowledge, OVD-Explorer ﬁrstly addresses the negative impact of aleatoric uncertainty for exploration in continuous RL.
P RELIMINARIES 2. D ISTRIBUTIONAL RL
To model the randomness in the observed long-term return, which characterises aleatoric uncertainty, distributional RL methods (Bellemare et al., 2017; Dabney et al., 2018b;a) are used to estimate Q -value distribution rather than expected Q -value
(Mnih et al., 2015).
In our paper, we focus on quantile regression used in QR-DQN (Dabney et al., 2018b), where the distribution of Q -value is represented by the quantile random variable Z .
Z maps the state-action pair to a uniform probability distribution supported on the values at all corresponding quantile fractions.
Based on value distribution estimator parameterized as θ , we denote the value at quantile fraction τ i as ˆ Z i ( s, a, θ ) given state-action pair ( s, a ) .
Similar to the Bellman operator in the traditional Q-Learning (Watkins & Dayan, 1992), the distributional Bellman operator (Bellemare et al., 2017) T πD under policy π is given as:
[Equation]
Notice that T πD operates on random variables, D = denotes that distributions on both sides have equal probability laws.
Based on operator T πD , Dabney et al.
(2018b) propose QR-DQN to train quantile estimations via the quantile regression loss (Koenker & Hallock, 2001), which is denoted as:
[Equation]
where TD error δ i,j = R ( s t , a t )
+ γ
ˆ Z i ( s t +1 , a t +1 ; ¯ θ ) − ˆ Z j ( s t , a t ; θ ) , the quantile Huber loss ρ τ ( u ) =
u ∗ ( τ − 1 u< 0 ) , and ˆ τ i means the quantile midpoints, which is deﬁned as ˆ τ i
=
τ i +1 + τ i.
2. D ISTRIBUTIONAL S OFT A CTOR -C RITIC METHOD
Distributional Soft Actor-Critic (DSAC)
(Ma et al., 2020) seamlessly integrates distributional RL with Soft Actor-Critic (SAC) (Haarnoja et al., 2018).
Basically, based on the Equation 1, the distributional soft Bellman operator T πDS is deﬁned considering the maximum entropy RL as follows:
[Equation]
where a t +1 ∼ π ( ·| s t +1 ) , s t +1 ∼ P ( ·| s t , a t ) .
Then, the quantile regression loss differs from Equation 2 on δ i,j , by extending clipped double Q-Learning (Fujimoto et al., 2018) based on the maximum entropy RL framework, to overcome overestimation of Q value estimation:
[Equation]
where ¯ θ and ¯ φ represents their target networks respectively.
The actor is optimized by minimizing the following actor loss, the same as SAC,
[Equation]
where D is the replay buffer, f ( s, (cid:15) ; φ ) means sampling action with re-parameterized policy and (cid:15) is a noise vector sampled from any ﬁxed distribution, like standard spherical Gaussian, and Q value is the minimum value of the expectation on certain distributions, as
[Equation]
M ETHODOLOGY
We introduce a general exploration method, named OVD-Explorer, to achieve efﬁcient and robust exploration for continuous RL.
Overall, OVD-Explorer estimates the upper bounds of the policy’s return, and uses the bounds as criteria to ﬁnd a behavior policy for online exploration from the information-theoretic perspective.
Innovatively, to avoid over-exploration, OVD-Explorer particularly takes the aleatoric uncertainty into account, leveraging the optimistic value distribution (OVD) to formulate the upper bounds of policy’s return.
Then, OVD-Explorer derives the exploration policy via maximizing the mutual information between policy and corresponding upper bounds for an optimistic exploration.
Meanwhile, the derived behavior policy avoids being optimistic to aleatoric uncertainty, thus avoiding over-exploration issue.
In the following, we ﬁrst give the intuition together with the theoretical derivation of OVD-Explorer.
Then we practically describe how we formulate OVD based on uncertainty estimation.
Lastly, analysis are given to further illustrate why OVD-Explorer can explore effectively and robustly.
3. O PTIMISTIC VALUE DISTRIBUTION GUIDED EXPLORATION
Most OFU-based algorithms follow the exploration principle: “select the action that leads to areas with high uncertainty”.
However, this principle is incomplete as the high uncertainty may also be caused by the aleatoric uncertainty, which misleads the exploration direction.
Hence, we complement the principle by introducing a constrain: “Not only select the action that leads to areas with high uncertainty, but also avoid the ones that only lead to the area with high aleatoric uncertainty” .
Notable, this new principle intuitively describes a good exploration ability, and we will illustrate how to quantitatively measure a policy’s exploration ability in the later section.
By following this principle, OVD-Explorer derives the behavior policy for exploration.
Intuitively, each policy π in the policy space Π has different exploration ability (at state s ), denoted by F π ( s ) .
Given current state s , OVD-Explorer aims at ﬁnding the exploration policy π E with the best exploration ability by solving the following optimization problem:
[Equation]
We propose to quantitatively measure the policy’s exploration ability from an information-theoretic perspective by measuring the mutual-information of multi-variables as follows:
[Equation]
where π ( ·| s ) is the action random variable, and
¯ Z π ( s, a i ) the random variable describing the upper bounds of return that action a i can reach under policy π at state s .
Here, a i ∈
A denotes any legal action, thus k could be inﬁnite in continuous action space.
The higher F π ( s ) is, the better ability that policy π has in exploring towards higher epistemic uncertainty while simultaneously avoiding higher aleatoric uncertainty, which satisﬁes the principle we raised.
(see analysis in Section 3.3.)
Now we state the Theorem 1 to measure the mutual information above in continuous action space.
Theorem 1.
The mutual information in Equation 8 at state s can be approximated as:
[Equation]
Φ x ( · ) is the cumulative distribution function (CDF) of random variable x , ¯ z ( s, a ) is the sampled upper bound of return from its distribution ¯ Z π ( s, a ) following policy π , Z π describes the current return distribution of the policy π , and C is a constant (See proof in Appendix B.1).
Theorem 1 reveals that F π ( s ) is only proportional to Φ Z π (¯ z ( s, a )) , by maximizing which the policy’s exploration ability can be improved.
Concretely, given any policy π θ (parameterized by θ ), the derivative ∇ θ Φ Z πθ (¯ z ( s, a )) can be measured, thus gradient ascent can be iteratively performed to derive a better exploration policy.
(The complete closed form solution is given in later Section 4)
Note that, to perform above optimization procedure, we need to formulate two critical components in Φ Z π (¯ z ( s, a )) :
1 the return of the policy Z π ( s, a ) and 2 upper bound of the return of the policy ¯
Z π ( s, a ) .
Therefore, we ﬁrst details the formulations in Section 3.2, and then analyze why maximizing Φ Z π (¯ z ( s, a )) can derive a better exploration in Section 3.3.
3. F ORMULATING THE RETURN OF POLICY AND CORRESPONDING UPPER BOUND
Now, we introduce the formulation of the return of policy Z π ( s, a ) and corresponding upper bound ¯
Z π ( s, a ) .
As mentioned before, most OFU-based methods neglect the aleatoric uncertainty when formulating the return and upper bound of return (Mavrin et al., 2019;
Chen et al., 2017), resulting in the over-exploration issue.
Therefore, OVD-Explorer particularly takes into account the aleatoric uncertainty, leveraging distributional RL paradigm to characterize aleatoric uncertainty (Bellemare et al., 2017; Dabney et al., 2018b;a).
OVD-Explorer uses two value distribution estimators ˆ Z ( s, a ; θ 1 ) and ˆ Z ( s, a ; θ 2 ) parameterized by θ 1 and θ 2 , as ensemble estimators to formulate ¯ Z π and Z π in different ways.
Unless stated otherwise, the ( s, a ) is omitted hereafter to ease notation.
Formulation of ¯ Z π .
The ¯ Z π denotes the upper bounds of value that policy π can reach via different actions.
We propose Gaussian distribution with optimistic mean value for formulation as follows, and accordingly refer to it as Optimistic Value Distribution (OVD):
[Equation]
where µ ¯ Z ( s, a ) and σ 2 aleatoric ( s, a ) is mean value and variance, respectively.
Notable, Chen et al.
(2017) discovers the optimisticity is beneﬁcial for better estimating the upper bound, which motivates us to optimistically estimate the µ ¯ Z ( s, a ) by considering epistemic uncertainty as follows:
[Equation]
[Table] [Table]
where µ ( s, a ) represents the Q -value estimation, σ epistemic ( s, a ) is the epistemic uncertainty, β is a hyper-parameter controlling the magnitude of epistemic uncertainty, U is uniform distribution, N is the number of quantiles, and ˆ Z i ( s, a ; θ k ) is the value of the i -th quantile drawn from ˆ Z ( s, a ; θ k ) .
Moreover, the aleatoric uncertainty should be considered as it will affect the policy’s performance, thus we propose to model it as the variance of ¯ Z π .
In practice, the aleatoric uncertainty σ aleatoric ( s, a ) can be captured by value distribution estimators (Clements et al., 2019) as follows:
[Equation]
Leveraging optimistic value estimations (via epistemic uncertainty) together with explicitly modeling aleatoric uncertainty, the upper bound of the value ¯ Z π can be effectively formulated as an optimistic Gaussian distribution.
Such an upper bound value estimation provides a useful guidance for OVD-Explorer to derive the behavior policy for online exploration.
Formulation of Z π .
Z π estimates the value obtained following policy π .
Inspired by TD3 (Fujimoto et al., 2018), to alleviate overestimation, we propose to formulate Z π in a pessimistic way.
In practice, Z π can be measured in two ways.
First, similar to formulating ¯ Z π in Equation 10, Z π can also be formulated as Gaussian distribution as follows:
[Equation]
where µ ( s, a ) , σ aleatoric ( s, a ) and σ epistemic ( s, a ) are the same deﬁned in Equation 11.
Differently, σ epistemic ( s, a ) is subtracted from µ ( s, a ) , which reveals the pessimistic value estimation of Z π .
Another way is to formulate Z π as multivariate uniform distribution as follows:
[Equation]
where each quantile value z πi ( s, a ; θ ) equals to the minimum estimated value among ensemble estimators (i.e., z i ( s, a ; θ k ).
As such, the estimated value of Z π is relative pessimistic.
Adopting pessimistic value estimation avoids overestimation issue, thereby the value distribution
Z π can be more accurately formulated.
Additional, Gaussian distribution helps more when the environment randomness follows a unimodal distribution, and multivariate uniform distribution can be more ﬂexible and suitable for scenarios with multi-modal distributions.
(see more in Section 5.3.)
3. A NALYSIS OF OVD-E XPLORER
This section analyzes how OVD-Explorer optimistically explores the informative areas and avoids the over-exploration issue.
According to Theorem 1, the exploration policy OVD-Explorer derives can maximize F π ( s ) , which is proportional to CDF value Φ Z π (¯ z ( s, a )) .
In the following, Figure 2(a) and 2(b) illustrate such CDF values for different actions (shaded area), and Table 1 shows how uncertainties affect the exploration.
In these cases, the value distribution Z π ( s, a ) is speciﬁed as a Gaussian distribution (in Equation 13), and the sampled optimistic value ¯ z ( s, a ) is speciﬁed as the mean of OVD µ ¯ Z
( s, a ) (in Equation 11).
Speciﬁcally, at state s , we assume that the means of Z π at actions a 1 and a 2 are the same for ease of clariﬁcation.
Basically, Figure 2(a) shows OVD-Explorer can achieve more optimistic exploration.
We assume the aleatoric uncertainty at a 1 and a 2 are the same, but epistemic uncertainty is higher at a 1 , causing µ ¯ Z ( s, a 1 ) > µ ¯ Z ( s, a 2 ) .
Thus the CDF value is larger at a 1 , which means the action with higher epistemic uncertainty is preferred.
P r ob a b ilit y D
e
n s it y π 1 ( , )
Z s a π 2 ( , ) Z s a 1 ( , )  Z s a 2 ( , )  Z s a ( , ) zs a (a)
P r ob a b ilit y D
e
n s it y (b)
More crucially, Figure 2(b) shows that OVDExplorer could guide to avoid the area with higher aleatoric uncertainty given the equal optimism.
We assume that the epistemic uncertainty at a 1 and a 2 are the same, causing their optimistic value estimation to be equal to µ ¯ Z ( s, a ) , but aleatoric uncertainty is lower at a 1 , i.e., PDF curve of Z π ( s, a 1 ) is ”thinner and taller”.
Thus the CDF value is larger at a 1 , meaning that the action with lower aleatoric uncertainty is preferred.
In the limit of t → ∞ , the epistemic uncertainty tend to be 0 , and the aleatoric uncertainty estimated in Equation 12 converges to represent the true environment randomness.
This is comparable to the second case (Figure 2(b)), and OVD-Explorer tends to choose the one with lower aleatoric uncertainty, which is a great advantage of our method over other OFU methods.
To summarize, the exploration guided by OVD-Explorer trades off between two criteria: exploring the areas with higher epistemic uncertainty, to ensure exploring optimistically, and avoiding the areas with higher aleatoric uncertainty, to avoid over-exploration caused by high environment randomness.
OVD-E XPLORER FOR MODERN RL ALGORITHMS
For continuous action space, the argmax operator in Equation 7 is intractable.
To address that, in this section, we derive the behavior policy π E in closed form and state the scheme to incorporate it with existing policy-based algorithms.
First, we denote the policy learned by any policy-based algorithm as the target policy π T .
To avoid the gap between training data collected using the behavior policy π E and the target policy π T , we need to constrain the difference between π E and π T , thus we derive π E in the vicinity of π T .
Second, to derive exploration π E for modern RL algorithms with stochastic policy based on OVDExplorer, where both the exploration π E = N ( µ E , Σ E ) and target policy π T = N ( µ T , Σ T ) are
Gaussian distributions, we introduce the following proposition:
Proposition 1.
The OVD-Explorer behavior policy π E = N ( µ E , Σ E ) is as follows:
[Equation]
and
[Equation]
In speciﬁc, α is the step size controlling the exploration level and m = log Φ Zπ ( s,µT ) (¯ z
[Equation]
s [Equation] )) C
[Equation]
(see proof in Appendix B.2).
The expectation E ¯ Z π can be estimated by sampling K samples, then Equation 15 is simpliﬁes as:
[Equation]
Algorithm 1 summarizes the overall procedure of OVD-Explorer, including the formulation of ¯ Z π and Z π (Line 2 and line 3) and behavior policy generation (Line 4).
The generated behavior policy can be integrated with any modern policy-based RL algorithms to render the stable and wellperformed algorithm.
Please see Appendix C for the entire pseudo-code and details about how the
OVD-Explorer-based behavior policy is incorporated with DSAC.
E XPERIMENTS
To reveal the consistency between our theoretical analysis and the performance of OVD-Explorer algorithm, we conduct experiments to address the following questions:
5. E XPERIMENT S ETUP
Baselines include SAC (Haarnoja et al., 2018), DSAC (Ma et al., 2020), and DOAC, which is the distributional variant of OAC (Ciosek et al., 2019).
We implement Gaussian and quantile formulations of Z π as in Equation 13 and 14, which are denoted in the following as OVDE G and OVDE Q, respectively.
We test OVD-Explorer on a novel task GridChaos and several tasks in Mujoco (Todorov et al., 2012) including the stochastic variants.
The appendix gives more details, including environment settings, implementation details, hyperparameters settings, and computing infrastructure used, as well as more experiment evaluations, including tasks with different noise scales, tasks with different episode horizon, the sensitivity to β and ablation study on the pessimistic formulation of Z π ( s, a ) .
5. E XPLORATION IN G RID C HAOS (RQ1)
To illustrate the exploration pattern of OVD-Explorer and show the advantage of OVD-Explorer over DSAC and DOAC, we evaluate OVD-Explorer on a novel continuous and stochastic control task called GridChaos.
[Table]
Figure 3(a) shows the map of GridChaos, in which we control the cyan triangle (agent) aiming to reach the ﬁxed dark blue goal.
The state is the current coordinate, and the action is a two-dimensional vector including the movement angle and distance.
One episode terminates when the agent reaches the goal or maximum steps (i
[Table]
e., MAX STEP, typically 100).
Also, it receives a +100 reward when reaching the goal otherwise 0.
The reason why it is chaos is that the randomness of the transition is heterogeneous in different parts of this environment.
Table 2 shows the environment settings.
Moreover, Figure 3(b) shows that OVDExplorer can reach the goal faster, with more efﬁcient exploration.
Figure 3(c) shows the values of uncertainty estimation and exploration objective (mutual information) taken four different actions at the state shown in Figure 3(a) and at training epoch 1249.
Basically, Figure 3(c) illustrates that the estimated aleatoric uncertainty of left is higher than that of right, indicating that OVD-Explorer models aleatoric uncertainty properly.
Further, OVD-explorer encourages to explore the right side at that time, since the value of exploration objective (in green) is highest.
It implies that OVD-explorer tends to explore areas with higher epistemic uncertainty and avoiding higher aleatoric uncertainty, which is in accordance with our exploration principle.
On the other hand, if high epistemic uncertainty is considered only like in OAC(Ciosek et al., 2019), the
[Figure]
agent would be guided to the left.
Then the agent may be trapped in the left side due to the high aleatoric uncertainty, and it is the possible reason about why OAC fails tackling such heteroscedastic stochastic task.
In appendix, we show more analysis about the exploration pattern, please refer to
Appendix E.2 and Appendix E.10.
Besides, we conduct the evaluation when the randomness is high around the goal, as well as many other noise settings, the results enhance the ability of OVD-Explorer.
More experiments that evaluate the performance on several other noise settings, as well as more detailed value histogram on different epoch in training process can be found in Appendix.
5. P ERFORMANCE ON M UJOCO TASKS (RQ2)
To demonstrate the performance of OVD-Explorer more generally, we evaluate it on several Mujoco tasks.
For 5 standard Mujoco tasks 1 , the transition is deterministic and the randomness is only from the stochastic policy.
For 5 noisy tasks, the heteroscedastic Gaussian noise is added in Mujoco tasks.
Speciﬁcally, in each state transition on noisy tasks, Gaussian noise of different scales is randomly injected following a certain probability.
Overall, we have the following ﬁndings.
Basically, OVD-Explorer can perform stably in the standard tasks with slight or little randomness, guiding efﬁcient exploration.
From the results 2 in standard Mujoco tasks in table 3, it is shown that in the relatively easy tasks, i.e., Hopper-v2, Reacher-v2, and HalCheetah-v2, OVD-Explorer does not obtain much gain beyond the baselines, which seems that these tasks are not profoundly demanding for exploration.
In the high-dimension tasks, i.e., Ant-v2 as shown in Figure 4(a), OVD-Explorer signiﬁcantly outperforms DSAC and DOAC.
It means that the aleatoric uncertainty caused by policy indeed degrades the performance of DOAC, where the aleatoric and epistemic uncertainties are not distinguished.
Besides, OVD-Explorer can avoid the impact of heteroscedastic aleatoric uncertainty on exploration and thus improve robustness.
As the results of noisy Mujoco tasks shown in Table 3, DOAC is worse than DSAC in most cases, which means that heteroscedastic aleatoric uncertainty causes signiﬁcant degrades of DOAC.
Simultaneously, OVD-Explorer signiﬁcantly outperforms DOAC and DSAC, especially in Noisy Ant-v2 as shown in Figure 4(b).
Furthermore, OVDE G takes the Gaussian prior and concretely obtains better estimation of critic in the noisy tasks, while OVDE Q performs better in the standard tasks due to the ﬂexibility of quantile distribution.
Generally, the difference between OVDE G and OVDE Q is that they use different formulations of Z π ( s, a ) .
Theoretically, the value function distribution in OVDE Q is more ﬂexible and should perform better than OVDE G, as the reported results on ﬁve standard Mujoco tasks show.
On the other hand, OVDE G performs better in the noisy tasks, and it is because that the transition probability is Gaussian in the stochastic Mujoco tasks.
[Table]
Here, the reported results may be slightly different from previously reported results, partly due to the statistic approach, and partly due to the implementation.
Nevertheless, the patterns of these baselines (e.g.,
DSAC outperforms SAC in the vast majority of cases) are consistent with previous results.
The details about the implementation of baselines are described in the Appendix D. 2 .
[Table]
47 ±
[Figure] [Figure]
28 ±
[Figure] [Figure]
63 ±
[Figure]
45 N
[Figure]
OPPER
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
74 ±
[Figure] [Figure]
[Figure]
237.68 ± 13.11 N
[Figure]
P
[Figure]
-46.92 ± 12.
-25.31 ± 2.
[Figure]
13 ± 9.
41 ± 0.
[Figure] [Figure]
51 ±
[Figure] [Figure]
S ENSITIVITY TO α (RQ3)
The step size α in Equation 15 controls how much the behavior policy derived from OVD-Explorer, i.e. π E , is far away from the target policy π T , which can be essential for exploration beneﬁt.
To investigate the appropriate range of it, we test several α value on Noisy Ant-v2 task using OVDE G, and the result is shown in Figure 4(c).
If α is quite small, OVD-Explorer degenerates to DSAC and implies little exploration.
In contrast, if α is larger, the performance becomes worse because of the huge gap between behavior policy and target policy.
The result demonstrates that α should be taken in a suitable range to facilitate more adequate exploration.
In our experiments, we uniformly use α to be equal to 0.05 to show the performance of the OVD-Explorer.
In addition, we ﬁnd that a smaller α leads to higher gains when the task is much more difﬁcult, and the details can be found in
Appendix E.6.
We also verify the sensitivity of OVD-Explorer to β in Appendix E.7 and found that there is a broad range of settings for β , which can lead to well performance.
C ONCLUSION
This paper proposes an information-theoretic exploration method OVD-Explorer, which introduces a novel measurement of exploration ability, i.e., the mutual information between the policy and the upper bounds of return.
By maximizing the mutual information, OVD-Explorer is able to derive the behavior policy, that follows the OFU principle, and further avoids exploring the areas with high aleatoric uncertainty.
Integrated with SAC, OVD-Explorer addresses the negative impact of aleatoric uncertainty for exploration in continuous RL for the ﬁrst time.
A R ELATED W ORK
In this work, we consider the exploration strategy under the principle of Optimism in the Face of
Uncertainty (OFU)
(Auer et al., 2002), especially in the heteroscedastic stochastic environment.
We aim to improve exploration efﬁciency, and alleviate the over exploration issue caused by aleatoric uncertainty.
Basic exploration strategies, like (cid:15) -greedy (Sutton & Barto, 2018), noise perturbation (Lillicrap et al., 2016), entropy regularization (Mnih et al., 2016) and stochastic policy (Haarnoja et al., 2018), lead to undirected exploration through random perturbations.
With the increasing emphasis on exploration efﬁciency in RL, various exploration methods have been developed.
One kind of methods uses intrinsic motivation to stimulate agent to explore, such as count-based novelty (Martin et al., 2017; Ostrovski et al., 2017; Bellemare et al., 2016; Tang et al., 2017; Fox et al., 2018), prediction error (Pathak et al., 2017), reachability (Savinov et al., 2019) and information gain on environment dynamics (Houthooft et al., 2016).
Some recent methods, originating from tracking uncertainty, guide efﬁcient exploration under the principle of OFU, such as Thompson Sampling (Thompson, 1933; Osband et al., 2016),
IDS (Nikolov et al., 2019; Clements et al., 2019) and other customized methods (Moerland et al., 2017; Pathak et al., 2019).
The base of OFU methods is to model epistemic and aleatoric uncertainties in RL.
Bootstrapped
DQN (Osband et al., 2016) has become the well-used approach for capturing epistemic uncertainty (Kirschner & Krause, 2018;
Ciosek et al., 2019), and distributional RL methods (Bellemare et al., 2017;
Zhou et al., 2020; Dabney et al., 2018a;b) are used for capturing aleatoric uncertainty.
However, most traditional OFU methods do not distinguish the two types of uncertainty, which can easily lead the naive solution to favor actions with higher variances in stochastic tasks, i.e., over-exploration issue.
To address that, Mavrin et al.
(2019) study how to take advantage of value distribution for efﬁcient exploration under both types of uncertainty, proposing Decaying Left Truncated Variance (DLTV) based on QR-DQN.
Besides, Nikolov et al.
(2019) and Clements et al.
(2019) propose to use Information Direct Sampling (Kirschner & Krause, 2018) for efﬁcient exploration in RL, which formulates epistemic and heteroscedastic aleatoric uncertainty and maximizes information gain on globally optimal action to explore informative state-action pairs.
However, such methods are complicated when deriving a behavior policy and is limited to discrete control.
Meanwhile, there is not any strategy that can help the well-performed continuous RL algorithms (Haarnoja et al., 2018; Ciosek et al., 2019; Ma et al., 2020) to address aleatoric uncertainty when exploration.
OAC (Ciosek et al., 2019) proposes exploration bonus guided by the upper bound of Q estimation to facilitate exploration based on Soft Actor-Critic (SAC) (Haarnoja et al., 2018).
Nevertheless, OAC ignores the potential impact of the aleatoric uncertainty, which may cause misleading exploration.
Our proposed OVD-Explorer is a novel exploration strategy, which can guide agent to explore towards higher epistemic uncertainty, and also avoid the areas with high aleatoric uncertainty, improving the robustness of exploration especially facing heteroscedastic aleatoric uncertainty.
To capture aleatoric uncertainty, OVD-Explorer models the value distribution and uses mutual information to guide exploration following the principle of OFU, measuring the correlations between the policy distribution and upper bounds distribution of return.
There are some other informationtheoretic exploration strategies using mutual information, such as VIME (Houthooft et al., 2016), which measures the information gain on environment dynamics, and EMI (Kim et al., 2019), which generates intrinsic reward using prediction error of representation learned by mutual information.
Those methods can solve sparse reward problem very well by using intrinsic reward.
Nevertheless, those exploration methods use mutual information neither on the value distribution, nor for OFUbased exploration.
Besides, unlike the mechanisms used in measuring mutual information, such as variational inference (Hinton & van Camp, 1993; Houthooft et al., 2016) and f-divergence (Nowozin et al., 2016; Kim et al., 2019), we ﬁnd the correlation between policy and upper bounds of return through uncertainty as shown in Theorem 1, thus we can directly derive the close form exploration policy.
B P ROOFS B. P ROOF OF T HEOREM
In order to prove the Theorem 1, we ﬁrst propose the following lemma about F π ( s ) .
Lemma 1.
The mutual information of ¯ Z π ( s, a 0 ) , · · · , ¯ Z π ( s, a k − 1 ) and π ( ·| s ) at state s is:
[Equation]
where p ( a | ¯ z
( s, a ) , s ) represents the posterior probability distribution of policy given current state s and the sampled upper bound of return
¯ z ( s, a ) .
Proof.
For simplicity, we assume that the size of action space is k = 2 , and the actions are denoted as a 0 and a 1 , a 0 (cid:54) = a 1 .
Then we derive the mutual information among three random variables MI ( ¯ Z π ( s, a 0 ) , ¯ Z π ( s, a 1 ) , π ( ·| s )) , where the action sampled from π ( ·| s ) is either a 0 or a 1 .
Considering the formula for the mutual information, F π ( s ) is derived as follows:
[Equation]
where the posterior distribution p ( a | ¯ z
( s, a 0 ) , ¯ z ( s, a 1 )) is the probability of choosing action a on the condition of the samples from upper bounds of action a 0 and a 1 .
Considering that in the decision-making process, the probability of action a 0 is independent to the upper bound of other actions, such as ¯ z ( s, a 1 ) , which means that p ( a 0 | ¯ z ( s, a 0 ) , ¯ z ( s, a 1 )) = p ( a 0 | ¯ z ( s, a 0 )) .
Therefore, the above equation can be further reduced as follows.
[Equation]
It is easy to extend to inﬁnite-action case. Based on that for any action a i ∈ [ a 0 , a k − 1 ] and k → ∞ , the conditional probability p ( a i | ¯ z
( s, a 0 ) , ¯ z ( s, a 1 ) , ..., ¯ z
( s, a k − 1 )) = p ( a i | ¯ z
( s, a i )) , F π ( ·| s ) can be simpliﬁed as following,
[Equation]
Lemma 1 tells that the mutual information F π ( s t ) is in direct proportion to p ( a |
¯ z ( s, a ) , s ) , which measures how much it is worth acting under the current policy π ( a | s ) when the upper bound is known.
Next, to measure the posterior probability p ( a | ¯ z ( s, a ) , s ) , we use a general and practically effective approach (Wang & Jegelka, 2017; Belakaria et al., 2020; Perrone et al., 2019;
Li et al., 2020) of approximating the posterior probability given upper bound value.
Speciﬁcally, we approximate p ( a | ¯
z
( s, a ) , s ) using the prior that z π ( s, a ) ≤ ¯ z ( s, a ) with given policy π ( s, a ) , since ¯ z ( s, a ) is the upper bound of z π ( s, a ) .
Hence, we use the indicator function 1 z π ( s,a ) ≤ ¯ z
( s,a ) to truncate the policy π ( s, a ) , and utilize the constant C to normalize the probability, as is shown in the following equation.
[Equation]
Here, E z π ( s,a ) ∼
[Equation] [Equation]
[Equation]
= Φ Z [Equation]
s [Equation]
) (¯ z ( s, a )) , where Φ x is the cumulative distribution function (CDF) of x , ¯ Z π and Z π are the random variables, whose distributions describe the randomness of the returns, and ¯ z ( s, a ) is the value of random variable ¯ Z π .
Therefore, the posterior probability can be measured as follows,
[Equation]
In our method, we do not use the commonly used mechanisms about mutual information such as neural network estimation (Belghazi et al., 2018) and upper bound estimation (Cheng et al., 2020).
Instead, we can ﬁnd the correlation between random variables as shown in Equation 20, which helps to derive mutual information directly.
According to Lemma 1 and Equation
[Equation]
we can give the proof of Theorem 1 in the following.
Proof.
By Combining Lemma 1 and Equation 20, F π ( s ) can be further derived as follows.
[Equation]
Here [Equation] the last equality follows from Theorem 1.
B. P ROOF OF PROPOSITION
Proof.
Similar to (Ciosek et al., 2019) [Equation] we set the covariance matrix of behavior policy π E is that of target policy π T , i [Equation]
e., Σ E = Σ T .
Hence, the OVD-Explorer problem is simpliﬁed as:
[Equation]
To ensure that the behavior policy samples actions around the target policy, we derive the π E upon mean µ T of target policy π T .
In speciﬁc, we ﬁrstly obtain the gradient of ˆF ( s, µ ) at π T , which is given as follows:
[Equation]
1: Initialise: networks θ θ
2: for each iteration do 3: for each do 4: a t ∼ π E ( a t , s t ) according to Algorithm5:
D ← D ∪ { ( s t , a t , r ( s t , a t ) , s t +1 ) } 6: end for 7: for each training step do 8: for i = 1 to N do 9: for j = 1 to N do 10: calculate δ ki,j , k = 1 , 2 , following Eq.11: end for 12: end for 13:
Calculate L QR ( θ k ) , k = 1 , 2 using δ ki,j following Eq.14:
Update θ k with ∇L QR ( θ k ) 15:
Calculate J π ( φ ) , following Eq.16:
Update φ with ∇J π ( φ ) 17: end for 18:
Update target value network with ¯ θ k ← τθ k + (1 − τ )¯ θ k , k = 1 ,19:
Update target policy network with ¯ φ ← τφ + (1 − τ )¯ φ 20: end for where ˆ m is given as: [Equation] and φ ( x ) is the probability distribution function (pdf). Hence, the µ E is given as follows:
[Equation]
where α is the step size controlling exploration level and m = log Φ Zπ
[Equation]
s,µT ) (¯ z ( s,µ T )) C
+ 1 .
C A LGORITHM 2: OVD-E XPLORER FOR DSAC
In this section, we show the whole algorithm of our implementation of OVD-Explorer based on
DSAC in Algorithm 2.
All the code can be found in the supplementary material.
D M ORE DETAILS ABOUT THE EXPERIMENTS D. G RID C HAOS
GridChaos is an environment built on OpenAI’s Gym toolkit, whose map is shown as in Figure 3(a) and Figure 5.
In this section we illustrate more details in addition to Section 5.2.
The movable cyan triangle and the ﬁxed symmetric dark blue goal are two parts split from square, and the goal is to make the triangle embedded in the goal to recover the original square, which is to say that it is an isosceles triangle whose base side is equal to the height.
The triangle is always initialised randomly in the cyan rectangle, and the black line in the map represents the wall, where the triangle will be [Table] adsorbed once hits the wall.
The state transition is stochastic, and we add Gaussian noise to the action resulting in Gaussian transition probability.
To represent the location of the triangle, we establish a Cartesian coordinate system using the centroid of the map as the origin, as shown in Figure 5.
Then the coordinates of the triangle are represented by the midpoint of the altitude of the triangle which is shown as the red point in the triangle.
In the case shown in Figure 5, the initial coordinate of the triangle (agent) is in the negative half of the x-axis and the task target is in the ﬁrst quadrant.
D. B ASELINES
Ma et al.
(2020) shows the performance of DSAC and TD4, which is the distributional extension of TD3 (Fujimoto et al., 2018), and can also be used to capture epistemic and aleatoric uncertainty.
Moreover, DSAC outperforms TD4 on Mujoco tasks as shown in Ma et al.
(2020), so we evaluate only on SAC and DSAC, and further implement OVD-Explorer based on DSAC to develop the exploration ability.
SAC.
The SAC (Haarnoja et al., 2018) implementation is mainly based on OAC repository, and the results in Ant-v2 and Hopper-v2 are similar to reported results by OAC.
Our SAC report a better result than OAC’s implementation for SAC on HalfCheetah-v2, which is because the high variance of this environment as explained as in OAC.
DSAC.
The DSAC (Ma et al., 2020) implementation is based on our implementation of SAC, except that the distributional Q function in the DSAC repository is used instead of the traditional Q function in SAC.
As it is based on SAC, we set the hyper-parameters of DSAC to be consistent with SAC to ensure the fair comparison, which also results in the different reported results from original paper of
DSAC.
In our results, DSAC can guarantee an absolute advantage over SAC in most cases, which is consistent with the previous conclusion.
DOAC.
The DOAC implementation is mainly based on our implementation of DSAC as well as the open source code of OAC.
As DSAC shows great advantage due to the distributional value estimation, to ensure a fair comparison, we extend OAC (Ciosek et al., 2019) to its distributional version, i.e., DOAC, by replacing the exploration process of DSAC by the behavior policy derived by OAC.
We set the hyper-parameters the same as used by OAC in Mujoco 3 , and our results of
DOAC on Ant-v2 and HalfCheetah-v2 are signiﬁcantly better than that OAC reported.
D. I MPLEMENTATION
Our implementation of OVD-Explorer is based on the open source code of OAC 4 , also refer to the code of DSAC 5 as well as softlearning 6 .
All experiments are performed on NVIDIA GeForce RTX
Ti 11GB graphics card.
The training process of OVD-Explorer and DOAC are the same as in DSAC, except for the different behavior policy used, while OVD-Explorer and DOAC enrich the experience replay with the data using the the derived exploration policies, respectively.
To ensure the fair comparison, the hyperparameters for training process of baselines and OVD-Explorer are the same.
Besides, we have three hyper-parameters associated with OVD-Explorer as mentioned before, including α that controls the exploration level, β that determines the magnitude of uncertainty we use, as well as C that is the normalization factor.
The hyper-parameters in our experiments are shown in Table 4.
E M ORE E XPERIMENT S TUDY E. R UNTIME ANALYSIS
[Figure]
Figure 6 shows the time consumption of algorithms relative to SAC.
As can be seen, the distributional value estimation used in
DSAC, DOAC and our methods introduces extra time consumption distinctly.
Nevertheless, the relative time consumption of OVDE G and
OVDE Q to SAC is 1.21 and 1.17, respectively, which means that OVD-Explorer spends about 20% more time than SAC to achieve up to nearly 100% performance gain as shown in Figure 6(b).
This demonstrates the extra time consumption is well worth it.
Besides, the time consumption of OVDE Q is close to that of
DSAC, only with a larger variance, which indicates that the additional time consumption of
OVDE Q is minimal while performing better exploration.
E. A NALYSIS ABOUT OVD-E XPLORER ’ S ADVANTAGE IN THE CASE OF G RID C HAOS
With the heatmap of the visiting frequency of agent during exploration, and the heatmap about the uncertainty estimation, we can visually analyze the patterns and advantages of OVD-Explorer.
[Figure]
[Figure]
The distinctly different exploration patterns can be easily found.
Figure 7(a), 7(b) and 7(c) present the state visiting frequency of OVD-Explorer, DSAC and DOAC, respectively.
We can see that
OVD-Explorer explores directly to the right half, where the environmental randomness is lower, whereas DSAC and DOAC are both stuck in the left half with higher environmental randomness.
Furthermore, we show how OVD-Explorer could explore directly without being trapped by the randomness through the estimated uncertainty.
In speciﬁc, Figure 7(d) shows that the aleatoric uncertainty estimated by OVD-Explorer is consistent with environment settings, where the environment noise is higher on the left half.
Figure 7(e) shows the ratio of estimated epistemic uncertainty and aleatoric uncertainty (i.e., epistemic-aleatoric ratio) of OVD-Explore, and higher ratio means higher epistemic uncertainty or lower aleatoric uncertainty, which is exactly the direction OVD-Explorer explores .
The ratio is larger on the right half, which means that OVD-Explorer can avoid being stuck in the left half.
Meanwhile, Figure 7(f) presents the estimated uncertainty in DOAC, which is larger on the left half.
As DOAC encourages exploring area with relatively large estimated uncertainty, it explains why DOAC is stuck in the left half.
E. E VALUATION ON SEVERAL OTHER NOISE SCALE IN G RID C HAOS
OVD-Explorer can explore efﬁciently in heteroscedastic stochastic environment by considering differently about epistemic and aleatoric uncertainty for exploration as shown in Section 5.2.
To further empirically prove its strength, we test OVD-Explorer in GridChaos with different noise scales in four quadrants, and the result is shown as Figure 8 and Table 5.
We can ﬁnd that OVD-Explorer can perform well in all those different noise injection of environment.
[Figure]
[Table] [Figure] [Table] Concretely
[Table]
from the results [Table] the following observations deserve to be noticed.
First, OVD-Explorer can signiﬁcantly achieve better average return in all those settings, especially when the noise is set high [Table] and can learn to reach the goal faster (see column about FRG).
It shows the ability of OVDExplorer to guide agent explore against higher aleatoric uncertainty on the left side (the second and third quadrants).
Second, for the task without noise as shown in row E , which means the state transition is deterministic, OVD-Explorer still learns quickly.
The results in row E show the inherently high difﬁculty of this task, not only because of the very sparse reward, but also the gate leading to the goal is set very small (the width of the gate is only 30% of the length of agent, i.e., the base of the isosceles triangle, which means that at the doorway the agent can only move a very small distance horizontally, otherwise it would be adsorbed to the wall and immobile).
E. E VALUATION IN G RID C HAOS WHEN THE ALEATORIC UNCERTAINTY IS HIGH AROUND GOAL
In the previous experiments in GridChaos, the noise (i.e., aleatoric uncertainty) near the goal is set lower.
In such situation, OVD-Explorer, which follows the principle of OFU and further avoids exploring areas with higher aleatoric uncertainty, could bring signiﬁcant advantage.
Such setup of heterogeneous noise is reasonable, because in real life, the goal or optimal policy is always not expected to be highly stochastic.
Nevertheless, the evaluation about tasks with the existence of high randomness in the target region is valuable, so we conducted the following experiment in GridChaos, where the environment randomness in the right half (ﬁrst and fourth quadrants), where the target is located, was set larger.
The results are shown in Table 6.
Note that we use OVDE(P) to denote the usual implementation that pessimistically estimates the value distribution (i.e., using Equation 13).
Besides, OVDE(M) denotes the implementation that does not pessimistically estimate the value distribution (i.e., we modify the mean of Gaussian distribution Z π in Equation 13 from the lower bound to expected value of the Q estimation µ ( s, a ) as in Equation 11.).
[Table]
Our experimental ﬁndings are mainly the following three aspects.
Firstly, when facing extremely high aleatoric uncertainty around the goal (see row F), which causes the interaction around goal to be very unstable, chaotic and disorder, OVD-Explorer would strongly discourage exploring such a area [Table] and thus performance would be damaged.
In contrast, DSAC and
DOAC have no restriction on aleatoric uncertainty, and high randomness may instead increase the probability of achieving the goal.
Second, in most cases (see G, H, I, J), OVD-Explorer always can guide better exploration and achieve better performance than DSAC and DOAC, especially when the noise is negligible (see row
J).
This reﬂects the fact that our exploration objective (the mutual information shown in Equation 8) makes great sense, achieving an appropriate trade-off between avoiding high aleatoric uncertainty and being optimistic about high epistemic uncertainty.
Third, an interesting ﬁnding is that OVD-Explorer may perform better by turning off the pessimistic estimation facing higher aleatoric uncertainty around the goal (see column OVDE(M)).
This suggests that excessive pessimism is unnecessary if there is a need to explore areas with high aleatoric uncertainty.
Overall, from the results in Table 5 and Table 6, OVD-Explorer is able to tackle most of the cases quite well.
When there is high randomness around the goal, OVD-Explorer has a shortcoming that it will inevitably slow down the efﬁciency of reaching the goal, because it limit the exploration towards such area.
Fortunately, this shortcoming can be mitigated by turning off the pessimistic estimation.
E. E VALUATION OF STATISTICAL SENSE
[Table]
To counteract the randomness from a statistical perspective
[Table]
we conduct all experiments for 5 trails with different seeds (typically 0-5) [Table] and report the average results with standard deviation.
Next, to verify that the 5 trails are sufﬁcient to mitigate the statistical randomness [Table] we run other 5 runs (seeds are set as 5, 6, 7, 8, 9, respectively) for those algorithms on Ant-v [Table]
and show the results in the following. Note that the ﬁrst row of results is from our previously reported results, which is the same as Table [Table] and the second row show the results new.
The experimental results in Table 7 show that the results of 5 trials are sufﬁciently representative of the overall level, while the performance of OVD-Explorer undoubtedly stays ahead.
E. S TUDY ON EPISODIC HORIZON
Section 5.3 has shown great advantage of OVD-Explorer over DSAC and DOAC in stochastic Mujoco tasks, which limits the length for an episode to 100 steps.
To further empirically verify the efﬁciency of OVD-Explorer, we test on Noisy Ant-v2 task with different maximum episodic length setup.
Our results in Figure 9 show that OVD-Explorer can signiﬁcantly perform better than baselines in different maximum episodic length (i
[Figure]
e., 250, 500, 750 and 1000).
Noting that longer maximum episodic length renders higher difﬁculty of solving tasks, especially for the high-dimensional tasks demanding exploration.
In speciﬁc [Figure] we have the following two conclusions
[Figure]
Firstly, the exploration should be more conservative in the harder tasks, where we should set smaller α in OVD-Explorer.
In Figure 9(a), (b) and (c), α is set to 0.05 by default, while we can ﬁnd that the advantage of OVD-Explorer G decreases gradually with the increasing of the difﬁculty of tasks.
Further, if α is set to 0.005, then there is a substantial performance improvement as shown in
Figure 9(d).
Also, as shown in Figure 9(e), both OVD-Explorer methods perform well when the task episodic horizon is 750 with α set to 0.005.
On the hardest task we tested, i.e., the Noisy Ant-v with horizon 1000 as shown in Figure 9(f), OVD-Explorer gain remarkable performance, while α is set smaller as 0.001.
Secondly, OVD-Explorer Q is more stable than OVD-Explorer G, which is consistent with the conclusion in Section 5.3.
We can ﬁnd from Figure 9(a), (b) and (c) that OVD-Explorer Q performs stably better while OVD-Explorer G degrades.
OVD-Explorer G is better in easier task with horizon 100, which is due to the Gaussian prior of noise.
But when the task becomes harder, the prior helps less, and OVD-Explorer Q shows the advantage of more ﬂexibly modeling aleatoric uncertainty and thus the performance is more stable.
E. S TUDY ON HYPER - PARAMETERS β
OVD-Explorer is sensitive to α , as shown in Section 5.4.
There is another hyper-parameter β , which controls the scale of uncertainty quantiﬁcation as shown in Equation 11 and Equation 13, further having an impact on ¯ Z π and Z π .
To evaluate its sensitivity about β , we conduct the experiment on Noisy Ant-v2 task using OVD-Explorer G, and the result is shown in Figure 10.
The results demonstrate that there is a broad range of settings for β , which can lead to well performance.
[Figure]
e R e t u r n
E. A BLATION STUDY ON VALUE DISTRIBUTION ESTIMATION
As mentioned in Section 3.2, we estimate Z π pessimistically to alleviate over-estimation.
Also, as mentioned in Appendix E.4, the pessimism is unnecessary if there is a need to explore areas with high aleatoric uncertainty.
In the following, to investigate the beneﬁt of pessimistic estimation in general case, we compare the performance of OVD-Explorer to the modiﬁed versions that use normally estimated Z π .
Our results show that pessimistic estimation can mostly be better than that using normal estimation.
For OVDE G (mean), we modiﬁed the mean of Gaussian distribution Z π in Equation 13 from the lower bound to average value of the Q estimation µ ( s, a ) as in Equation 11.
For OVDE Q (mean), we modiﬁed the z
πi ( s, a ) in Equation 14 from the minimum value to the average value, i.e., z πi ( s, a ; θ ) =
E k =1 , 2 ˆ Z i ( s, a ; θ k ) .
As shown in Figure 11, both OVDE G (mean) and OVDE Q (mean) perform worse than the pessimistic version.
To draw a conclusion, pessimistic estimate is indeed required in general cases.
Only when there is a need to explore areas with high aleatoric uncertainty, is such pessimistic estimation worth being turned off. [Figure]
E. T HE PERFORMANCE COMPARED WITH RND
RND also follows OFU principle, modeling uncertainty based on network distillation and using it as an intrinsic motivation signal to facilitate agent exploration.
For the sake of fairness, we implement
RND based on DSAC, denoted by DSAC+RND in the following, and evaluate it on 3 standard
Mujoco tasks and 3 noisy Mujoco tasks.
We show the results in the following.
[Figure]
[Table]
[Figure]
[Table]
For the complex task Ant-v2, RND brings much greater improvement by facilitating exploration based on the DSAC.
For the other simpler tasks, RND does not bring signiﬁcant performance improvement.
This experiment demonstrates to some extent the effectiveness of RND exploration on several Mujoco tasks, but at the same time, our algorithm OVD-Explorer is still better.
E. A NALYSIS ABOUT EXPLORATION PROCESS OF OVD-E XPLORER
In the following, we further verify from statistical analysis that OVD-Explorer is in compliance with our raised exploration principle, i.e., OVD-Explorer can achieve better trade-off between optimistic exploration and effectively avoiding exploring the areas with high aleatoric uncertainty .
We show the values of uncertainty estimations and our exploration objective (mutual information) at different stages during the training processes of two trials with different noise settings in ﬁgures.
In Figure 12, the environment noise is set lower around the goal, noting that the darker background color in the map represents higher aleatoric uncertainty, and the red dot represents the coordinate of the current state.
The performance under this trail is given and the agent hardly ever reaches the goal before the 1000th epoch.
Therefore, in the early stage, the aleatoric uncertainty is inaccurate and remains very low, as the value distribution shows little divergence.
The ﬁgure also shows that our exploration objective (in green) is high when the epistemic uncertainty is high.
So before the 1000th epoch, the exploration is guided by epistemic uncertainty, which follows the OFU principle.
Later, once the goal has been explored, the aleatoric uncertainty is properly modelled, i.e., the aleatoric uncertainty towards left is larger than right at current state (see epoch 1240).
Then the mutual information value towards left is lower than right, although the epistemic towards left is higher.
It indicates that OVD-Explorer can property balance epistemic uncertainty and aleatoric uncertainty, and effectively avoid to explore the areas with higher aleatoric uncertainty.
In Figure 13, the environment noise is set higher around the goal.
At the stage before the 1000th epoch, the exploration guidance is similar to Figure 12.
The agent would hardly estimate the accurate aleatoric uncertainty without obtaining any reward.
In the later stage, at the 1240th epoch, OVDExplorer suggests exploring to the right, even though it has been recognized that the environmental uncertainty on the right is high.
This is because the epistemic unertainty dominates under the mutual information.
In contrast, at the 1249th epoch, when the action towards right has been explored much, the signiﬁcant higher aleatoric uncertainty towards right dominates.
Therefore, following the mutual information, the action towards left is preferred.
This demonstrates the trade off that OVD-Explorer make, which satisﬁes our raised exploration principle.
[Figure]
$OHDWRULFXQFHUWDLQW\ 0XWXDOLQIRUPDWLRQ
[Figure] [Figure]