R EWARDING E PISODIC V ISITATION D ISCREPANCY FOR E XPLORATION IN R EINFORCEMENT L EARNING
A BSTRACT
Exploration is critical for deep reinforcement learning in complex environments with high-dimensional observations and sparse rewards.
To address this problem, recent approaches propose to leverage intrinsic rewards to improve exploration, such as novelty-based exploration and prediction-based exploration.
However, many intrinsic reward modules require sophisticated structures and representation learning, resulting in prohibitive computational complexity and unstable performance.
In this paper, we propose Rewarding Episodic Visitation Discrepancy (REVD), a computation-efﬁcient and quantiﬁed exploration method.
More specifically, REVD provides intrinsic rewards by evaluating the R ´ enyi divergence-based visitation discrepancy between episodes.
To estimate the divergence efﬁciently, a k -nearest neighbor estimator is utilized with a randomly-initialized state encoder.
Finally, the REVD is tested on Atari games and PyBullet Robotics Environments.
Extensive experiments demonstrate that REVD can signiﬁcantly improve the sample efﬁciency of reinforcement learning algorithms and outperform the benchmarking methods.
I NTRODUCTION
Efﬁcient and complete exploration remains one of the signiﬁcant challenges in deep reinforcement learning.
Inspired by human learning patterns, recent approaches propose to leverage intrinsic motivation to improve exploration, especially when the agent receives sparse or missing rewards in real-world scenarios (Schmidhuber, 1991; Oudeyer & Kaplan, 2008; Oudeyer et al., 2007).
Intrinsic rewards encourage the agent to visit novel states (Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017; Burda et al., 2019b) or to increase its knowledge and prediction accuracy of the environment dynamics (Stadie et al., 2015; Pathak et al., 2017;
Yu et al., 2020; Houthooft et al., 2016;
Oh et al., 2015).
For instance, Ostrovski et al.
; Bellemare et al.
(2016) leveraged a density model to evaluate the novelty of states, and the intrinsic rewards are deﬁned to be inversely proportional to the visiting frequencies.
As a result, the agent is motivated to revisit the infrequently-seen states and realize comprehensive exploration.
In contrast, Stadie et al. (2015); Pathak et al.
Yu et al.
(2020) designed an auxiliary model to learn the dynamics of the environment, and the prediction error is utilized as the intrinsic reward.
Furthermore, Burda et al.
(2019a) performed experiments on Atari games only using the prediction-based intrinsic rewards, demonstrating that the agent could also achieve remarkable performance.
However, the aforementioned methods cannot provide sustainable exploration incentives because the exploration bonus will vanish as the agent gets to know the environment.
As a result, the agent may miss the downstream learning opportunities and fail to learn in hard-exploration tasks.
To address the problem, Badia et al.
(2020) proposed a never-give-up (NGU) framework that produces intrinsic rewards composed of episodic and long-term state novelty.
The episodic state novelty is evaluated using an episodic memory and pseudo-count method, encouraging the agent to visit diverse states in each episode.
Since the memory is erased at the beginning of each episode, the intrinsic reward will not vanish across the episodes.
In contrast, Raileanu & Rockt¨aschel (2020) proposed a more straightforward method entitled rewarding-impact-driven-exploration (RIDE).
RIDE takes the difference between two consecutive encoded states as the intrinsic reward, encouraging the agent to choose actions that result in large state changes.
However, NGU and RIDE are expensive and un-
[Figure]
Current Episode
Previous Episode stable because they rely heavily on auxiliary models.
Poor representation learning may incur severe performance loss, making it difﬁcult to be applied to arbitrary tasks.
In this paper, we propose a novel intrinsic reward method entitled R ewarding E pisodic V isitation
D iscrepancy (REVD), which provides powerful exploration incentives with simple, computationefﬁcient, and robust architecture.
Our main contributions can be summarized as follows:
• REVD requires no memory model or database, which overcomes the problem of vanishing intrinsic rewards and provides sustainable exploration bonuses; • REVD signiﬁcantly promotes the sample efﬁciency of reinforcement learning algorithms without introducing any representation learning and additional models; • REVD provides a quantiﬁed and intuitive metric to evaluate the exploration change between episodes; • Extensive experiments are performed to compare the performance of REVD against existing methods using Atari games and PyBullet Robotics Environments.
Simulation results demonstrate that REVD achieves remarkable performance and outperforms the benchmarking schemes.
R ELATED W ORK 2. N OVELTY -B ASED E XPLORATION
Novelty-based exploration methods encourage the agent to explore novel states or state-action pairs.
Strehl & Littman (2008) utilized the state visitation counts as the exploration bonus in tabular setting and provided theoretical justiﬁcation for its effectiveness.
Bellemare et al.
; Machado et al.
(2020); Martin et al.
; Bellemare et al.
; Ostrovski et al.
; Tang et al.
Burda et al.
(2019b) extended this method to environments with high-dimensional observations.
More speciﬁcally, Burda et al.
(2019b) employed a predictor network and target network (ﬁxed and randomly-initialized) to record the visited states and allocated higher rewards to infrequently-seen states.
Martin et al.
(2017) proposed a φ -pseudo count method by exploiting the feature space of value function approximation, which can estimate the uncertainty related to any state.
Machado et al.
(2020) designed the exploration bonus using the norm of the successor representation of states, and improved the exploration by capturing the diffusion properties of the environment.
However, these novelty-based approaches suffer from vanishing intrinsic rewards and cannot maintain exploration across episodes.
Our method can effectively address this problem because it requires no additional memory models or database.
2. P REDICTION -B ASED E XPLORATION
Prediction-based exploration methods encourage the agent to take actions that reduce the uncertainty of predicting the consequence of its own actions (Stadie et al., 2015; Pathak et al., 2017; Burda et al., 2019a; Yu et al., 2020).
Pathak et al.
(2017) proposed an intrinsic-curiosity-module (ICM) framework, which uses an inverse-forward pattern to learn a representation space of state and deﬁned the intrinsic reward as the error of predicting the encoded next-state.
A similar representation learning technique is also leveraged in RIDE (Raileanu & Rockt¨aschel, 2020).
In particular, Yu et al.
(2020) introduced the intrinsic reward concept to imitation learning and redesigned the ICM using the variational auto-encoder (Kingma & Welling, 2014).
However, prediction-based approaches also suffer from vanishing intrinsic rewards and sophisticated representation learning, making it expensive and unstable.
Our method follows (Seo et al., 2021) and takes a randomly-initialized neural network to encode the state space, which is cheap and robust to be applied to arbitrary tasks.
2. C OMPUTATION -E FFICIENT E XPLORATION
Computation-efﬁcient exploration aims to provide intrinsic rewards in a cheap, robust, and generalized pattern (Seo et al., 2021; Mutti et al., 2021; Yuan et al., 2022).
Seo et al.
(2021) suggested maximizing the state entropy and designed a random-encoder-for-efﬁcient-exploration (RE3) method, which uses a k -NN estimator to make efﬁcient entropy estimation and divides the sample mean into particle-based intrinsic rewards.
RE3 can signiﬁcantly promote the sample efﬁciency of model-based and model-free reinforcement learning algorithms without any representation learning.
However, Zhang et al.
(2021a) indicated that maximizing the Shannon entropy may lead to a policy that visits some state-action pairs with a vanishing probability and proposed to maximize the
R´enyi entropy of the state-action distribution (MaxR´enyi).
Yuan et al.
(2022) combined the RE and MaxR´enyi by introducing the k -NN estimator and proposed a R´enyi state entropy maximization method, and discussed k -value selection and state space encoding in detail.
However, P ´ oczos &
Schneider (2011) found that the k -NN estimators employed in their work may lead to severe estimation errors because of the inappropriate use of the reverse Fatou lemma.
In this paper, we leverage an improved k -NN estimator proposed by (P ´ oczos & Schneider, 2011) to make an efﬁcient divergence estimation.
Our method also provides an intuitive and quantiﬁed metric to reﬂect the exploration change between episodes.
A closely related work to us is MADE proposed by (Zhang et al., 2021b) that maximizes the deviation from the explored regions.
However, MADE needs to estimate the visitation density over state-action pairs, which is non-trivial in complex environments with highdimensional observations.
Therefore, our method is more efﬁcient and can be easily deployed in arbitrary tasks.
P RELIMINARIES 3. P ROBLEM F ORMULATION
In this paper, we study the reinforcement learning problem considering a single agent Markov decision process (MDP), which can be deﬁned as a tuple M = (cid:104)S , A , P, ˇ r, ς ( s 0 ) , γ (cid:105) (Bellman, 1957).
Here, S is the state space,
A is the action space, P : S ×
A → Π( S ) is the state-transition function, where Π( S ) is the space of probability distributions over S .
In addition, ˇ r : S ×
A → R is the extrinsic reward function, ς ( s 0 ) is the initial state distribution, and γ ∈ [0 , 1) is a discount factor.
In the sequel, we denote the intrinsic reward by ˆ r to distinguish it from the extrinsic reward.
Denote by π ( a | s ) the policy of the agent, the objective of reinforcement learning is to learn a policy that maximizes the expected discounted return R t =
E π (cid:104)(cid:80) Tk =
γ k ˇ r t + k +
(cid:105) .
3. R´ ENYI D IVERGENCE
In this paper, we introduce R´enyi divergence to evaluate the visitation discrepancy between different episodes, which is a generalized and ﬂexible distance measure.
In particular, R´enyi divergence of order 1 equals Kullback–Leibler divergence and R´enyi divergence of order 0 .
5 is a function of the square Hellinger distance (Van Erven & Harremos, 2014).
Let X ∈ R
d be a random vector that has two density functions p ( x ) , q ( x ) with respect to Lebesgue measure on R d , and let { x ∈ R d : p ( x ) , q ( x ) > 0 } be the support set.
The R ´ enyi divergence of order α ∈ R \ { 1 } is deﬁned as
[Equation]
Since it is non-trivial to evaluate the metric when p and q are unavailable, the following k -NN estimator can be employed for efﬁcient estimation.
Let X =
{ X i } N i =1 be i.i.d samples from the distribution with density p and Y =
Mi be i. samples from the distribution with density q , we estimate the R´enyi divergence as follows (P´oczos & Schneider, 2011; Kandasamy et al., 2015):
[Equation]
where µ k ( X i , X ) is the Euclidean distance between X i and its k -th nearest neighbor in X , ν
k ( X i , Y ) is the Euclidean distance between X i and its k -th nearest neighbor in Y , C k,
[Equation]
k > | α − 1 | , and Γ( · ) is the Gamma function.
REVD
In this section, we present the REVD:
R ewarding E pisodic V isitation D iscrepancy, a computationefﬁcient framework for providing powerful and sustainable exploration incentives in reinforcement learning.
The key insight of REVD is to encourage exploration by providing intrinsic rewards based on the episodic visitation discrepancy.
Consider the on-policy learning scenario in which the policy is updated at the end of each episode, and let ˜ π
(the former one) and π denote the learned policies of two consecutive episodes.
The episodic visitation discrepancy can be evaluated using the R ´ enyi divergence:
[Equation]
where ρ π is the state distribution induced by the policy π .
Therefore, if the agent ﬁnds novel states as compared to the previous episode, it will lead to a large divergence.
This metric reﬂects the change of episode-level exploration status.
In the exploration phase of reinforcement learning, the agent is expected to visit as many distinct states as possible within limited iterations.
It is feasible to encourage exploration by maximizing D α ( ρ π (cid:107) ρ ˜ π ) .
4. I NTRINSIC R EWARD D ESIGN
Next, we transform Eq.
(3) into computable intrinsic rewards.
Denote by { ˜ s i } Ti =1 and { s i } Ti =1 the visited states sets of two consecutive episodes, the Eq.
(3) can be approximated using Eq.
(2) as
[Equation]
where e i = f φ ( s i ) is an encoding vector produced by a ﬁxed and randomly-initialized encoder,
E = { f φ ( s i ) } Ti =1 , and α ∈ (0 , 1) .
Therefore, we deﬁne the intrinsic reward that takes each transition as a particle:
[Equation]
where (cid:15) is a ﬁxed and small constant.
A critical problem of this formulation is that the agent may obtain more intrinsic rewards by lingering in a small area.
To address the problem, we consider reformulating the intrinsic reward as
[Equation]
where L ( E ) =
tanh (cid:16) 1 T (cid:80) Ti =0 µ 1 ( e i , E ) (cid:17) is a scaling coefﬁcient.
Here we use the average distance between the states and their nearest neighbors to characterize the intra-episode difference.
As a result, the agent lingering in a small region leads to a small L , and the derived intrinsic reward will be decreased.
We use the following example to further demonstrate the effectiveness of L .
Figure 2 illustrates the trajectories of three episodes.
Consider k = 3 , (cid:15) =
0 .and
α = 0 .
5 , the average intrinsic reward computed by Eq.
(6) of episode (b) and episode (c) is 0.
and 0.67, respectively.
In episode (b), the agent visited unseen and distinct states when compared to the reference episode, which emphasizes both the intra-episode exploration and exploration across episodes.
In contrast, the agent visited novel and repetitive states in episode (c), which will be punished because of low intra-episode difference.
S
S S
[Figure]
4. T RAINING O BJECTIVE
Equipped with the intrinsic reward, the total reward of each transition ( s t , a t , s t +1 ) is computed as
[Equation]
where λ t = λ 0 (1 − κ ) t ≥ 0 is a weighting coefﬁcient that determines the exploration preference, where κ is a decay rate.
In particular, this intrinsic reward can be used to perform unsupervised pre-training in the absence of extrinsic rewards ˇ r .
Then the pre-trained policy can be employed in the downstream task adaptation with extrinsic rewards.
Letting π θ denote the policy represented by a neural network with parameters θ , the training objective is to maximize the expected discounted return
E π θ (cid:104)(cid:80) Tk =
γ k r total t + k +1 (cid:105) .
Finally, the detailed workﬂow of REVD is summarized in Algorithm 1.
E XPERIMENTS
We designed the experiments to answer the following questions:
• Can REVD improve the sample efﬁciency and policy performance of reinforcement learning algorithms in complex environments with high-dimensional observations?
(See Figure 3) •
How does REVD compare to computation-efﬁcient exploration approaches and other exploration approaches empowered by auxiliary models?
(See Figure 4 and Figure 8) •
Can REVD achieve remarkable performance in sparse-reward setting?
(See Figure 5 and Figure 9) • Can REVD also improve the sample efﬁciency in continuous control tasks?
(See Figure 7, Figure 8, and Figure 9)
5. A TARI G AMES 5.1. S ETUP
We ﬁrst tested REVD on six Atari games with graphic observations and discrete action space (Brockman et al., 2016).
To evaluate the sample efﬁciency of REVD, two representative reinforcement learning algorithms were introduced to serve as baselines, namely the Advantage Actor-Critic (A2C) and Proximal Policy Optimization (PPO) (Schulman et al., 2017; Mnih et al., 2016).
For comparison with computation-efﬁcient exploration, we selected RE3 (Seo et al., 2021) as the benchmarking scheme, which maximizes the state entropy using a non-parametric entropy estimator.
For comparison with exploration approaches empowered by auxiliary models, we selected RIDE (Raileanu &
Rockt¨aschel, 2020) that uses the difference between consecutive states as intrinsic rewards.
A brief introduction of RE3 and RIDE can be found in Appendix A.
The following results were obtained by setting k = 5 , α = 0 .
5 , λ 0 = 0 .
1 , κ = 0 .
00001 and (cid:15) = 0 . 0001 , and more details are provided in
Appendix B.
5.1. R ESULTS
[Figure]
Figure 3 illustrates the comparison of the average episode return of six Atari games.
It is obvious that REVD signiﬁcantly improved the sample efﬁciency of PPO in various tasks.
In particular,
PPO+REVD achieved an average episode return of 52,000 in Demon Attack task, producing a giant performance gain when compared to the vanilla PPO agent.
In contrast, the vanilla A2C agent achieved low performance in these tasks.
But REVD also promoted its sample efﬁciency in most tasks, demonstrating the effectiveness of REVD
[Figure]
t
[Figure]
Next, we compared REVD with other exploration methods and considered combining them with
PPO agent, i.e. , PPO+RE3 and PPO+RIDE.
As shown in Figure 4, PPO+REVD also achieved the highest policy performance in all tasks.
As a computation-efﬁcient method, PPO+RE3 achieved the second highest performance in four tasks.
In contrast, PPO+RIDE achieved the second highest performance in two tasks.
It is clear that the exploration-empowered agent consistently obtained higher performance than the vanilla PPO agent.
Figure 3 and Figure 4 demonstrate that REVD can effectively promote the sample efﬁciency and policy performance of reinforcement learning algorithms without introducing any auxiliary models and representation learning.
Furthermore, we tested REVD considering the sparse-reward setting, in which the original extrinsic rewards will be randomly set as zero.
Figure 5 demonstrates that REVD can still achieve superior sample efﬁciency in sparse-reward setting.
In particular, REVD produced an oscillatory learning curve in Assault task, which indicates that REVD provided aggressive exploration incentives.
The combination of PPO and RE3 also obtained remarkable performance in all the games, which demonstrates the of high effectiveness and efﬁciency of computation-efﬁcient exploration.
[Figure]
5. P Y B ULLET R OBOTICS E NVIRONMENTS
[Figure]
5.2. S ETUP
Next
[Figure]
we tested REVD using physical simulation tasks of PyBullet Robotics Environments (See Figure 6) Coumans & Bai (2016–2018).
We also selected A2C and PPO as the baselines and compared the sample efﬁciency of REVD with RE3 and RIDE.
The following results were obtained by setting k = 3 , α = 0 .
5 , λ 0 = 0 .
1 , κ = 0 .
00001 and (cid:15) = 0 . 0001 , and more details are provided in Appendix
C
[Figure]
5. R ESULTS
[Figure]
Figure 7 illustrates the comparison of the average episode return of six PyBullet tasks.
REVD also signiﬁcantly promoted the sample efﬁciency of PPO and A2C in various tasks.
In Walker 2D task, PPO+REVD obtained the highest average episode return, while the vanilla PPO agent prematurely fell into local optima.
In Cart Pole task, PPO+REVD solved the problem using the minimum environment steps.
For Hopper and Humanoid task, PPO+REVD achieved the highest convergence rate, demonstrating the powerful exploration capability of REVD.
In contrast, A2C achieved low performance in Half Cheetah and Humanoid task.
But A2C+REVD also achieved higher performance than the vanilla A2C agent.
Figure 8 illustrates the comparison of REVD with other exploration methods, and PPO+REVD consistently achieved the highest sample efﬁciency in most tasks.
Finally, we also tested REVD in
[Figure]
the sparse-reward setting.
Figure 9 demonstrates that REVD also outperformed the benchmarking methods with higher policy performance.
Therefore, REVD can effectively improve the sample efﬁciency in both the continuous and discrete control tasks
[Figure]
C ONCLUSION
In this paper
[Figure]
we propose a novel and computation-efﬁcient exploration method entitled REVD, which provides high-quality intrinsic rewards without auxiliary models or representation learning.
Since REVD computes intrinsic rewards based on episodic visitation discrepancy, it can overcome the problem of vanishing rewards and provides sustainable exploration incentives.
In particular,
REVD provides a quantiﬁed metric that reﬂects the episode-level exploration change.
Finally, the
REVD is tested on Atari games and PyBullet Robotics Environments.
The simulation results demonstrate that REVD and can signiﬁcantly improve the sample efﬁciency and outperform the benchmarking schemes.
It can be envisaged that the computation-efﬁcient exploration will be widely applied in arbitrary tasks due to its high efﬁciency and convenience.
This work is expected to facilitate and inspire more consequent research on computation-efﬁcient exploration methods.