Redeeming Intrinsic Rewards via Constrained Policy Optimization
Abstract
State-of-the-art reinforcement learning (RL) algorithms typically use random sampling (e.g., ϵ -greedy) for exploration, but this method fails in hard exploration tasks like Montezuma’s Revenge.
To address the challenge of exploration, prior works incentivize the agent to visit novel states using an exploration bonus (also called an intrinsic reward or curiosity).
Such methods can lead to excellent results on hard exploration tasks, but can suffer from intrinsic reward bias and underperform when compared to an agent trained using only task rewards.
This performance decrease occurs when an agent seeks out intrinsic rewards and performs unnecessary exploration even when sufficient task reward is available.
This inconsistency in performance across tasks prevents the widespread use of intrinsic rewards with RL algorithms.
We propose a principled constrained policy optimization procedure that automatically tunes the importance of the intrinsic reward: it suppresses the intrinsic reward when exploration is unnecessary and increases it when exploration is required.
This results in superior exploration that does not require manual tuning to balance the intrinsic reward against the task reward.
Consistent performance gains across sixty-one ATARI games validate our claim.
The code is available at https://github.com/Improbable-AI/eipo .
Introduction
The goal of reinforcement learning (RL) [1] is to find a mapping from states to actions (i e., a policy) that maximizes reward.
At every step of learning, an agent is faced with a question: has the maximum possible reward been achieved?
In many practical problems, knowledge about optimal rewards is unavailable.
Even when the maximum achievable reward is known, if the current policy is sub-optimal then the agent is faced with another question: would spending time improving its current strategy lead to higher rewards ( exploitation ), or should it attempt a different strategy in the hope of discovering potentially higher rewards ( exploration )?
Pre-mature exploitation is akin to getting stuck in a local-optima and precludes the agent from exploring.
Too much exploration on the other hand can be distracting, and prevent the agent from perfecting a good strategy.
Resolving the exploration-exploitation dilemma [1] is therefore essential for data/time efficient policy learning.
In simple decision making problems where actions do not affect the state (e.g. bandits or contextual bandits [2]), provably optimal algorithms for balancing exploration against exploitation are known.
However, in the general settings where RL is used, such algorithms are unknown.
In the absence of methods that work well in both theory and practice, state-of-the-art RL algorithms rely on heuristic exploration strategies such as adding noise to actions or random sampling of sub-optimal actions
[Figure]
i
[Figure]
i c
[Figure] [Figure]
15.0K
[Figure]
RND
Ours (b)
This type of distraction is a possible reason why
(an intrinsic reward method) does not consistently outperform trained using only rewards rewards help in the or is sparse ( but hurt in as
Bond
Our method needed
(e.g., ϵ -greedy).
However, such strategies fail in sparse reward scenarios where infrequent rewards hinder policy improvement.
One such task is the notorious ATARI game, Montezuma’s Revenge [3].
Sparse reward problems can be solved by supplementing the task reward (or extrinsic reward r E ) with a dense exploration bonus (or intrinsic reward r I ) generated by the agent itself [3–7].
Intrinsic rewards encourage the agent to visit novel states, which increases the chance of encountering states that yield task rewards.
Many prior works [3, 7, 8] show that jointly optimizing for intrinsic and extrinsic reward (i.e., r E + λr I , where λ ≥ 0 is a hyperparameter) instead of only optimizing for extrinsic reward (i.e., λ = 0 ) improves performance on sparse reward tasks such as Montezuma’s revenge [3, 8, 9].
However, a recent study found that using intrinsic rewards does not consistently outperform simple exploration strategies such as ϵ -greedy across ATARI games [10].
The reason is that the joint objective ( r E + λr I ) is biased for | λ | > 0 , and optimizing it does not necessarily yield the optimal policy with respect to the extrinsic reward alone [11].
Fig.
1a illustrates this problem using a toy example.
Here the green triangle is the agent and the blue/pink circles denote the location of extrinsic and intrinsic rewards, respectively.
At the start of training, all states are novel so the pink intrinsic rewards are evenly spread out in the environment and easy to accumulate.
The joint objective of maximizing intrinsic and extrinsic rewards can initially lead the agent to only collect the pink circles and therefore move towards the right along the bottom corridor, essentially distracting the agent from the blue task rewards at the top.
In this example, since it is not hard to find the task reward, better performance is obtained if only the extrinsic reward ( λ = 0 ) is maximized.
The trouble, however, is that in most environments one doesn’t know a priori how to optimally trade off intrinsic and extrinsic rewards (i.e., choose λ ).
A common practice is to conduct an extensive hyperparameter search to find the best λ , as different values of λ are best suited for different tasks (see Fig. 4).
Furthermore, as the agent progresses on a task, the best exploration-exploitation trade-off can vary, and a constant λ may not be optimal throughout training.
In the initial absence of extrinsic rewards, exploration is preferred.
Once enough extrinsic rewards are obtained, the agent might prefer exploiting these rewards instead of exploring further.
The exact dynamics of the exploration-exploitation trade-off is task-dependent, and per-task tuning is tedious, undesirable, and often computationally infeasible.
Consequently, prior works use a fixed λ during training, which our experiment reveals is sub-optimal.
We present an optimization strategy that alleviates the need to manually tune the relative importance of extrinsic and intrinsic rewards as training progresses.
Our method leverages the bias of intrinsic rewards when it is useful for exploration and mitigates this bias when it does not help accumulate higher extrinsic rewards.
This is achieved using an extrinsic optimality constraint that forces the extrinsic rewards earned after optimizing the joint objective to be equal to the extrinsic rewards accumulated by the optimal policy that maximizes extrinsic rewards only.
Enforcing the extrinsic optimality constraint in general settings is intractable because the optimal extrinsic reward is unknown.
We devise a practical algorithm called Extrinsic-Intrinsic Policy Optimization (EIPO) , which uses an approximation to solve this constrained optimization problem (Section 3).
While in principle we can apply EIPO to any intrinsic reward method, we evaluate performance using the state-of-the-art random network distillation (RND) [8].
Fig.
1b presents teaser results on two
ATARI games: (i) Montezuma’s Revenge - where joint optimization with RND (red) substantially outperforms a PPO policy [12] optimized with only extrinsic rewards (black); (ii) James Bondwhere PPO substantially outperforms RND.
These results reinforce the notion that bias introduced by intrinsic rewards helps in some games, but hurts in others.
Our algorithm EIPO (blue) matches the best algorithm in both games, showing that it can leverage intrinsic rewards as needed.
Results across 61 ATARI games reinforce this finding.
Additionally, in some games EIPO outperforms multiple strong baselines with and without intrinsic rewards, indicating that our method can not only mitigate the potential performance decreases caused by intrinsic reward bias, but can also improve performance beyond the current state-of-the-art.
Preliminaries
We consider a discrete-time Markov Decision Process (MDP) consisting of a state space S , an action space A , and an extrinsic reward function R E : S ×
A → R .
We distinguish the extrinsic and intrinsic reward components by E and I , respectively.
The extrinsic reward function R E refers to the actual task objective (e.g., game score).
The agent starts from an initial state s 0 sampled from the initial state distribution ρ 0 : S → R .
At each timestep t , the agent perceives a state s t from the environment, takes action a t sampled from the policy π , receives extrinsic reward r Et = R E
( s t , a t ) , and moves to the next state s t +1 according to the transition function T ( s t +1 | s t , a t ) .
The agent’s goal is to use interactions with the environment to find the optimal policy π such that the extrinsic objective value J E ( π ) is maximized:
max
π J E ( π ) , where J E ( π ) =
E s 0 ,a
[Equation]
π (cid:104)
[Equation]
where γ denotes a discount factor.
For brevity, we abbreviate E s 0 ,a 0 , ···∼ π (cid:104) .
(cid:105) as E π (cid:104) .
(cid:105) unless specified.
Intrinsic reward based exploration strategies[3, 7, 8] attempt to encourage exploration by providing “intrinsic rewards" (or “exploration bonus") that incentivize the agent to visit unseen states.
With the intrinsic reward function R I : S ×
A → R , the optimization objective is:
[Equation]
where λ denotes the intrinsic reward scaling coefficient.
We abbreviate the intrinsic reward at timestep t as r I t =
R I ( s t , a t ) .
State-of-the-art intrinsic reward based exploration strategies [8, 13] often optimize the objective in Eq.
2 using Proximal Policy Optimization (PPO) [12].
Mitigating the Bias of Intrinsic Rewards
Simply maximizing the sum of intrinsic and extrinsic rewards does not guarantee a policy that also maximizes extrinsic rewards.
Formally, arg max π E + I ∈
Π J E
+ I
( π E + I )
̸ = arg max π E ∈ Π J E ( π E ) .
At convergence the optimal policy π ∗ E
+ I = arg max π E
+ I J E
+ I
( π E + I ) could be suboptimal w.r.t.
J E , which measures the agent’s task performance.
Because the agent’s performance is measured using extrinsic reward only, we propose enforcing an extrinsic optimality constraint that ensures the optimal “mixed" policy π ∗ E + I = arg max π E
+ I J E
+ I
( π E + I ) leads to as much extrinsic reward as the optimal “extrinsic" policy π ∗ E = arg
max π E ∈ Π J E
( π E ) .
The resulting optimization objective is:
[Equation]
) subject to
[Equation]
(Extrinsic optimality constraint) .
Solving this optimization problem can be viewed as proposing a policy π E + I that maximizes J E
+ I , and then checking if the proposed π E
+ I is feasible given the extrinsic optimality constraint.
The constrained objective is difficult to optimize because evaluating the extrinsic optimality constraint requires J E ( π ∗ E ) , which is unknown.
We show that this optimization problem can be solved by transforming it into an unconstrained min-max optimization problem using Lagrangian duality (Section 3.1).
We then present an iterative algorithm that solves this min-max optimization by alternating between minimization and maximization (Section 3.2), and we describe its implementation (Section 3.3).
3.1 The Dual Objective: Unconstrained Min-Max Optimization Problem
The Lagrangian dual problem for the primal constrained optimization problem in Eq.
3 is:
[Equation]
where α ∈ R + is the Lagrangian multiplier.
We simplify Eq.
4 by merging J E
+ I ( π )
and J E ( π ) :
[Equation]
The simplification above provides an intuitive interpretation of α : larger values correspond to increasing the impetus on extrinsic rewards (i.e., exploitation).
Substituting J αE + I ( π E + I ) into Eq.and rearranging terms yields the following min-max problem:
[Equation]
3.2 Algorithm for Optimizing the Dual Objective
We now describe an algorithm for solving π E , π E + I , and α in each of the sub-problems in Eq. 5.
Extrinsic policy π E (min-stage).
π E is optimized via the minimization sub-problem, which can be re-written as a maximization problem:
[Equation]
The central challenge in this optimization is that evaluating the objectives J αE + I ( π E + I ) and J E ( π E ) requires sampling trajectories from both policies π E
+ I and π E , which is data inefficient.
Intuitively, if we can assume that the policies are similar, then we can use the trajectories from one policy (say π E + I ) to approximate the return of the other policy ( π E ).
We rely on theory introduced in prior work [14, 15, 12] to make this approximation.
Using the performance difference lemma from [14], the objective αJ E ( π E ) − J αE + I ( π E + I ) can be re-written as (see Appendix A.1.3 for detailed derivation):
[Equation]
where U π E
+ I
[Equation]
( s
[Equation]
, a
[Equation]
:=
[Equation] [Equation]
Now we can use the approximation from [12] to lower bound Eq.
and obtain the following surrogate objective:
[Equation]
where ϵ ∈ [0 , 1] denotes a threshold.
Intuitively, this clipped objective (Eq.
8) penalizes the policy π E that behaves differently from π E + I because overly large or small π E ( a t | s t ) π
E + I ( a t | s t ) terms are clipped.
More details are provided in Appendix A.2.
Mixed policy π E
+ I (max-stage).
The sub-problem of solving for π E
+ I is posed as
[Equation]
We again rely on the approximation from [12] to derive a lower bound surrogate objective for
J αE + I ( π E + I ) − αJ E ( π E ) as follows (see Appendix A.1.2 for details):
[Equation]
where U π E max and V π E E are defined as follows:
[Equation]
Lagrangian multiplier α .
We solve for α by using gradient descent on the surrogate objective derived above.
Let g ( α ) :
=
[Equation]
E + I ) − αJ E ( π E ) .
Therefore, ∇ g ( α ) =
J E ( π E + I ) − J E ( π E ) .
We approximate ∇ g ( α ) using the lower bound surrogate objective:
[Equation]
where A π E ( s t , a t ) =
r Et + γV π E E ( s t +1 )
− V π E E ( s t ) is the advantage of taking action a t at state s t , and then following π E for subsequent steps.
We update α using a step size β (Appendix A.1.4):
[Equation]
3.3 Implementation
Min-max alternation schedule.
We use an iterative optimization scheme that alternates between solving for π E by minimizing the objective in Eq. 6, and solving for π E + I by maximizing the objective in Eq. 9.
We found that switching the sampling policy every iteration hinders performance.
We hypothesize that if switching is too frequent, each policy suffers from a lack of training.
Instead, we switch the stage when the objective value does not improve anymore.
Let J [ i ] be the objective value J
[Equation]
+ I
[Equation]
E + I ) −
[Equation]
E
[Equation]
π E ) at iteration i .
The switching rule can be summarized as:
[Equation]
max_stage ← True , if max_stage =
False , where max_stage is a binary variable indicating whether the current stage is the max-stage.
When the max-stage is done, α is updated using Eq.
Parameter sharing.
The policies π E
+ I and π E are parametrized by two separate multi-layer perceptrons (MLP) with a shared CNN backbone.
The value functions V π E + I E
+ I and V π E E are represented in the same fashion, and share a CNN backbone with the policy networks.
When working with image inputs (e.g., ATARI), sharing the convolutional neural network (CNN) backbone between π E and π E
+ I helps save memory, which is important when using GPUs (in our case, an NVIDIA RTX 3090Ti).
If both policies share a backbone however, maximizing the objective (Eq. 9) with respect to
1: Initialize policies π E
+ I and π E , max_stage[0] ← False , and J [0] ←2: for i = 1 · · · do ▷ i denotes iteration index 3: if max_stage[i - 1] then ▷ Max-stage: rollout by π E and update π E + I 4:
Collect trajectories τ E using π E and compute U π E max ( s t , a t ) ∀ ( s t , a t ) ∈
τ E 5:
Update π E + I by Eq.
10 and π E by auxiliary objective (Section 3.3) 6:
J [ i ] ← J αE + I ( π E + I ) − αJ E ( π E ) 7: max_stage[i] ← J [ i ] − J
[ i − 1] ≤8: else ▷ Min-stage: rollout by π E + I and update π E 9:
Collect trajectories
τ E
+ I using π E + I and compute U π E
+ I min ( s t , a t ) ∀ ( s t , a t ) ∈
τ E
+ I 10:
Update π E by Eq.
8 and π E
+ I by auxiliary objective (Section 3.3) 11:
J [ i ] ← J αE + I ( π E + I ) − αJ E ( π E ) 12: max_stage[i] ← J [ i ] − J
[ i − 1] ≥13: end if 14: if max_stage[i - 1] =
True and max_stage[i]
=
False then 15:
Update α (Eq. 32) ▷ Update when the max-stage is done 16: end if 17: end for π E
+ I might interfere with J E ( π E ) , and impede the performance of π E .
Similarly, minimizing the objective (Eq. 6) with respect to π E might modify J αE + I and degrade the performance of π E
+ I .
Interference can be avoided by introducing auxiliary objectives that prevent decreases in J E ( π E ) and
J αE + I when optimizing for π E
+ I andπ E , respectively.
For example, in the max-stage when updating π E + I to maximize the objective outlined in Eq. 9, max π E + I J αE
+ I ( π E + I )
− αJ E ( π E ) , the auxiliary objective max π E J E ( π E ) can prevent any updates in the shared CNN backbone from decreasing
J ( π E ) .
We do not require any additional hyperparameters to incorporate this auxiliary objective (Appendix A.2.3).
The overall objective is outlined in Appendix A.2.4.
Extrinsic-Intrinsic Policy Optimization (EIPO) - the policy optimization algorithm
we introduce to solve Eq.
5, is outlined in Algorithm 1.
Psuedo-code can be found in Algorithm 2, and full implementation details including hyperparameters can be found in Appendix A.2.
Experiments
While the EIPO framework is agnostic to the choice of intrinsic rewards, we mainly experiment with
RND [8] because it represents the state-of-the-art amongst intrinsic reward methods.
We compare
EIPO against the baselines listed below:
• PPO [12] : The PPO policy is trained using extrinsic rewards only: π ∗ = arg max π ∈ Π J E ( π ) .
• RND [8] : The PPO policy is trained using the sum of extrinsic rewards r Et and intrinsic rewards λr
It generated by RND: π ∗ = arg max π ∈ Π J E + I
( π ) .
For ATARI experiments, we chose a single value of λ that works best across games.
• EN (Ext-norm-RND) :
During our experiments we found that a variant of RND where the extrinsic rewards are normalized using running mean and standard deviation (Appendix A.2.2) outperforms the original RND, especially in games where RND performs worse than PPO.
Our method EIPO is implemented based off of Ext-norm-RND .
• DY (Decay-RND) : Instead of having a fixed trade-off between exploration and exploitation throughout training, dynamically adjusting exploration vs. exploitation can lead to better performance.
In the absence of theory describing the best way to dynamically
adjust exploration vs. exploitation in the general RL setting, a commonly used heuristic is to gradually transition from exploration to exploitation.
One example is ϵ -greedy exploration [16], where ϵ is decayed over the course of training.
Similarly, we propose a variant of Ext-norm-RND where intrinsic rewards are progressively scaled down to eliminate exploration bias over time.
Intrinsic rewards r It are scaled by λ ( i ) , where i denotes the iteration number.
The objective function turns into J E + I
=
E π (cid:104) (cid:80) ∞
t =
γ t ( r Et + λ ( i ) r It ) (cid:105) , rewards Distractor
[Figure]
(a) 0.
2.0 Frames 1e
M e a n e x t r i n s i
c r e t u r n
PPO
RND
EIPO (b) 0.
2.0 Frames 1e 0.
(c) where λ ( i ) is defined as λ ( i ) =
clip ( i I ( λ max − λ min ) , λ min , λ max ) .
Here, λ max and λ min denote the predefined maximum and minimum λ ( i ) , and I is the iteration after which decay is fixed.
We follow the linear decay schedule used to decay ϵ -greedy exploration in DQN [16].
• DC (Decoupled-RND) [17] : The primary assumption in EIPO is that π E
+ I andπ E are similar (Section 3.2).
A recent work used this assumption in a different way to balance intrinsic and extrinsic
rewards [17]: they regularize the mixed policy to stay close to the extrinsic policy by minimizing the Kullback–Leibler (KL) divergence D KL ( π E || π E + I ) .
However, they do not impose the extrinsic-optimality constraint , which is a key component in EIPO.
Therefore, comparing against this method called Decoupled-RND ( DC ) [17] will help separate the gains in performance that come from making the similarity assumption versus the importance of extrinsic-optimality constraint .
We adapted DC to make use of Ext-norm-RND intrinsic rewards:
[Equation]
4.1 Illustrative example
We exemplify the problem of intrinsic reward bias using a simple 2D navigation environment (Fig. 2a) implemented using the Pycolab game engine [18].
The gray sprite denotes the agent, which observes a 5 × 5 pixel window view of its surroundings.
If it reaches the ( black ) location at the top of the map, an extrinsic reward of +1 is provided.
The red circles are randomly placed in the bottom corridor at the start of each episode.
Because these circles randomly change location, they induce RND intrinsic rewards throughout training.
Because intrinsic rewards along the bottom corridor are easier to obtain than the extrinsic reward, optimizing the mixed objective J E + I can yield a policy that results in the agent exploring the bottom corridor (i.e., exploiting the intrinsic reward) without ever discovering the extrinsic reward at the top.
Fig.
2b plots the evolution of the average extrinsic returns during training for PPO, RND, and EIPO across 5 random seeds.
We find that EIPO outperforms both RND and
PPO.
RND gets distracted by the intrinsic rewards (red blocks) and is worse than PPO optimized using only extrinsic rewards.
Notably, PPO performs slightly worse than EIPO, possibly because some seeds struggle to find the extrinsic reward at the top without the guidance of intrinsic rewards.
To understand why EIPO performs better, we plot the evolution of the parameter α that trades-off exploration against exploitation during training (Fig. 2c).
Lower values of α denote that the agent is prioritizing intrinsic rewards (exploration) over extrinsic rewards (exploitation; see Section 3.1).
This plot shows that for the first ∼ 0 .
5 M steps, the value of α decreases (Fig. 2c) indicating that the agent is prioritizing exploration .
Once the agent finds the extrinsic reward (between 0 .
5 M − 1 M steps), the value of α stabilizes which indicates that further prioritization of exploration is unnecessary.
After ∼ 1 M steps the value of α increases as the agent prioritizes exploitation , and extrinsic return increases (Fig. 2b).
These plots show that EIPO is able to dynamically trade-off exploration against
RND EN
DY
DC Ours Algorithm A 0.
P ( A
PP O ) 0.54 0.58 0.62 0.57 0.
(a)
PPO RND EN
DY
DC Algorithm B 0.
P ( E P I O - R N D > B ) 0.64 0.60 0.59 0.59 0.
All Atari
[Figure]
Frames (millions) 0.
[Figure]
exploitation during training.
The dynamics of α during training also supports the intuition that EIPO transitions from exploration to exploitation over time.
4.2 Does EIPO generally improve over PPO?
Even though intrinsic reward methods that encourage exploration of novel states are more effective than methods that encourage random actions (e.g., entropy regularization in PPO) [3] in hard exploration tasks, intrinsic reward methods do not consistently outperform PPO when evaluated across diverse tasks [10].
As a result, intrinsic reward methods are not part of the general RL toolkit.
The results in the 2D navigation experiment (Fig. 2) suggest that EIPO mitigates the drawbacks of intrinsic reward based exploration and might help realize their full potential across tasks.
To test if
EIPO can indeed “redeem" intrinsic reward based exploration we ran experiments on ATARI games from the Arcade Learning Environment (ALE) [19], which is the de-facto benchmark for exploration methods [3, 10].
We used a “probability of improvement" metric P ( A ≥ PPO )
[20] with a 95% -confidence interval to judge if algorithm A consistently performs on par or better than PPO across all ATARI games (details in Appendix A.4).
These statistics are calculated using the median of extrinsic returns over the last hundred episodes for at least 5 random seeds for each method and game.
Each method is terminated when either PPO or RND converges (details in Appendix A.4.1).
Comparison to PPO.
If P ( A ≥ PPO )
= 0 .
5 , it indicates that on some tasks A is better than PPO, and on others PPO is better than A .
In other words, a value of 0.5 would indicate that algorithm A is not better than PPO.
Results in Fig.
3a show that P ( RND ≥ PPO )
= 0 .
54 indicating inconsistent performance gains of RND over PPO, supporting prior works [10].
Ext-norm-RND ( EN ) does better, suggesting that normalizing extrinsic rewards helps the joint optimization of intrinsic and extrinsic rewards.
Like EIPO, Decoupled-RND assumes that π E and π E +
I are similar, but Decoupled-RND is not any better than Ext-norm-RND when compared against PPO.
This suggests that the similarity assumption on its own does not improve performance.
We find that P ( EIPO ≥ PPO )
= 0 .
71 , indicating that EIPO indeed outperforms PPO in the majority of trials, and that the extrinsic-optimalityconstraint plays an important role in improving performance.
Strict comparison against baselines.
As all our baselines improve upon vanilla RND, we compare
EIPO with each baseline pairwise in terms of “strict probability of improvement".
Strict probability of improvement P ( A > B ) (Appendix A.4.2) measures the likelihood that an algorithm A is strictly better than an algorithm B .
Fig.
3b shows P ( EIPO-RND > B ) ∀ B .
This metric shows that EIPO outperforms all the baselines by more than 50% .
As suggested in [20], P ( EIPO-RND > B ) > 0 .indicates that a result has statistical significance.
We can therefore conclude that EIPO generally performs better than all other baselines across all 61 ATARI games.
An interesting note is that
Decay-RND ( DY ) has higher probability of improvement than Ext-norm-RND .
This indicates that decaying the scale of intrinsic rewards over time is also a viable method to alleviate the performance drop of RND in ATARI tasks, despite the notable drawback that the decay schedule has to be tuned beforehand.
Comparison of alpha dynamics across environments.
Decay-RND requires hand-tuning the decay schedule of λ to control the weighting between extrinsic r E and intrinsic rewards r
I (i.e., r E + λr I ).
[Figure]
[Figure] [Figure] [Figure] By contrast, the Lagrangian multiplier α in EIPO adjusts the relative importance between both reward functions automatically.
Fig.
3c shows that the EIPO-optimized trajectory of α during training is quite different across games, indicating that the optimal decay schedule is task-dependent.
As such, it is unlikely that a single decay schedule will work well across all environments.
Additionally, Fig.
3c shows that α decreases over time and yet still outperforms Decay-RND in a handful of environments (see the learning curve on Carnival in Appendix A.8).
This implies that in certain domains it may actually be better to amplify the intrinsic reward rather than decay it.
4.3 Does EIPO outperform tuned RND?
EIPO is meant to eliminate the need to tune the intrinsic reward scaling coefficient λ .
So, one critical question to ask is: how does EIPO performance compare with the performance of RND tuned through an exhaustive hyperparameter search over λ ?
This section answers this question by comparing EIPO with λ from Section 4.2, with RND using different values of λ .
We tune λ for RND in several games where RND substantially under-performs PPO in order to see if EIPO can match or outperform the strongest tuned baseline.
Because we empirically find that the performance of RND is heavily dependent on the difference in magnitude between r E and r I , a variety of λ values are selected for each environment based on the relative difference in magnitudes between intrinsic rewards r I and extrinsic rewards r E in each environment ( λ values are detailed in Appendix A.4.4).
Fig.
4 shows the extrinsic return for each choice of λ across 6 environments.
Overall, no choice of λ consistently generates performance improvements over RND.
For instance, r E ≈ λr I substantially improves
RND in Jamesbond , yet deteriorates RND in YarsRevenge .
This shows that the range of acceptable constant λ values is highly environment-dependent, and could be narrow or potentially nonexistent.
Moreover, the effect of λ is unpredictable.
Intuitively, raising λ increases the weight of intrinsic rewards in the optimization objective and potentially introduces more bias, yet r E < λr I outperforms r E ≫ λr I in an easy exploration game Jamesbond (classified based on the taxonomy introduced in [3]).
By contrast, EIPO consistently performs better than PPO and RND using λ = 1 (used by each baseline in Section 4.2) in all environments, demonstrating that EIPO is far less susceptible to a poorly tuned λ .
4.4 Does EIPO improve over RND only in Easy Exploration Tasks?
Our results indicate that EIPO outperforms RND across all sixty-one ATARI games.
It is possible that this performance increase stems from improvements on easy exploration games (the majority of games) where intrinsic rewards are likely to be distracting, and that EIPO performs worse on hard exploration games.
To address this question we selected games where RND outperforms PPO as a proxy for games with hard exploration.
Evaluations of PPO-normalized scores [20] and 95% confidence intervals are reported in Table 1.
EIPO-RND achieves slightly higher mean score than Ext-norm-RND , but lies within the confidence interval.
This result suggests that EIPO not only mitigates the bias introduced by intrinsic rewards in easy exploration games, but either matches or improves performance in hard exploration tasks.
Discussion
[Table]
EIPO can be a drop-in replacement for any
RL algorithm.
Intrinsic rewards improve the performance of an RL algorithm in hard exploration tasks, yet are not a standard component in state-of-the-art RL algorithms due to inconsistent performance and sensitivity to the scaling of intrinsic rewards.
Intrinsic reward methods do not consistently outperform random sampling based exploration (e.g., ϵ -greedy), which inhibits intrinsic rewards methods from being used by default for exploration.
Furthermore, as we showed in
Section 4.3, intrinsic reward methods are sensitive to the choice of λ , and the best λ is task-dependent.
This requires that practitioners conduct an exhaustive hyperparameter search for each task, which makes training far more expensive than standard methods.
In contrast, EIPO exhibits consistent performance improvements in both hardand easy-exploration tasks with the same initial α across sixty-one ATARI games, demonstrating its potential as a drop-in replacement algorithm that is highly likely to outperform typical random sampling based methods.
Though we performed experiments with PPO, the primary optimization objective (Eq. 3.1) can be optimized by any RL algorithm.
As such, EIPO is agnostic to the RL algorithm being used, making it easily applicable in any RL pipeline.
Also, EIPO is not restricted to discrete action ATARI games.
Appendix 9 presents experimental results testing EIPO in MuJoCo [21], showing that EIPO either matches or exceeds RND and PPO and is also applicable in continuous control.
Limitations.
While EIPO reduces the need for hand-tuning hyperparameters for each task and consistently performs well with the same initial α across 61 ATARI games, the initial α and step size β could depend on the intrinsic reward function.
Using the same α and β used in Section 4, we tested EIPO with another intrinsic reward method - ICM
The results shown in Appendix A. show that in some tasks, EIPO with ICM exceeds or matches the best baseline.
However in some tasks, EIPO-RND can outperform all the baselines while EIPO-ICM fails to do so.
This indicates that different intrinsic reward functions could need different settings of initial α and step size β to work well across tasks.
An interesting direction for future work would be to study the interactions between
EIPO hyperparameters and different intrinsic reward methods.
Potential applications to reward shaping.
Reward shaping adds heuristic rewards on the task’s rewards (i.e., extrinsic rewards) for improving sample efficiency and is a common trick used in a variety of practical applications of RL [22, 23].
Despite its widespread efficacy, it often requires tedious hyperparameter tuning to balance heuristic rewards and task rewards properly; otherwise, performance degrades.
As intrinsic rewards can be regarded as reward shaping, EIPO could better balance heuristic rewards and task rewards and relax the requirements of tuning the importance of both rewards.
Acknowledgments
[Table]
Government
.TheU .S Governmentisauthorizedtoreproduceanddistributereprintsfor Governmentpurposesnotwithstandinganycopyrightnotationherein.
• EricChen developedheuristic-basedmethodsforbalancingintrinsicandextrinsicrewards,whichlaidthegroundsfortheinceptionofEIPO.Cameupwiththe intuitionthatledtotheextrinsic-intrinsicoptimalityconstraint,andprovidedcriticalinputduringEIPOprototyping.
Implementedbaselines,ranexperimentsatscale, andhelpedwithpaperwriting.
• Zhang-Wei Hong conceived of the extrinsic-intrinsic optimality constraint, developed the first prototype of EIPO, derived the optimization objective, ran small-scale experiments,playedanadvisoryroletoEricandtheprimaryroleinpaperwriting.
• JoniPajarinen providedfeedbackonthetechnicalcorrectnessofEIPOformulationandwriting.
• PulkitAgrawal conceivedandoversawtheproject.
Hewasinvolvedinthetechnicalformulation,researchdiscussions,paperwriting,overalladvisingandpositioning ofthework.
A Appendix A.1 Full derivation
We present the complete derivation of the objective function in each sub-problem defined in Section 3.2.
We start by clarifying notation:
A.1.1 Notations
• J E
( π ) =
E s 0 ,a 0 , ···∼ π (cid:104) (cid:80) ∞ t =
γ t r Et (cid:105) , s 0 ∼ ρ 0 , a t ∼ π ( a | s t ) , s t
+1 ∼ T ( s t +1 | s t , a t ) ∀ t >• J E
+ I ( π ) =
E π (cid:104) (cid:80) ∞
t =
γ t ( r Et + r It ) (cid:105) • V πE ( s t ) :
=
E π (cid:104) (cid:80) ∞
t =0 γ t r Et | s 0 =
s t (cid:105) • V πE
+ I ( s t ) :
= E π (cid:104) (cid:80) ∞ t =0 γ t ( r Et + r It ) | s 0 =
s t (cid:105) • V π,αE
+ I ( s t ) :
= E π (cid:104) (cid:80) ∞ t =
γ t ((1 + α ) r
Et + r It ) | s 0 =
s t (cid:105)
A.1.2 Max-stage objective U π E max
We show that the objective (LHS) can be lower bounded by the RHS shown below
[Equation]
We can then expand the LHS as the follows:
[Equation]
For brevity, let r t
=
(1 + α ) r Et + r It and αV π E E ( s t ) =
V t .
Expanding the left-hand of Eq. 13: − V 0 + ∞ (cid:88)
[Equation]
To facilitate the following derivation, we rewrite the objective J αE + I ( π E + I ) − αJ E ( π E ) :
[Equation]
where d π E + I ,γ ρis the discounted state visitation frequency of policy π E
+ I with the initial state distribution
ρ 0 and discount factor γ , defined as:
[Equation]
Note that for brevity, we write E s ∼ d πE + I,γ ρ,a ∼ π (cid:104) .
(cid:105) as E π E
+ I (cid:104) .
(cid:105) instead.
To get rid of the dependency on samples from π E
+ I , we use the local approximation [15, 14] shown below:
[Equation]
The discounted state visitation frequency of π E + I is replaced with that of π E .
This local approximation is useful because if we can find a π 0 such that L αE + I ( π 0 ) =
J αE + I ( π 0 ) , the local approximation matches the target in the first order: ∇ π E + I L αE + I ( π E + I ) | π E + I
[Equation] [Equation]
| π
[Equation]
I =
[Equation]
This implies that if L αE + I ( π E + I ) is improved, J αE + I ( π E + I ) | π E + I
=
π 0 will be improved as well.
Using the technique in Schulman et al.
[15], this local approximation can lower bound J αE
+ I ( π E + I ) − αJ E ( π E ) with an additional KL-divergence penalty as shown below:
[Equation]
where C denotes a constant.
As it is intractable to evaluate D max KL ( π E || π E + I ) , the clipping approach proposed in Schulman et al.
[12] receives wider spread of use.
Following [12], we convert Eq.
22 to a clipped objective shown as follows:
[Equation]
Note that to make use of the approximation proposed in [15, 14], we make the assumption that in the beginning of the max-stage, π E = π E
+ I .
Under this assumption, π E serves as π 0 (see above).
This enables updating π E + I using the local approximation.
We leave relaxing this assumption as future work.
A.1.3 Min-stage objective U π E + I min
We show that the objective (LHS) can be approximated by the RHS shown below
[Equation]
The derivation for the min-stage is quite similar to that of the max-stage.
Thus we only outline the key elements:
[Equation]
Since we empirically find that V π E
+ I ,α E
+ I is hard to fit under a continually changing α , we replace
V π E
+ I ,α E
+ I with V π E
+ I E
+ I in Equation 29, which leads to Eq.
Following the same recipe used in deriving the objective of max-stage gives rise to the following objective:
[Equation]
A.1.4 α optimization
Let g ( α ) :
= max π E + I
∈
Π min π E ∈ Π J αE + I ( π E + I ) − αJ E ( π E ) .
As π E and π E
+ I are not yet optimal during the training process, we solve min α g ( α ) using stochastic gradient descent as shown below:
[Equation] [Equation]
[Equation]
(34) where β is the learning rate of α .
J E ( π E + I ) − J E ( π E ) can be lower bounded using the technique proposed in [12] as follows:
[Equation]
where A π E E ( s t , a t ) :
= r Et + γV π E E ( s t +1 )
− V π E E ( s t ) denotes the extrinsic advantage of π E .
A.2 Implementation details A.2.1 Algorithm
Clipped objective We use proximal policy optimization (PPO)
[12] to optimize the constrained objectives in Equation 6 and Equation 9.
The policies π E and π E
+ I are obtained by solving the following optimization problems with clipped objectives:
• Max-stage:
[Equation]
• Min-stage:
[Equation]
where ϵ denotes the clipping threshold for PPO, and π old E and π old E + I denote the policies that collect trajectories.
We will detail the choices of ϵ in the following paragraphs.
Rearranging the expression for GAE To leverage generalized advantage estimation (GAE) [24], we rearrange U π E max and U π E + I min to relate them to the advantage functions.
The advantage function
A π E E and A π E + I E
+ I are defined as:
[Equation]
As such
[Equation]
we can rewrite U π E max and U π E
+ I min as:
[Equation]
A.2.2 Extrinsic reward normalization
For each parallel worker
[Equation]
we maintain the running average of the extrinsic rewards ¯ r E .
This value is updated in the following manner at each timestep t :
[Equation]
The extrinsic rewards are then rescaled by the standard deviation of ¯ r E across workers as shown below:
[Equation]
A.2.3 Auxiliary objectives
The auxiliary objectives for each stage are listed below:
• Max-stage: We train the extrinsic policy π E to maximize J E ( π E ) using PPO as shown below:
[Equation]
where π old E denotes the extrinsic policy that collects trajectories at the current iteration.
• Min-stage: We train the mixed policy π E + I to maximize J E
+ I
( π E + I ) using PPO as shown below:
[Equation]
where π old E + I denotes the mixed policy that collects trajectories at the current iteration.
A.2.4 Overall objective
In summary
[Equation]
we outline all the primary objectives Eqs.
36 and 37 and auxiliary objectives Eqs.and 45 as follows:
[Equation]
The overall objectives optimized in both stages is summarized as follows:
[Equation]
Clipping the derivative of α
The derivative of α , δα (see Section 3.3), is clipped to be within ( − ϵ α , ϵ α ) , where ϵ α is a non-negative constant.
Codebase We implemented our method and each baseline on top of the rlpyt 2 codebase.
We thank Adam Stooke and the rlpyt team for their excellent work producing this codebase.
Summary We outline the steps of our method in Algorithm 2.
1: Initialize policies π E
+ I and π E and value functions
V π E + I E
+ I and V π E E 2: Set max_stage[0] ← False , and J [0] ←3: for i = 1 · · · do ▷ i denotes iteration index 4: if max_stage[i - 1] then ▷ Max-stage: rollout by π E and update π E
+ I 5:
Collect trajectories τ E using π E 6:
Update π E + I and π E by Eq.7:
Update V π E E (see [24]) 8:
J [ i ] ← J αE + I ( π E + I ) − αJ E ( π E ) 9: max_stage[i] ← J [ i ] − J
[ i − 1] ≤10: else ▷ Min-stage: rollout by π E
+ I and update π E 11:
Collect trajectories
τ E
+ I using π E
+ I 12:
Update π E + I and π E by Eq.13:
Update V π E + I E
+ I
(see [24]) 14:
J [ i ] ← J αE + I ( π E + I ) − αJ E ( π E ) 15: max_stage[i] ← J [ i ] − J
[ i − 1] ≥16: end if 17: if max_stage[i - 1] =
True and max_stage[i]
=
False then 18:
Update α (Eq. 32) ▷ Update when the max-stage is done 19: end if 20: end for
A.2.5 Models
Network architecture Let Conv2D(ic, oc, k, s, p) be a 2D convolutional neural network layer with ic input channels, oc output channels, kernel size k , stride size s , and padding p .
Let
[Table] [Table]
LSTM(n, m) and MLP(n, m) be a long-short term memory layer and a multi-layer perceptron (MLP) with n -dimensional inputs and m -dimensional outputs, respectively.
For policies and value functions, the CNN backbone is implemented as two CNN layers,
Conv2D(1, 16, 8, 4, 0) and Conv2D(16, 32, 4, 2, 1) , followed by an LSTM layer,
LSTM($CNN_OUTPUT_SIZE, 512) .
The policies π E
+ I and π E
+ I , and the value functions V π E + I E
+ I and V π E
+ I E
+ I have separate MLPs that take the LSTM outputs as inputs.
Each policy MLP is MLP(512, |A| ) , and each value function MLP is MLP(512, 1) .
For the prediction networks and target networks in RND , we use a model architecture with three CNN layers followed by three MLP layers.
The CNN layers are defined as follows: Conv2D(1, 32, 8, 4, 0) , Conv2D(32, 64, 4, 2, 0) , and Conv2D(64, 64, 3, 1, 0) , with LeakyReLU activations in between each layer.
The MLP layers are defined as follows: MLP(
*7*64, 512) , MLP(512, 512) , and MLP(512, 512) , with ReLU activations in between each layer.
Hyperparameters The hyperparameters for PPO, RND, and EIPO are listed in Table 2, Table 3, and Table 4, respectively.
A.3 Environment details
Pycolab
• State space S : R 3 × 84 × 84 , 5 × 5 cropped top-down view of the agent’s surroundings, scaled to an 84 × 84 RGB image (see the code in the supplementary materials for details).
• Action space A : { UP , DOWN , LEFT , RIGHT , NO ACTION } .
• Extrinsic reward function R E : See section 4.1.
[Table] [Table]
• State space S : R 1 × 84 × 84 , 84 × 84 gray images.
• Action space A : Depends on the environment.
• Extrinsic reward function R E : Depends on the environment.
A.4 Evaluation details A.4.1 Training length
For most ATARI games, we train each method for 3000 iterations where each iteration consists of 128 ∗ 128 frames and thus in total for 49152000 frames.
Except for Montezuma’s Revenge , we train 6000 iterations.
For Pycolab, we train each method for 1563 iterations where each iteration consists of 64 ∗frames and thus 50 , 000 , 000 frames.
A.4.2 Probability of improvement
We validate whether EIPO prevents the possible performance degradation introduced by intrinsic rewards, and consistently either improves or matches the performance of PPO in 61 ATARI games.
As our goal is to investigate if an algorithm generally performs better than PPO instead of the performance gain, we evaluate each algorithm using the “probability of improvement" metric suggested in [20].
We ran at least 5 random seeds for each method in each environment, collecting the median extrinsic returns within the last 100 episodes and calculating the probability of improvements P ( X ≥ PPO )with 95% -confidence interval against PPO for each algorithm X .
The confidence interval is estimated using the bootstrapping method.
The probability of improvement is defined as:
[Equation]
where x i and y j denote the samples of median of extrinsic return trials of algorithms X and Y , respectively.
We also define strict probability of improvement to measure how an algorithm dominate others:
[Equation]
.4.3 Normalized score
In addition, we report the PPO-normalized score [20] to validate whether EIPO preserves the performance gain granted by RND when applicable.
Let p
X
be the distribution of median extrinsic returns over the last 100 episodes of training for an algorithm X .
Defining p PPO as the distribution of mean extrinsic returns in the last 100 episodes of training for PPO, and p rand as the average extrinsic return of a random policy, then the PPO-normalized score of algorithm X is defined as: p X − p rand p
PPO − p rand .
A.4.4 λ tuning
Table 5 lists the λ values used in Section 4.3.
A.5 RND-dominating games
Table A.5 shows that the mean and median PPO-normalized score of each method with 95%- confidence interval in the set of games where RND performs better than PPO.
The set of games where RND performs better than PPO are listed below:
[Table]
[Table]
• AirRaid • Alien • Assault • Asteroids • BankHeist •
Berzerk • Bowling • Boxing • Breakout • Carnival • Centipede • ChopperCommand • DemonAttack • DoubleDunk • FishingDerby • Frostbite • Gopher • Hero •
Kangaroo •
KungFuMaster • MontezumaRevenge • MsPacman • Phoenix • Pooyan • Riverraid • RoadRunner • SpaceInvaders
P P O
R N D
E N
[Table]
Algorithm B
PPO
RND
EN
DY
DC
Ours
[Table]
• Tutankham • UpNDown • Venture
A.6 Scores for each ATARI game
The mean scores for each method on all ATARI games are presented in Table 7.
A.7 Complete comparison
In addition, Fig. 5 shows the percentage of ATARI games in which one method matches or exceeds the performance of another method with a pairwise comparison.
Note that “percentage of games A ≥ B "
R ( A, B ) is not equivalent to probability of improvements P ( A ≥ B ) .
Let µ A be the mean score of algorithm
A over 5 random seeds in an environment.
R ( A, B ) is defined as: R ( A, B ) =(cid:2) µ A ≥ µ B (cid:3)
N where N denotes the number of ATARI games.
The last row of Fig.
5 shows that EIPO performs better than all the baselines in the majority of games (i.e., R ( EIPO , B ) > 0 . 5 ).
A.8 Complete learning curves
We present the learning curves of each method in Figure 6, and the evolution of α in EIPO in Figureon all ATARI games.
A.9 ICM
In addition to RND, we test our method on ICM [7] - another popular bonus-based exploration method.
The learning curves on 6 ATARI environments can be seen in Fig. 8.
A.10 MuJoCo results
We present the experimental results of EIPO and other baselines on MuJoCo [21] in Figure 9.
It can be seen that our method exceeds or matches PPO in most tasks.
However, as the reward function of
MuJoCo are often dense, there is no visble difference between RND and PPO.
PPO
RND
Ext-norm RND
Decay-RND
Decouple-RND
Ours
Adventure 0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
AirRaid
[Table] [Table]
[Table] [Table]
[Table] [Table]
Alien
[Table] [Table]
[Table] [Table]
[Table] [Table]
Amidar
[Table] [Table]
[Table] [Table]
[Table]
Assault
[Table] [Table]
[Table] [Table]
[Table] [Table]
Asterix
[Table] [Table]
[Table] [Table]
[Table] [Table]
Asteroids
[Table] [Table]
[Table] [Table]
[Table] [Table]
BankHeist
[Table] [Table]
[Table] [Table]
[Table] [Table]
BattleZone 83826.
[Table] [Table]
[Table] [Table]
[Table] [Table]
BeamRider
[Table] [Table]
[Table] [Table]
[Table] [Table]
Berzerk
[Table] [Table]
[Table] [Table]
[Table] [Table]
Bowling
[Table] [Table]
[Table] [Table]
[Table] [Table]
Boxing
[Table] [Table]
[Table] [Table]
[Table] [Table]
Breakout
[Table] [Table]
[Table] [Table]
[Table] [Table]
Carnival
[Table] [Table]
[Table] [Table]
Centipede 5938.
ChopperCommand 8225.
CrazyClimber 151202.
DemonAttack 5678.
DoubleDunk -1.
ElevatorAction 45703.
Enduro 1024.
FishingDerby 35.
Freeway 31.
Frostbite 1011.
Gopher 5544.
Gravitar 1682.
Hero 29883.
IceHockey 6.
Jamesbond 13415.
JourneyEscape -429.
Kaboom 1883.
Kangaroo 6092.
Krull 9874.
KungFuMaster 47266.
MontezumaRevenge 0.
MsPacman 4996.
NameThisGame 11127.
Phoenix 8265.
Pitfall 0.
Pong 20.
Pooyan 5773.
PrivateEye 97.
Qbert 23863.
Riverraid 10231.
RoadRunner 45922.
Robotank 37.
Seaquest 1453.
Skiing -12243.
Solaris 2357.
SpaceInvaders 1621.
StarGunner 21036.
Tennis -0.
TimePilot 19544.
Tutankham 199.
UpNDown 276884.
Venture 102.
VideoPinball 360562.
WizardOfWor 11912.
YarsRevenge 92555.
[Table]
[Figure] [Figure]
[Figure] [Figure]
5K
[Figure]
5K
[Figure]
E x t r i n s i
c r e t u r n
Alien
[Figure]
[Figure] [Figure]
Adventure
[Figure]
AirRaid
[Figure]
performance however, see that RND does perform well (Fig. 6)
This implies that we need more investigation to uncover the relationship between EIPO and intrinsic rewards functions.
[Figure]
A.11 Related Work
Reward design.
Our work is related to the paradigm of reward design.
Meriçli et al.
[25] uses genetic programming to optimize the reward function for robot soccer.
Sorg et al.
[26] learns a reward function for planning via gradient ascent on the expected return of a tree search planning algorithm
(e
[Figure]
g., Monte Carlo Tree Search).
Guo et al.
[27] extends [26] using deep learning, improving tree search performance in ATARI games.
The work [28] learns a reward function to improve the performance of model-free RL algorithms by performing policy gradient updates on the reward function.
Zheng et al.
[29] takes a meta-learning approach to learn a reward function that improves an RL algorithm’s sample efficiency in unseen environments.
Hu et al.
[30] learns a weighting function that scales the given shaping rewards [31] at each state and action.
These lines of work are complimentary to EIPO, which is agnostic to the choice of intrinsic reward and could be used in tandem with a learned reward function.
Multi-objective reinforcement learning.
The problem of balancing intrinsic and extrinsic rewards is related to multi-objective reinforcement learning (MORL) [32–35] since both optimize a policy with the linear combination of two or multiple rewards.
Chen et al.
[32] learns a meta-policy that jointly optimizes all the objectives and adapts to each objective by fine-tuning the meta-policy.
Ultes et al.
[34] search for the desired weighting between dialog length and success rate in conversation AI system using Bayesian optimization.
Abels et al.
[36], Yang et al.
[35] make the policy conditional on the preference of each reward and train the policy using the weighted rewards.
However, MORL approaches could not resolve the issue of intrinsic rewards bias in our problem setting since intrinsic rewards are non-stationary and change over the training time, yet MORL approaches require the reward functions to be fixed.
Also, MORL is aimed at learning a Pareto-optimal policy that satisfies all the objectives, but intrinsic rewards are just auxiliary training signals, and thus, we do not require maximizing intrinsic rewards.