Reconciling intrinsic rewards via constrained policy optimization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The use of intrinsic rewards in policy optimization via reinforcement learning hasled to impressive performance gains on several hard exploration Atari games suchas Montezuma’s Revenge.
However, recent studies show that intrinsic rewardscan bias the policy optimization and cause worse performance than optimizingonly the environment reward in most tasks other than hard exploration ones.
Suchinconsistent performance prevents the use of intrinsic rewards with RL algorithmson a general set of tasks.
We propose using constrained policy optimization toeliminate the detrimental effects of intrinsic rewards while preserving their meritswhen applicable.
Experimental results across 61 Atari games validate our claim. The end result is a superior exploration algorithm that does not require manualtuning to trade intrinsic rewards against extrinsic rewards.
Introduction
Exploration-exploitation is at the core of reinforcement learning (RL) [1].
The central challenge ofexploration-exploitation is to balance the time spent discovering better policies (exploration) andtaking greedy actions to maximize rewards (exploitation).
In the absence of methods that workwell in both theory and practice, state-of-the-art methods rely on strategies such as adding a smallamount of noise to actions or randomly sampling a suboptimal action with small probability (e.g.,✏ -greedy).
However, such strategies fail on notoriously hard exploration tasks such as the Atarigame Montezuma’s Revenge where the rewards are sparse [2].
In sparse reward scenarios, non-zerorewards are infrequent which hinders policy improvement.
A popular approach for solving hard exploration tasks is to supplement the environment (or theextrinsic) reward ( r E ) with an exploration bonus (or intrinsic reward r I ) generated by the agentitself [3–5, 2, 6].
Intrinsic rewards encourage the agent to visit novel states and thus increase thechance of visiting states that yield extrinsic rewards.
Many prior works [6, 2, 7] show that jointlyoptimizing intrinsic and extrinsic rewards (i.e., r E +  r I , where   0 is a hyperparameter) insteadof only optimizing the extrinsic reward (i.e.  = 0 ) improves performance on hard exploration tasks.
Despite success on hard exploration tasks such as Montezuma’s Revenge , a recent study found thatusing intrinsic rewards does not consistently outperform simple exploration strategies such as ✏ -greedyacross Atari games [8].
Whitney et al.
[9] suggest that intrinsic rewards can degrade performancedue to the bias they introduce in the policy optimization objective.
Jointly maximizing intrinsic andextrinsic rewards does not necessarily yield the optimal policy with regard to extrinsic rewards alone.
Fig.
1a illustrates this intuition with the help of a toy example.
Here the green triangle is the agent,and blue/pink circles denote the location of extrinsic and intrinsic rewards, respectively.
Maximizingthe sum of intrinsic and extrinsic rewards can lead the learned policy to move right along the bottomcorridor and therefore be suboptimal with respect to the extrinsic rewards which are located at the
[Figure] top. In such a scenario, better performance will be obtained if the agent only optimizes the extrinsicreward.
The trouble, however, is that a priori one does not know how to trade off (i.e., choose  )intrinsic v/s extrinsic rewards.
A common practice is to ﬁnd the best  using an extensive hyperparameter search.
The problem is thatdifferent values of  are adept for different tasks (see Fig. 4).
Furthermore, as the agent progresseson the task, the exploration-exploitation trade off can vary.
In the initial absence of extrinsicrewards, exploration is preferred.
Once enough extrinsic rewards are obtained, the agent mightprefer exploitation of these rewards instead of exploration.
The exact dynamics of the explorationexploitation trade off are task-dependent, and per-task tuning is tedious, undesirable and oftencomputationally infeasible.
We present an exploration strategy that alleviates the need to manually tune the relative importance ofextrinsic and intrinsic rewards.
Our method leverages the bias of intrinsic rewards when it is usefulfor exploration and mitigates this bias when it does not lead to an increase in extrinsic rewards.
This isachieved using an extrinsic optimality constraint that forces the extrinsic rewards earned by the jointpolicy to be equal to those earned by the optimal policy with respect to the extrinsic rewards.
Directlyoptimizing this constraint in general settings is intractable because the optimal extrinsic reward isunknown.
We devise a practical algorithm which we call Extrinsic-Intrinsic Policy Optimization(EIPO) that uses an approximation to solve this constrained optimization problem (Section 3).
While in principle we can apply EIPO to any intrinsic reward method, we evaluate performance usingthe state-of-the-art, random network distillation (RND) [7].
Fig.
1b presents teaser results on two
Atari games: (i) Montezuma’s revenge on which joint optimization with RND (red) substantiallyoutperforms a PPO policy [10] optimized with only extrinsic rewards (black); (ii) Jamesbond where
PPO substantially outperforms RND.
These results demonstrate that in some games the bias ofintrinsic rewards helps, but in others it can hurt.
In contrast, our algorithm EIPO (blue) matches thebest algorithm in both games, showing that it can leverage intrinsic rewards as necessary.
A large-scalestudy on all 61 Atari games reinforces this ﬁnding.
Additionally, in some games EIPO outperformsmultiple strong baselines with and without intrinsic rewards, indicating that our method both mitigatesperformance degradation resulting from intrinsic reward bias and improves performance beyond thecurrent state-of-the-art.
Preliminaries
We consider a discrete-time Markov Decision Process (MDP) consisting of a state space S , an actionspace A , and an extrinsic reward function R E : S ⇥
A 7! R .
Note that we distinguish the components 67 for extrinsic and intrinsic rewards by E and I , respectively.
The extrinsic reward function R E deﬁnes 68 the objective of the task (e.g., game score).
The agent starts from an initial state s 0 sampled fromthe initial state distribution ⇢ 0 :
S 7!
R .
At each timestep t , the agent perceives a state s t of theenvironment, takes action a t sampled from the policy ⇡ , receives extrinsic reward r Et = R E ( s t , a t ) ,and moves to the next state s t +1 according to the transition function T ( s t +1 | s t , a t ) .
Our goal is to use the experience of interactions with the environment to ﬁnd the optimal policy ⇡ such that theextrinsic objective value J E ( ⇡ ) is maximized:
[Equation]
The  denotes a discount factor.
For brevity, we abbreviate E s 0 ,a 0 , ··· ⇠ ⇡ h .
i [Equation] E ⇡ h .
i unless speciﬁed.
Curiosity-driven exploration methods [6, 7, 2] attempt to ease exploration by providing “intrinsicrewards" to the agent to encourage it to visit unseen states.
With the intrinsic reward function R I : S ⇥
A 7!
R , the optimization objective is:
[Equation]
where  denotes the intrinsic reward scaling coefﬁcient.
We abbreviate intrinsic reward at timestep t asr It =
R I ( s t , a t ) .
State-of-the-art curiosity-exploration strategies [7, 11] often optimize the objectivein Eq.
2 using Proximal Policy Optimization (PPO) [10].
Using PPO, we take trajectories ⌧ ⇡ old :=( s 0 , a 0 , r E 0 , s 1 , a 1 , r E 1 , · · · ) generated by the current policy ⇡ old and intrinsic rewards ( r I 0 , r I 1 , · · · )from an arbitrary R I to solve a new policy ⇡ for max ⇡ 2 ⇧ J E + I ( ⇡ ) .
⇡ is then set as ⇡ old and used tosample trajectories for solving another new policy.
Mitigating the Bias of Intrinsic Rewards
Simply maximizing the sum of intrinsic and extrinsic rewards does not guarantee a policy that alsomaximizes the extrinsic reward.
Formally, put arg max ⇡ E + I[-
 ⇧-] J E
+ I ( ⇡ E
+ I ) 6 = arg max ⇡ E 2 ⇧ J E ( ⇡ E )
At convergence, the optimal policy ⇡ ⇤ E + I = arg max ⇡ E
+ I J E
+ I ( ⇡ E + I ) could be suboptimal w.r.t.
J E that deﬁnes the performance of a policy on the task.
Because the performance of the agent ismeasured only based on extrinsic reward, we propose to enforce an extrinsic optimality constraint toensure that the optimal “mixed" policy ⇡ ⇤ E + I = arg max ⇡ E
+ I J E
+ I ( ⇡ E + I ) leads to as much extrinsicreward as the optimal “extrinsic" policy ⇡ ⇤ E = arg max ⇡ E 2 ⇧ J E (
⇡ E ) .
The resulting optimizationobjective is the following:
[Equation]
[Equation]
(Extrinsic optimality constraint)
[Equation]
Solving this optimization problem can be viewed as proposing a policy ⇡ E + I that maximizes J E + I ,and checking if the proposed ⇡ E
+ I is feasible given the extrinsic optimality constraint.
The constrained objective is difﬁcult to optimize because evaluating the extrinsic optimality constraintrequires J E ( ⇡ ⇤ E ) , which is unknown.
We show that this optimization problem can be solved bytransforming it into an unconstrained min-max optimization problem using Lagrangian duality(Section 3.1).
Next, we present an iterative algorithm for min-max optimization by alternatingbetween maximization and minimization in Section 3.2 and an implementation in Section 3.3.
3.1 The Dual Objective: Unconstrained Min-Max Optimization Problem
The Lagrangian dual problem for the primal constrained optimization problem in Eq.
3 is:
[Equation]
where ↵ 2 R is the Lagrangian multiplier.
We simplify Eq.
4 by merging J E + I ( ⇡ ) and J E ( ⇡ ) : J ↵ E [Equation] I
[Equation]
I
[Equation]
Substituting J ↵ E
+ I ( ⇡ E + I ) into Eq.
4 and rearranging terms yields the following min-max problem:
[Equation]
3.2 Algorithm for Optimizing the Dual Objective
We now describe an algorithm for solving ⇡ E + I , ⇡ E , and ↵ in each of the subproblems in Eq. 5.
Extrinsic policy ⇡ E (min-stage).
⇡ E is optimized via the minimization sub-problem, which can bere-written as a maximization problem:
[Equation]
The central challenge in this optimization is that evaluating objectives J ↵ E + I ( ⇡ E + I ) and J E ( ⇡ E )requires sampling trajectories from both policies ⇡ E + I and ⇡ E which is data inefﬁcient.
If the twopolicies are similar, we can perform the optimization using data collected only from ⇡ E + I using theapproximation proposed in [12].
To make this approximation we ﬁrst use an identity from [13] tore-write the objective ↵ J E ( ⇡ E )  J ↵ E + I ( ⇡ E + I ) as (see Appendix A.1 for detailed derivation):
[Equation]
where U ⇡ E + I min
[Equation]
s t ,
[Equation] = ↵ [Equation] +
[Equation]
⇡
[Equation]
+ I
[Equation]
Now we can use the approximation from [12] to formulate Eq.
7 as:
[Equation]
subject to E s ⇠
[Equation]
where D KL denotes the Kullback–Leibler divergence and  is a small constant threshold that limits thedivergence between two policies.
Note that the expectation is computed with respect to data collectedonly from ⇡ E
+ I .
We solve for the new ⇡ E by optimizing a clipped version of the objective in Eq.
8,similar to PPO.
More details are provided in Appendix A.2.
Mixed policy ⇡ E + I (max-stage).
This subproblem of solving ⇡ E
+ I is posed as
[Equation]
Similarly to solving ⇡ E we use the approximation used in Eq.
8 to rewrite the objective J ↵ E + I ( ⇡ E + I )
↵ J E ( ⇡ E ) as follows (see Appendix A.1 for details):
[Equation]
subject to
[Equation]
where U ⇡ E max ( s t , a t ) :
= (1 + ↵ ) r Et + r It + ↵ V ⇡ E E ( s t +1 )  ↵ V ⇡ E E ( s t ) V ⇡ E E ( s t ) :
=
E ⇡ E h
X t
[Equation]
t r Et | s [Equation] = s t i .
Lagrangian multiplier ↵ .
Let g ( ↵ ) :
=
max ⇡ E + I 2 ⇧ min ⇡ E 2 ⇧ J ↵ E + I ( ⇡ E + I )
↵ J E ( ⇡ E ) .
Wesolve min ↵ g ( ↵ ) using gradient descent with step size  as shown below (see Appendix A.1 fordetails):
[Equation]
3.3 Implementation
Min-max alternation schedule We use an iterative optimization scheme that alternates betweensolving for ⇡ E by minimizing the objective in Eq. 6, and solving for ⇡ E + I by maximizing the objectivein Eq. 9.
Typically, min-max optimization switches between minimization and maximization stagesat every iteration.
We ﬁnd that switching in each iteration hinders the performance of our method.
Wehypothesize that frequently switching leads to insufﬁcient training for each policy, and thus we usethe objective value between two subsequent iterations to determine when to switch.
At each iteration,trajectories are collected using one of the the two policies, and the other policy is trained using thesetrajectories.
Let J [ i ] be the objective value J ↵ E
+ I ( ⇡ E + I )
↵ J E ( ⇡ E ) at iteration i .
We switch thestage when the objective value does not improve anymore.
The switching rule can be summarized as:
[Equation]
False , if max_stage =
True
[Equation]
True , if max_stage =
False , where max_stage is a binary variable indicating whether the current stage is max-stage.
Whenmax-stage is done, ↵ is updated using Eq. 11.
Parameter sharing In an image state space (e.g., Atari) it is preferable to share the convolutionalneural network (CNN) backbone between ⇡ E and ⇡ E
+ I for the sake of data efﬁciency.
However,if the backbone is shared, maximizing the objective (Eq. 9) over ⇡ E + I could decrease ↵ J E ( ⇡ E )and impede the performance of ⇡ E due to gradient interference in the shared backbone.
Similarly,minimizing the objective (Eq. 6) over ⇡ E could decrease J ↵ E + I and degrade the performance of⇡ E
+ I as well.
To prevent such interference, we jointly optimize both ⇡ E + I and ⇡ E with additionalauxiliary objectives
[Equation]
( ⇡ E + I ) and max ⇡ E J E ( ⇡ E ) during the min-stage and maxstage, respectively.
Intuitively, the auxiliary objective compensates against gradient interference byintroducing a counteracting force that attempts to ﬁx the objective not being directly optimized duringthe current stage.
For example, when maximizing J ↵ E + I ( ⇡ E + I )  ↵ J E ( ⇡ E ) over ⇡ E + I , the auxiliaryobjective max ⇡ E J E ( ⇡ E ) can prevent any updates from decreasing J ( ⇡ E ) .
In summary, we outline the practical policy optimization algorithm, Extrinsic-Intrinsic Policy Optimization (EIPO) , for solving Eq.
5 in Algorithm 1.
The detailed version is presented in Algorithm 2in the Appendix.
Note that we solve the constrained problem (Eq. 10 and Eq.
8) using a PPO-basedapproach.
The full implementation details, including hyperparameters can be found in Appendix A.2.
Update ⇡ E by Eq.
8 and ⇡ E
+ I by auxiliary objective (Section 3.3) 11:
J [ i ]
J ↵ E + I ( ⇡ E + I )
↵ J E ( ⇡ E ) 12: max_stage[i]
J [ i ]  J [ i  1]  0 13: end if 14: if max_stage[i - 1] =
True and max_stage[i]
=
False then 15:
Update ↵ (Eq. 28) . when max is done 16: end if 17: end for
Experiments
We evaluate our method in 61 Atari games from the Arcade Learning Environment (ALE) [14]benchmark, as well as Pycolab environments [15] for illustrative examples.
We compare against thefollowing baselines, where the text inside the parentheses denotes the acronym.
• PPO [10] : The policy is trained using PPO with extrinsic rewards only: ⇡ ⇤ = arg max ⇡ 2 ⇧ J E ( ⇡ ) .•
RND [7] : The policy is trained using PPO with the sum of extrinsic rewards r Et and intrinsicrewards  r
It generated by RND: ⇡ ⇤ = arg max ⇡ 2 ⇧ J E + I ( ⇡ ) .
• Ext-norm-RND ( EN )
: We propose a variant of RND where the extrinsic rewards are normalizedusing running mean and standard deviation (Appendix A.2).
We include this as a baseline since ourempirical results show that extrinsic reward normalization improves over RND in games wherevanilla RND performs much worse than PPO.• Decay-RND ( DY ) :
We propose a variant of Ext-norm-RND where intrinsic rewards are progressivelyscaled down to eliminate exploration bias over time.
Intrinsic rewards r It are scaled by  ( i ) , wherei denotes the iteration number.
The objective function turns into J E
+ I = E
⇡
[Equation]
( i ) r It ) i , where  ( i ) is deﬁned as  ( i ) =
clip ( i I (  max   min ) ,  min ,  max ) , where  max and  mindenote the predeﬁned maximum and minimum  ( i ) , and I is the iteration after which decay isﬁxed.
We follow the linear decay schedule used to decay ✏ -greedy exploration in DQN [16].• Decoupled-RND ( DC ) [17] : We adapt the method proposed in [17] to Ext-norm-RND .
Two policies⇡ E
+ I and ⇡ E are trained as follows:
[Equation]
The exploration policy ⇡ E + I collects trajectories for training ⇡ E + I and ⇡ E .
⇡ E + I and ⇡ E maximize mixed and extrinsic objectives, respectively.
The D KL ( ⇡ E || ⇡ E + I ) term in the objectiveincentivizes ⇡ I to perform differently from ⇡ E .
Our method EIPO is implemented using Ext-norm-RND .
The policies ⇡ E + I and ⇡ E are parametrizedby two separate multi-layer perceptrons (MLP) with a shared CNN backbone.
The value functions
V ⇡ E + I E
+ I and V ⇡ E E are represented in the same fashion, and share the same CNN backbone with bothpolicy networks.
The hyperparameters and implementation details are described in the Appendix A.2.
Note that while we chose to use RND due to it’s state-of-the-art performance in hard exploration
Atari games, our method is applicable to any bonus-based exploration method.
4.1 Illustrative example
We exemplify intrinsic reward bias using a simple navigation environment shown in Fig.
2a whichillustrates a task with conﬂicting intrinsic and extrinsic rewards.
The white sprite denotes the agent,which observes a 5 ⇥ 5 window view of it’s surroundings.
If it reaches the (red) location at the top ofthe map, an extrinsic reward is provided.
The (blue and magenta) blocks are randomly placed in thebottom corridor at the start of each episode.
Because these blocks randomly change location, theyinduce RND intrinsic rewards throughout the course of training.
Because intrinsic rewards alongthe bottom corridor are easier to obtain than the extrinsic reward, optimizing the mixed objective
J E
+ I can yield a policy that results in the agent exploring the bottom corridor (i.e., exploiting theintrinsic reward) without ever discovering the extrinsic reward at the top.
Fig.
2b plots the evolutionof the average extrinsic returns for PPO, RND, and EIPO across 5 random seeds during training.
We ﬁnd that EIPO outperforms both RND and PPO.
RND gets distracted by colored blocks andunderperforms compared to vanilla PPO.
Notably, PPO performs slightly worse than EIPO, possiblybecause the extrinsic reward at the top cannot be trivially found without the guidance of intrinsicrewards.
We hypothesize that EIPO aggressively explores early on during training, and then focuses onexploiting extrinsic rewards after discovering the red location.
To validate this hypothesis, we plot theaverage ↵ parameter, which controls the importance of extrinsic optimality of EIPO at each trainingiteration in Fig. 2c.
It can be seen that ↵ decreases in the beginning, before increasing after theextrinsic return starts rising in Fig.
2b.
This observation corroborates our hypothesis that EIPO is ableto dynamically leverage intrinsic rewards for exploration while ultimately achieving high extrinsicreturns.
[Figure]
4.2 Does EIPO generally improve PPO performance?
We validate whether EIPO prevents the possible performance degradation introduced by intrinsicrewards, and consistently either improves or matches the performance of PPO in 61 Atari games.
Asour goal is to investigate if an algorithm generally performs better than PPO, rather than measuringraw performance we evaluate each algorithm using the “probability of improvement" (Appendix A.4)metric suggested in [18].
We ran at least 5 random seeds for each method in each environment,collecting the median extrinsic returns within the last 100 episodes and calculating the probability ofimprovement P ( A  PPO ) with 95% -conﬁdence interval against PPO for each algorithm A .
Theconﬁdence interval is estimated using the bootstrapping method.
Fig.
3a shows the probability ofimprovement for each method against PPO: P ( A > PPO )
A 6 = PPO.
It can be seen that RND is theleast likely to improve PPO, suggesting that RND does not yield consistent performance improvementsacross all 61 Atari games.
Ext-norm-RND achieves a higher probability of improvement than RND,which suggests that normalizing extrinsic rewards improves the performance degradation of RND.
Decay-RND has higher probability of improvement than Ext-norm-RND .
This indicates that decayingthe scale of intrinsic rewards over time is also a viable method to alleviate the performance drop of
RND in Atari tasks, with the notable drawback that the decay schedule has to be tuned beforehand.
Decoupled-RND attains a probability of improvement that is higher than RND yet lower than both
Ext-norm-RND and Decay-RND .
EIPO exhibited the highest probability of improvement among allthe baselines, which shows that EIPO is more likely to improve the performance of PPO across Atarigames.
In addition, Fig.
3c shows the percentage of Atari games in which one method matches or exceeds theperformance of another method with a pairwise comparison.
Note that “percentage of games A  B "
R ( A, B ) is not equivalent to probability of improvements P ( A  B ) .
Let µ A be the mean score ofalgorithm
A over 5 random seeds in an environment.
R ( A, B ) is deﬁned as: R ( A, B ) =⇥
µ A  µ B ⇤ N 225 where N denotes the number of Atari games.
The last row of Fig.
3c shows that EIPO performsbetter than all the baselines in the majority of games (i.e., R ( EIPO , B ) > 0 . 5 ).
4.3 Does EIPO perform better than the baselines?
As all our baselines improve upon vanilla RND, we compare EIPO with each baseline pairwisein terms of “strict probability of improvement".
Strict probability of improvement P ( A > B )(Appendix A.4.1) measures the likelihood that an algorithm A is strictly better than an algorithm B .
Fig.
3b shows P ( EIPO-RND > B ) 8 B .
It can be seen that EIPO outperforms all the baselines bymore than 50% .
As suggested in [18], P ( EIPO-RND > B ) > 0 .
5 with a conﬁdence interval lessthan 0 .
5 indicates that a results has statistical signiﬁcance.
We can therefore conclude that EIPO willgenerally perform better than all other baselines across all 61 Atari games.
[Figure]
4.4 Does EIPO preserve the performance gain of RND?
[Table]
We report the PPO-normalized score [18] (Appendix A.4) to validate whether EIPO preservesthe performance gain from RND when applicable.
Table 1 displays the environment-wise meanand 95% -conﬁdence interval of PPO-normalizedscores for each algorithm in the set of games where
RND achieves a higher mean extrinsic return than
PPO.
Surprisingly, EIPO-RND yields a higher performance gain than RND does, which implies that
EIPO not only mitigates the bias introduced byintrinsic rewards in Atari games, but can also improve performance.4.5 Does EIPO outperform tuned RND?
We empirically ﬁnd that the performance of RND is heavily dependent on the difference in magnitudebetween r E and r I .
Because this difference can be controlled through the intrinsic scaling coefﬁcient and the best scaling coefﬁcient is not known a priori, we tune  for RND in several games where
RND substantially under-performs compared to PPO in order to see if EIPO can match or outperformthe strongest tuned baseline.
A variety of  values are selected for each environment based on therelative difference in magnitudes between intrinsic rewards r
I and extrinsic rewards r E in eachenvironment (  values are detailed in Appendix A.4.3).
Fig.
4 shows the extrinsic return in 6 selectedenvironments given each  choice.
Overall, no choice of  consistently generates performanceimprovements over RND.
For instance, r E ⇡  r I substantially improves RND in Jamesbond , yetdeteriorates RND in YarsRevenge .
This shows that the range of acceptable constant  values is highlyenvironment-dependent, and could be narrow or potentially nonexistent.
Moreover, the effect of is unpredictable.
Intuitively, raising  increases the weight of intrinsic rewards in the optimizationobjective and potentially introduces more bias, yet r E <
r
I outperforms r E   r I in an easyexploration game Jamesbond (based on taxonomy in [2]).
In contrast, EIPO consistently performsbetter than PPO and RND using  = 1 in all environments, demonstrating that EIPO is far lesssusceptible to an ill-tuned  .
Discussion
Comparison with Decay-RND Decay-RND requires hand-tuning the decay schedule of  tocontrol the weighting between extrinsic r E and intrinsic rewards
r I (i.e., r E +  r I ).
By contrast, the Lagrangian multiplier ↵ in EIPO adjusts the relative importance between both reward functionsautomatically.
Fig.
5 shows that the EIPO-optimized trajectory of ↵ during training is quite differentacross environments, indicating that the optimal decay schedule is highly environment dependent.
Assuch, it is unlikely that there exists a single decay schedule that works well across all environments.
Additionally, Fig.
5 shows that ↵ decreases over time and yet still outperforms Decay-RND in ahandful of environments (see the learning curve on Carnival in Appendix A.6).
This implies that incertain domains it may be better to amplify the intrinsic reward rather than decay it.
vary drastically across
Comparison with Decoupled-RND Though Decoupled-RNDdoes not require tuning the decay schedule of  , the experimentalresults in Section 4 show that Decoupled-RND does not improveupon Ext-norm-RND , and can even deteriorate Ext-norm-RNDon Montezuma’s Revenge .
We hypothesize that this is because
Decoupled-RND cannot adaptively tune the importance of intrinsic and extrinsic rewards during optimization.
Decoupled-RNDenforces the constraint that the exploration policy ⇡ E + I performsimilarly to the exploitation policy ⇡ E , but if this constraint excessively penalizes ⇡ E + I in the early stages of training, ⇡ E + I willtend to degenerate to ⇡ E and lose the exploration incentive generated by r I .
EIPO can adaptively adjust the trade off between bothpolicies despite enforcing the KL-divergence constraint.
Consequently, EIPO loosens the extrinsic optimality constraint at thestart of training before tightening it later on.
The curve of ↵ w.r.t.time shown in Appendix A.6 validates our hypothesis, showingthat ↵ decreases ﬁrst before increasing again in the middle oftraining.
To conclude, we propose a theoretically motivated framework to mitigate the bias of intrinsic rewards.
Our resulting algorithm, EIPO, exhibits consistent performance improvements over PPO whilepreserving the performance gains of curiosity-driven exploration on hard exploration tasks.
Moreover,
EIPO is able to adapt the importance of intrinsic rewards during optimization, reducing the need tohand-tune hyperparameters such as exploration schedules ( ✏ -greedy), or  decay ( Decay-RND ).
Oneinteresting avenue for future work could be to incorporate intrinsic reward function optimization intoour framework, hopefully increasing the ﬂexiblity of our method on a range of tasks.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See thesupplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec.
A.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Sec.
A.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec.
A.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]