A
A.1 Full derivation
We present the complete derivation of the objective function in each subproblem deﬁned in Section3.2.
We start by clarifying notation:
Notations
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
Max-stage objective U ⇡ E max .
We show that the objective (LHS) can be approximated by the RHSshown below
[Equation]
subject to
[Equation]
where  denotes a constant KL-divergence threshold.
We can then expand the LHS as the follows:
[Equation]
E s 0 ⇠ ⇢h
[Equation]
[Equation]
⇡
[Equation]
s
[Equation]
h
[Equation]
=
[Equation]
⇡
[Equation]
↵ V ⇡ E E
[Equation]
s
[Equation]
X t =
[Equation]
t ((1 + ↵
[Equation]
r Et + r It ) i
For brevity, let r t
=
(1 + ↵ ) r Et + r It and ↵ V ⇡ E E ( s t ) =
V t .
Expanding the left-hand of Eq. [Equation] To facilitate the following derivation, we rewrite the objective J ↵ E + I ( ⇡ E + I )  ↵ J E ( ⇡ E ) : [Equation]
where d ⇡ E + I ,  ⇢is the discounted state visitation frequency of policy ⇡ E + I with the initial statedistribution ⇢ 0 and discount factor  , deﬁned as: [Equation] Note that for brevity, we write E s ⇠ d ⇡ E + I,  ⇢,a ⇠ ⇡ h .
i as E ⇡ E
+ I h . i instead. To get rid of the dependencyon samples from ⇡ E
+ I , we use the local approximation [12, 13] shown below:
[Equation]
[Equation]
The discounted state visitation frequency of ⇡ E + I is replaced with that of ⇡ E .
This local approximation is useful because if we can ﬁnd a ⇡ 0 such that L ↵ E
+ I ( ⇡ 0 )
=
J ↵ E + I ( ⇡ 0 ) , the local approximationmatches the target in the ﬁrst order: r ⇡ E
[Equation]
|
[Equation]
This implies that if L ↵ E + I ( ⇡ E + I ) is improved, J ↵ E + I ( ⇡ E + I ) | ⇡ E + I = ⇡ 0 will be improved as well.
Schulman et al.
[12] suggested that this local approximation is valid when E ⇡ E
[Equation]
,where ✏ is a predeﬁned threshold.
Rewriting the objective in Equation 13 using local approximation(Equation 17) leads to the desired objective:
[Equation]
[Equation]
Importance
[Equation] [Equation] 21) subject to E [Equation]
⇠ ⇡
[Equation]
Note that to make use of the approximation proposed in [12, 13], we make the assumption that in thebeginning of the max-stage, ⇡ E = ⇡ E + I . Under this assumption, ⇡ E serves as ⇡ 0 (see above). Thisenables updating ⇡ E + I using the local approximation. We leave relaxing this assumption as futurework.
Min-stage objective U ⇡ E + I min . We show that the objective (LHS) can be approximated by the RHSshown below [Equation] The derivation for the min-stage is quite similar to that of the max-stage.
Thus we only outline thekey elements: [Equation] Since we empirically ﬁnd that V ⇡ E + I , ↵ E + I is hard to ﬁt under a continually changing ↵ , we replace V ⇡ E + I , ↵ E + I with V ⇡ E + I E
+ I in Equation 27, and rewrite the objective as:
[Equation]
i
[Equation]
Rewriting
[Equation] [Equation] i (See Equations 16 to 21) [Equation]
31) subject to E [Equation] ↵ optimization Let g ( ↵ ) := max ⇡ E + I 2 ⇧ min ⇡ E 2 ⇧ J ↵ E + I ( ⇡ E + I )
↵ J E ( ⇡ E ) .
As ⇡ E and ⇡ E + Iare not yet optimal during the training process, we solve min ↵ g ( ↵ ) using stochastic gradient descentas shown below: [Equation] [Equation] subject to E s ⇠ ⇡ E
[Equation]
where  is the learning rate of ↵ and A ⇡ E E ( s t , a t ) :
= r Et +  V ⇡ E E ( s t +1 )
V ⇡ E E ( s t ) denotes theextrinsic advantage of ⇡ E . A.2 Implementation details A.2.1 Algorithm Clipped objective We use proximal policy optimization (PPO)
[10] to optimize the constrainedobjectives in Equation 6 and Equation 9.
The policies ⇡ E and ⇡ E
+ I are obtained by solving thefollowing optimization problems with clipped objectives: • Max-stage: [Equation] E
[Equation]
⇡ E
[Equation]
⇡ E max
[Equation]
oi
[Equation] [Equation] • Min-stage: [Equation]
oi
[Equation] where ✏ denotes the clipping threshold for PPO. We will detail the choices of ✏ in the followingparagraphs.
Rearranging the expression for GAE To leverage generalized advantage estimation (GAE) [19],we rearrange U ⇡ E max and U ⇡ E + I min to relate them to the advantage functions. The advantage function
A ⇡ E E and A ⇡ E + I E
+ I are deﬁned as: [Equation] As such, we can rewrite U ⇡ E max and U ⇡ E + I min as: [Equation]
Extrinsic reward normalization For each parallel worker, we maintain the running average of theextrinsic rewards ¯ r E . This value is updated in the following manner at each timestep t :
[Equation]
The extrinsic rewards are then rescaled by the standard deviation of ¯ r E across workers as shownbelow: [Equation] Et [Equation]
h [Equation] i . Auxiliary objectives The auxiliary objectives for each stage are listed below:
• Max-stage: We train the extrinsic policy ⇡ E to maximize J E ( ⇡ E ) using PPO as shown below [Equation] where
⇡ old E denotes the extrinsic policy that collects trajectories at the current
iteration • Min-stage: We train the mixed policy ⇡ E + I to maximize J E + I ( ⇡ E + I )
using PPO as shown below:
[Equation]
where ⇡ old E + I denotes the mixed policy that collects trajectories at the current iteration.
The derivative of ↵ , ↵ (see Section 3.3), is clipped to be within(  ✏ ↵ , ✏ ↵ ) , where ✏ ↵ is a non-negative constant. Codebase We implemented our method and each baseline on top of the rlpyt 1 codebase. Wethank Adam Stooke and the rlpyt team for their excellent work producing this codebase.
Summary We outline the steps of our method in Algorithm
1: Initialize policies ⇡ E + I and ⇡ E and value functions V ⇡ E + I E
+ I and V ⇡ E E 2: Set max_stage[0]
False , and J [0] 3: for i = 1 · · · do . i denotes iteration index 4: if max_stage[i - 1] then . Max-stage: rollout by ⇡ E and update ⇡ E
+ I 5:
Collect trajectories ⌧ E using ⇡ E 6: Compute U ⇡ E max ( s t , a t ) 8 ( s t , a t ) 2 ⌧ E using Eq.7:
Update ⇡ E + I by Eq.
36 and ⇡ E by Eq.8:
Update V ⇡ E E (see [19]) 9:
J [ i ]
J ↵ E + I ( ⇡ E + I )  ↵ J E ( ⇡ E ) 10: max_stage[i] J [ i ]  J [ i  1]  0 11: else .
Min-stage: rollout by ⇡ E + I and update ⇡ E 12: Collect trajectories ⌧ E + I using ⇡ E + I 13: Compute U ⇡ E + I min ( s t , a t ) 8 ( s t , a t ) 2 ⌧ E
+ I using Eq.14: Update ⇡ E by Eq. 36 and ⇡ E + I by Eq.15:
Update V ⇡ E + I E
+ I
(see [19]) 16:
J [ i ]
J ↵ E + I ( ⇡ E + I )
↵ J E ( ⇡ E ) 17: max_stage[i]
J [ i ]  J [ i  1]  0 18: end if 19: if max_stage[i - 1] =
True and max_stage[i]
=
False then 20:
Update ↵ (Eq. 32) .
Update when the max-stage is done 21: end if 22: end for
Network architecture Let Conv2D(ic, oc, k, s, p) be a 2D convolutional neural networklayer with ic input channels, oc output channels, kernel size k , stride size s , and padding p . Let LSTM(n, m) and MLP(n, m) be a long-short term memory layer and a multi-layer perceptron (MLP)with n -dimensional inputs and m -dimensional outputs, respectively. For policies and value functions, the CNN backbone is implemented as two CNN layers,
Conv2D(1, 16, 8, 4, 0) and Conv2D(16, 32, 4, 2, 1) , followed by an LSTM layer,
LSTM($CNN_OUTPUT_SIZE, 512) . The policies ⇡ E + I and ⇡ E + I , and the value functions V ⇡ E + I E + Iand V ⇡ E
+ I E + I have separate MLPs that take the LSTM outputs as inputs. Each policy MLP is MLP(512,|A| ) , and each value function MLP is MLP(512, 1) .
{+For the prediction networks and target networks in RND , we use a model architecture with three CNNlayers followed by three MLP layers. The CNN layers are deﬁned as follows: Conv2D(1, 32, 8, 4,0) , Conv2D(32, 64, 4, 2, 0) , and Conv2D(64, 64, 3, 1, 0) , with LeakyReLU activationsin between each layer. The MLP layers are deﬁned as follows: MLP(
+}
*7*64, 512) , MLP(512,512) , and MLP(512, 512) , with ReLU activations in between each layer. A.2.3 Baselines
• Decay-RND ( DY ) : We propose a variant of Ext-norm-RND where intrinsic rewards are pro- gressively scaled down to eliminate exploration bias over time.
Intrinsic rewards r It are scaled by  ( i ) , where i denotes the iteration number.
The objective function turns into J E + I = E
⇡
[Equation]
i ) [Equation] i , where  ( i ) is deﬁned as  ( i ) =
clip ( i I (  max   min ) ,  min ,  max ) , where  max and  min denote the predeﬁned maximum and minimum  ( i ) , and I is the iterationafter which decay is ﬁxed. In all of our experiments, we split the entire training process into 3000iterations with equal number of frames and set  max = 1 and  min = 0 . 00001 and I = 3000 .
• Decoupled-RND ( DC ) [17] : We adapt the method proposed in [17] to Ext-norm-RND.
Two policies
⇡ E + I and ⇡ E are trained as follows:
[Equation]
i . [Table] [Table] The exploration policy ⇡ E + I collects trajectories for training ⇡ E + I and ⇡ E .
⇡ E + I and ⇡ E maximize mixed and extrinsic objectives, respectively.
The D KL ( ⇡ E || ⇡ E + I ) term in the objectiveincentivizes ⇡ I to perform differently from ⇡ E .
We train both ⇡ E + I and ⇡ E using PPO.
In additionto policies, we train value functions V ⇡ E + I E
+ I and V ⇡ E E .
Both policies and value functions share thesame CNN backbone.
Hyperparameters The hyperparameters for PPO, RND, and EIPO are listed in Table 2, Table 3,and Table 4, respectively. A.3 Environment details
Pycolab
• State space S : R 3 ⇥ 84 ⇥ 84 , 5 ⇥ 5 cropped top-down view of the agent’s surroundings, scaledto an 84 ⇥ 84 RGB image (see the code in the supplementary materials for details).
• Action space A : { UP , DOWN , LEFT , RIGHT , NO ACTION } .
• Extrinsic reward function R E : See section 4.1.
Atari
• State space S : R 1 ⇥ 84 ⇥ 84 , 84 ⇥ 84 gray images.
• Action space A : Depends on the environment. • Extrinsic reward function R E : Depends on the environment.
[Table]
[Table]
A.4 Evaluation details
A.4.1 Probability of improvement
We validate whether EIPO prevents the possible performance degradation introduced by intrinsicrewards, and consistently either improves or matches the performance of PPO in 61 Atari games.
Asour goal is to investigate if an algorithm generally performs better than PPO instead of the performancegain, we evaluate each algorithm using the “probability of improvement" metric suggested in [18].
We ran at least 5 random seeds for each method in each environment, collecting the median extrinsicreturns within the last 100 episodes and calculating the probability of improvements P ( X  PPO ) 2with 95% -conﬁdence interval against PPO for each algorithm X .
The conﬁdence interval is estimatedusing the bootstrapping method.
The probability [-ofimprovement for each method against PPO: P ( A > PPO ) 
-]of improvement is deﬁned as:
[Equation]
where x i and y j denote the samples of median of extrinsic return trials of algorithms X and Y ,respectively.
We also deﬁne strict probability of improvement to measure how an algorithm dominate others:
[Equation]
< : 1 , x i > y j 12 , x i =
y j 0 , x i < y j ,
A.4.2 Normalized score
In addition, we report the PPO-normalized score [18] to validate whether EIPO preserves theperformance gain granted by RND when applicable.
Let p X
be the distribution of median extrinsicreturns over the last 100 episodes of training for an algorithm X .
Deﬁning p
PPO as the distribution ofmean extrinsic returns in the last 100 episodes of training for PPO, and p rand as the average extrinsicreturn of a random policy, then the PPO-normalized score of algorithm X is deﬁned as: p X  p rand p PPO  p rand . A.4.3  tuning
Table 5 lists the  values used in Section 4.5. A.5 RND-dominating games Table A.5 shows that the mean and median PPO-normalized score of each method with 95%-conﬁdence interval in the set of games where RND performs better than PPO.
The set of games where RND performs better than PPO are listed below: • AirRaid Alien
[Table]
• Assault
[Table]
• Asteroids
[Table] • BankHeist [Table]
• Berzerk
[Table] • Bowling
[Table] • Boxing [Table] • Breakout
[Table]
• Carnival [Table] • Centipede
• ChopperCommand • DemonAttack • DoubleDunk
• FishingDerby
• Frostbite • Gopher • Hero • Kangaroo
• KungFuMaster
• MontezumaRevenge
• MsPacman
• Phoenix
• Pooyan
Riverraid
• RoadRunner
• SpaceInvaders
• Tutankham
• UpNDown • Venture A.6 Scores for each Atari game The mean scores for each method on all Atari games are presented in Table 7.
A.7 Complete learning curves
We present the learning curves of each method in Figure 6, and the evolution of ↵ in EIPO in Figure 7on all Atari games.
Adventure 0.
AirRaid 34693.
Alien 1891.
Amidar 1053.
Assault 8131.
Asterix 14313.
Asteroids 1360.
BankHeist 1336.
BattleZone 83826.
BeamRider 7278. Berzerk 1113. Bowling 17. Boxing 79.
Breakout 565.
Carnival 5019.
Centipede 5938.
ChopperCommand 8225.
CrazyClimber 151202.
DemonAttack 5678.
DoubleDunk -1.
ElevatorAction 45703.
Enduro 1024. FishingDerby 35. Freeway 31. Frostbite 1011.
Gopher 5544. Gravitar 1682. Hero 29883. IceHockey 6.
Jamesbond 13415.
JourneyEscape -429.
Kaboom 1883.
Kangaroo 6092.
Krull 9874.
KungFuMaster 47266. MontezumaRevenge 0. MsPacman 4996. NameThisGame 11127.
Phoenix 8265.
Pitfall 0.
Pong 20.
Pooyan 5773.
PrivateEye 97.
Qbert 23863. Riverraid 10231. RoadRunner 45922. Robotank 37.
Seaquest 1453.
Skiing -12243. Solaris 2357. SpaceInvaders 1621. StarGunner 21036.
Tennis -0.
TimePilot 19544.
Tutankham 199. UpNDown 276884. Venture 102. VideoPinball 360562.
WizardOfWor 11912. YarsRevenge 92555.
[Table]
A.8 ICM
In addition to RND, we test our method on ICM [6] - another popular bonus-based explorationmethod. The learning curves on 6 Atari environments can be seen in Fig. 8.
Our work is related to the paradigm of reward design.
Meriçli et al.
[20] uses genetic programming tooptimize the reward function for robot soccer.
Sorg et al.
[21] learns a reward function for planningvia gradient ascent on the expected return of a tree search planning algorithm
(e.g., Monte Carlo
Tree Search).
Guo et al.
[22] extends [21] using deep learning, improving tree search performance in
Atari games.
The work [23] learns a reward function to improve the performance of model-free RLalgorithms by performing policy gradient updates on the reward function.
Zheng et al.
[24] takes ameta-learning approach to learn a reward function that improves an RL algorithm’s sample efﬁciencyin unseen environments.
Hu et al.
[25] learns a weighting function that scales the given shapingrewards [26] at each state and action.
These lines of work are complimentary to EIPO, which isagnostic to the choice of intrinsic reward and could be used in tandem with a learned reward function.