Decoupling Exploration and Exploitation in Reinforcement Learning
Abstract
Intrinsic rewards are commonly applied to improve exploration in reinforcement learning.
However, these approaches suffer from instability caused by non-stationary reward shaping and strong dependency on hyperparameters.
In this work, we propose Decoupled RL (DeRL) which trains separate policies for exploration and exploitation.
DeRL can be applied with on-policy and off-policy RL algorithms.
We evaluate DeRL algorithms in two sparse-reward environments with multiple types of intrinsic rewards.
We show that DeRL is more robust to scaling and speed of decay of intrinsic rewards and converges to the same evaluation returns than intrinsically motivated baselines in fewer interactions.
1. Introduction
Exploration is one of the essential challenges in reinforcement learning (RL).
RL algorithms often use simple randomised methods, e g. applying (cid:15) -greedy
policies (Watkins, 1989) or adding random noise to continuous actions (Lillicrap et al., 2015).
Such exploration techniques are inefﬁcient on tasks where rewards are sparse.
One category of exploration techniques which has been found to be particularly effective in sparse-reward environments are intrinsic rewards (Raileanu & Rocktäschel, 2020; Burda et al., 2018b; Pathak et al., 2017; Oudeyer et al., 2007; Chentanez et al., 2005).
These can serve as strong incentives for exploration and assist training in hard exploration problems where a steady stream of rewards is missing.
Two common approaches to deﬁne intrinsic rewards r
i are self-supervised predictions in the environment (Raileanu & Rocktäschel, 2020; Burda et al., 2018b; Pathak et al., 2017; Schmidhuber, 1991) and computing (pseudo-) counts of states (Ostrovski et al., 2017; Tang et al., 2017; Strehl Littman, 2008).
These intrinsic rewards are added to the extrinsic reward r e provided by the environment to deﬁne a combined reward signal r = r e
+
λr
i with some weighting factor λ .
However, optimising for this combined feedback introduces three challenges.
(1) Intrinsic rewards are designed to incentivise exploration and naturally decay as more exploration has been completed.
Therefore, intrinsic rewards are constantly changing.
Such non-stationary reward shaping violates the Markov assumption and can cause major instabilities.
(2) Careful selection of λ for intrinsic reward scaling is essential for their effective application.
If intrinsic rewards are too large, they might heavily distort training and effectively introduce non-stationary noise to the optimisation.
We show that small intrinsic rewards have no sufﬁcient impact and do not incentivise exploration as intended.
(3) Intrinsic rewards are designed to decay throughout training, but training is sensitive to their speed of decay .
Similar to the scale of intrinsic rewards, we show that slowly decaying intrinsic rewards disrupt training whereas quickly vanishing intrinsic rewards have no sufﬁcient impact on exploration.
These challenges lead to a signiﬁcant dependency of intrinsic rewards on hyperparameters to determine their scale and decay.
These dependencies make the application of intrinsic rewards highly task-dependent because the optimal scale and decay strongly depends on the scale of extrinsic rewards and required exploration in the respective task.
All these properties make the practical application of such methods difﬁcult (Taïga et al., 2019).
Motivated by these challenges and success in off-policy
RL (Fujimoto et al., 2018; Haarnoja et al., 2018; Silver et al., 2014; Degris et al., 2012), we propose to separate the
RL training into two separate policies.
We train a exploration policy π β for exploration with the combined signal of extrinsic and intrinsic rewards.
Alongside, we train an exploitation policy π e using only extrinsic rewards on the data collected by the exploration policy.
We refer to this approach as Decoupled RL (DeRL).
Using such decoupling addresses all the aforementioned challenges of previous application of intrinsic rewards.
The exploration policy is optimised using the combined objective of extrinsic and intrinsic rewards as in typical intrinsically motivated RL, but it is not required to be an effective policy for application in the task.
Instead, π β is only trained to generate data which can be used to train the exploitation policy.
Our experiments show that DeRL is less sensitive to scale and speed of decay of intrinsic rewards and π e is not affected by non-stationary reward shaping as it is only trained using extrinsic rewards.
We implement and evaluate on-policy and off-policy versions of DeRL with ﬁve types of intrinsic rewards in two learning environments that focus on exploration.
The exploitation policy of several DeRL algorithms is able to converge to highest evaluation returns using fewer interactions in a series of tasks compared to traditional intrinsically motivated RL baselines.
We also analyse the sensitivity of DeRL and RL baselines to the scale and the speed of decay of intrinsic rewards to verify the general dependency of these methods on the hyperparameters of intrinsic rewards and show that DeRL is more robust to varying hyperparameters.
Such improved robustness and sample efﬁciency can justify the additional cost of training a second policy.
However, we also observe that DeRL is only marginally more robust and still suffers from signiﬁcant instabilities in the off-policy optimisation of the exploitation policy π e in several tasks. We hypothesise that distribution shift caused by divergence of π e and π β leads to these instabilities, and propose future work to address these shortcomings.
2. Related Work
A variety of methods have been proposed to replicate the exploration incentive of curiosity for RL (Barto, 2013; Oudeyer & Kaplan, 2009; Oudeyer et al., 2008).
The underlying idea behind these approaches is fairly simple: agents should be incentivised to explore novel or poorly understood parts of the environment.
Therefore, an agent learns an intrinsic reward which rewards the agent for visiting novel states.
Over time, the agent should become less “curious” with completed exploration and exploitation will gradually take over.
There are two common branches of intrinsic rewards for exploration: (1) count-based and (2) predictionbased intrinsic rewards.
Below, we will provide an overview over several of these approaches.
2.1. Count-based Intrinsic Rewards
Count-based intrinsic rewards are deﬁned to be inverse proportional to the count of visitations of encountered states, as shown in Equation (1).
[Equation]
Thereby, agents are incentivised to visit states within the environment which are less frequently encountered.
Likewise, agents are discouraged from visiting frequently encountered states which are deemed less valuable for exploration.
While this approach is easily applicable in small, discrete state spaces, pseudo-counts have to be computed for large or continuous state spaces where encountering any state multiple times is rare.
These pseudo-counts can be computed using various techniques to group similar states together.
Density models: Bellemare et al.
(2016) and Ostrovski et al.
(2017) achieve such grouping with a density model predicting the visitations of states.
Such pseudo-counts improved exploration and thereby learning progress considerably in multiple Atari games.
Hash functions: Tang et al.
(2017) propose to generalise pseudo-counts across similar states with a locality-sensitive hash function (Andoni & Indyk, 2008) for states which is more efﬁcient to compute compared to deep density models.
The paper speciﬁcally proposes the SimHash (Charikar, 2002) function measuring angular distance based on the sign of a randomised mapping.
2.2. Prediction-based Intrinsic Rewards
A separate approach deﬁnes intrinsic rewards using predictions in the environment.
Schmidhuber (1991) proposed an intrinsic reward deﬁned as the prediction error of predicting the next state given the current state and action.
However, stochastic and thereby unpredictable dynamics within the environment would lead to the exploration signal constantly remaining high at the face of unpredictability.
While a concept of boredom has been proposed to decay rewards for “not learnable” dynamics, this remains a major challenge of these approaches and is commonly called the “noisy TV problem” (Burda et al., 2018a).
Intrinsic curiosity module (ICM): Pathak et al.
(2017) propose the intrinsic curiosity module to learn efﬁcient state representations φ ( s ) and assign an intrinsic reward for the prediction error of the next state
[Equation]
where φ is a learned self-supervised state-representation function trained using an inverse-dynamics objective: given a representation of current state, φ ( s t ) , and next state, φ ( s t +1 ) , predict the applied action a t .
Through this feature representation, the model only learns to encode information of states which might be affected through the agent’s actions.
Using such representation, the prediction error is based on the prediction (cid:98) φ ( s t +1 ) of the next state given current state s t and applied action a t .
Rewarding impact-driven exploration (RIDE): Raileanu & Rocktäschel (2020) propose rewarding impact-driven exploration which aims to reward the agent for applying actions which lead to signiﬁcant change in the environment.
Such impact is deﬁned as the difference between embeddings of consecutive states, where the embedding function is trained using an inverse dynamics model identical to
ICM (Pathak et al., 2017).
In order to avoid the agent going back and forth between a group of states, an episodic state-count is added to the objective
[Equation]
where φ is the trained state embedding and N ep represents the episodic (pseudo-) count of states.
Random network distillation (RND): Burda et al.
(2018b) propose a simpliﬁed prediction-based intrinsic reward for exploration based on two observations: (1) training and computation of complex models of dynamics is expensive and (2) static random feature representation is surprisingly effective for state predictions (Burda et al., 2018a).
Random network distillation computes a representation of encountered states trained to mimic a randomly initialised, ﬁxed target representation
[Equation]
where φ is a randomised state representation function and (cid:98) φ is trained to represent φ .
2.3. Off-Policy Reinforcement Learning
RL trains a policy to collect data and to maximise the extrinsic returns in the environment.
Exploration and exploitation typically apply different policies, but these are often derived from each other, e.g. a greedy policy for exploitation.
We propose to separate the training of two distinct policies for exploration and exploitation, but training the exploitation policy using experience generated by the exploration policy requires off-policy RL.
Off-policy RL is concerned with the optimisation of a policy π e using experience generated within the environment by following a separate behaviour policy π β .
Separating exploration and exploitation enables the exploration policy to be trained for exploration without modifying the training objective of the exploitation policy.
However, one major challenge of off-policy RL is distribution shift , caused by differences of both policies (Fujimoto et al., 2018).
Such differences cause the state distributions and action distributions of both policies to diverge and can cause signiﬁcant instabilities during training.
2.4. Unbiased Task Policy
Whitney et al.
(2021) recently proposed to separately train an exploration and task policy using off-policy RL.
In their work, they train a task policy separate from any intrinsic rewards and concurrently train an exploration policy using only intrinsic rewards.
Their behaviour policy is deﬁned as a factored policy of both the task and exploration policy.
In their work, they solely focused on off-policy task policies trained using soft Double-DQN (van Hasselt et al., 2016) and further optimisations focused on fast adaptation of the exploration policy.
Intrinsic rewards are deﬁned using a simple count-based deﬁnition which is also used for optimisitic initialisation (Rashid et al., 2020) but no further intrinsic reward deﬁnitions are considered.
In contrast, we consider a range of different intrinsic rewards and ﬁnd that on-policy training of the task policy leads to better results with off-policy correction.
3. Decoupled Reinforcement Learning
In this work, we propose to decouple exploration and exploitation into two separate policies.
We train an exploration policy π β with the intent to explore the environment.
Using the data collected by the exploration policy, a separate exploitation policy π e is trained, as visualised in Figure 1.
Formally, an agent trains a exploration policy π β which is trained to maximise the sum of intrinsic and extrinsic rewards to achieve exploration,
[Equation]
[Equation]
with G e + i denoting the discounted returns computed using the combination of extrinsic and intrinsic rewards with scaling factor λ and discount factor γ ∈ [0 , 1) .
During training of π β , experience samples ( s t , a t , r et , s t +1 ) with extrinsic rewards are collected in a set D .
Initialise: parameters φ , θ and π β
D ← ∅
i ←for ep = 0 , . . .
N ep do a t ∼ π β
( s t ) s t +1 , r et ← environment step with a t
Update π β
D ← D ∪ ( s t , a t , r et , s t +1 )
i ← i +if i mod T Dec = 0 then
Update φ with Equation (9) and D
Update θ with Equation (12) and D
D ← ∅ end if end for
In addition to this typical intrinsically motivated RL, we train a separate exploitation policy π e to maximise only expected cumulative extrinsic rewards using experience accu- mulated in D with G et denoting discounted extrinsic returns. [Equation] Both exploration π β and exploitation policy π e can be trained using any RL algorithm given the deﬁned objectives. Generally, the exploration policy π β is optimised as typical RL with intrinsic rewards, whereas π e is trained every T Dec timesteps on experience from D which is generated by π β ’s interaction in the environment.
Note that D only contains extrinsic rewards and is off-policy data for the optimisation of π e as it was generated by following π β .
Below, we propose two methods to apply such decoupled RL in an on-policy and off-policy fashion. 3.1. Decoupled On-Policy Reinforcement Learning In order to apply on-policy RL algorithms to train π e using D , off-policy correction can be applied to account for differences in trajectory distributions of both π e and π β .
One technique for off-policy correction is importance sam- pling (IS).
In the following, we assume π e is trained using an on-policy actor-critic RL algorithm with state value func- tion V , parameterised by θ , and policy π e , parameterised by φ .
The latter are optimised by minimising the actor loss given in Equation (9)
[Equation]
with bootstrapped return estimates (cid:98) G ( S t )
[Equation]
S t ; θ )) (10)
Algorithm 2 Decoupled
Initialise: parameters θ and π β
D ← ∅
i ←for ep = 0 , . . .
N ep do a t ∼ π [Equation]
( s t ) s t +1 , r et ← environment step with a t
Update π β
D ← D ∪ [Equation] t , a t , r [Equation]
) i ← i +if i mod T Dec = 0 then
Update θ with Equation (13) and D end if end for and IS weights ρ [Equation] [Equation]
Similarly, the value loss for π e using IS weights can be deﬁned as follows: [Equation] IS weights ρ can cause instabilities during off-policy training through exploding weights when π e ( A t | S t ; φ ) (cid:29) π β ( A t | S t ) or vanishing weights for π e ( A t | S t ; φ ) (cid:28) π β
( A t | S t ) .
In particular, such instabilities occur when one policy assigns approximately zero probability for some action.
Various techniques have been proposed to address such instabilities, including clipping of importance weights (Munos et al., 2016; Espeholt et al., 2018) to minimise vanishing or exploding gradients.
The pseudocode for the optimisation of π e using on-policy
RL can be found in Algorithm 1.
3.2. Decoupled Off-Policy Reinforcement Learning
Instead of optimising π e using on-policy RL algorithms with off-policy correction such as importance sampling, any off-policy RL algorithm can be applied without the need for any correction.
In this work, we consider optimising π e using off-policy Deep Q-Networks (DQN) (Mnih et al., 2015).
For DQN optimisation, the following Q-learning loss is minimised
[Equation]
with target Q-values
[Equation]
and ¯ θ denoting the parameters of the periodically updated target network.
Pseudocode for off-policy optimisation of π e can be found in Algorithm 2.
Note that D is only used for a single update in decoupled on-policy learning, whereas in off-policy optimisation D represents a replay buffer (Lin, 1992) which is continually ﬁlled with experience.
4. Evaluation
We evaluate on- and off-policy DeRL in two learning environments with a variety of RL algorithms and intrinsic rewards.
In particular, we investigate the following two hypotheses: (1) DeRL leads to higher returns or demonstrates improved sample-efﬁciency , converging to comparable returns than intrinsically motivated RL baselines in fewer environment interactions, and (2) DeRL is more robust than intrinsically motivated RL baselines to varying scale λ and speed of decay of intrinsic rewards.
4.1. Intrinsic Rewards
Count-based: We consider two count-based intrinsic rewards.
Dict-Count directly stores and increments state occurrences in a table.
Hash-Count additionally groups states using the SimHash function (Tang et al., 2017) before computing intrinsic rewards following Equation (1).
Prediction-based: Besides count-based intrinsic exploration deﬁnitions, we consider ICM (Pathak et al., 2017),
RND (Burda et al., 2018b), and RIDE (Raileanu & Rocktäschel, 2020) as prediction-based approaches.
ICM
and
RIDE both use the same inverse dynamics optimisation to compute state-representations and deﬁne intrinsic rewards based on next state prediction error and consecutive state differences, respectively.
In contrast, RND initialises a random state representation and trains a network to mimic such representation.
For more details, see Section 2.2.
4.2. Algorithms
Baselines: As baselines, we consider on-policy RL algorithms Advantage Actor-Critic (A2C)
(Mnih et al., 2016) and Proximal Policy Optimisation (PPO)
(Schulman et al., 2017).
Both algorithms are trained using the combined reward r t =
r
et
+ λr it with some weighting factor λ and intrinsic reward deﬁnition as given above.
DeRL:
For our decoupled RL optimisation, we consistently train π β using A2C as we found it to be more robust than PPO.
We train π β using Dict-Count and ICM as intrinsic rewards.
For the optimisation of π e , we consider A2C and PPO for on-policy DeRL and Deep Q-Networks (DQN) (Mnih et al., 2015) for off-policy DeRL.
We refer to these algorithms as DeA2C, DePPO and DeDQN.
4.3. Environments
DeepSea is an environment proposed as part of the Behaviour Suite (Bsuite) for RL (Osband et al., 2020), visualised in Figure 2.
The environment targets the challenge of exploration and represents a N × N grid where the agent starts in the top left and has to reach a goal in the bottom right location.
At each timestep, the agent moves one row down and can choose one out of two actions.
For each state, both actions are randomly assigned to left and right movement.
The agent observes the current location as a 2D one-hot encoding and receives a small negative reward of − 0 .
N for moving right and 0 reward for moving left.
Additionally, the agent receives a reward of +1 for reaching the goal and the episode ends after N timesteps.
The difﬁculty of the exploration in DeepSea can be adjusted using N : the larger N , the harder it becomes for the agent to reach the goal location for optimal returns of 0 . 99 .
We evaluate all algorithms in the DeepSea task for N ∈ { 10 , 14 , 20 , 24 , 30 } .
Hallway is a new environment proposed as part of this work to target exploration, visualised in Figure 3.
In this navigation task, similar to DeepSea, an agent is located in a hallway starting on the left.
A goal can be reached by moving N l cells to the right.
In contrast to DeepSea, the goal is not necessarily located at the right end of the hallway, but there might be further N r empty cells to the right of the goal location.
At each timestep, the agent can choose between three actions: move left, stay or move right.
The [Table] agent receives a reward of − 0 .
01 for moving right or stay and 0 reward for moving left.
Additionally, the episode ends after 2 N l steps and the agent receives a reward of +1 for reaching the goal for the ﬁrst time and every time it stays at the goal location for 10 steps.
Therefore, the optimal behaviour is to move to the goal location and stay there for the remaining timesteps of the episode to collect further reward.
We hypothesise that Hallway tasks, in particular with N r > 0 , exploration through intrinsic rewards have to be carefully balanced because exploration incentives and extrinsic rewards in Hallway are not aligned.
In contrast, receiving extrinsic reward in DeepSea requires the agent to reach states at the end of the environment, so intrinsic rewards for exploration strongly align with extrinsic rewards from the environment.
We evaluate all algorithms in the Hallway environment with N l ∈ { 10 , 20 , 30 } and N r either being 0 or equal to N l .
4.4. Implementation Details
We compute n-step returns for reduced bias of value estimates in all algorithms.
On-policy training uses four parallel, synchronous environments and an additional entropy regularisation term in the policy loss (Mnih et al., 2016).
Double-DQN (van Hasselt et al., 2016) targets are computed for DQN. For details on the conducted gridsearch for hyperparameter tuning as well as all hyperparameters used throughout experiments, see Appendix A.
We train all algorithms for 100,000 episodes and evaluate every 1,000 episodes for a total of 100 evaluations throughout training by applying the greedy (evaluation) policy in the respective task for 8 episodes.
Reported results are averaged across ﬁve random seeds with shading indicating a single standard deviation.
Optimal returns are indicated using a dashed horizontal line.
All baselines and DeRL algorithms use a weighting factor of λ = 1 for the combined reward signal unless stated otherwise.
5. Results
Evaluation results are summarised in Table 1 showing average evaluation returns of all algorithms across all DeepSea and Hallway tasks. Returns are averaged across allevaluations to indicate achieved returns as well as sample efﬁciency.
Additionally, we include plots of normalised returns, tables showing the maximum achieved evaluation returns at any evaluation and learning curves for each individual task in Appendix B.
Promising results: In all DeepSea tasks, DeDQN achieves highest average evaluation returns. In the majority of these tasks, it converges to similar returns and achieves comparable sample efﬁency to the best performing baselines. DeA2C and DePPO demonstrate similar returns and sample efﬁciency in some of these tasks (see Figures 9a and 9d).
In smaller Hallway tasks with N l = 10 , N r = 0 and
N l = N r = 10 (visualised in Figures 10a and 10d), the exploitation policies of DeA2C and DePPO converge to highest returns of 0.8 and are shown to be more sample efﬁcient reaching returns after up to 85% less episodes of training compared to the best performing baselines. Also DeA2C achieved such returns robustly throughout all training whereas other DeRL algorithms and baselines exhibit no learning or signiﬁcant instabilities.
DePPO with ICM intrinsic rewards achieves highest evaluation returns in the hardest Hallway task with N l = N r = 10 (see Figure 10f).
In most Hallway, we ﬁnd DeA2C to converge more robustly compared to DePPO which exhibits more variance but reaches higher returns quicker in several DeepSea and Hallway tasks.
We believe training the exploitation policy π e using PPO can be beneﬁcial when trained with off-policy data from π β due to the trust region optimisation applied by PPO (Schulman et al., 2015; 2017).
This conﬁnes the policy change in a single optimisation step by constraining divergence of behaviour and trained policy.
In our case,
DePPO uses a clipping approximation of the KL-divergence
[Figure]
E p
i s o d e r e t u r n I m p o r t a n c e S a m p li n g W e i g h t between π β and π e to enforce both policies to not diverge too far from each other.
Such an optimisation constraint seems to assist training of π e by stabilising the divergence of both policies.
Limitations and challenges: However, we also observe several remaining limitations and cases of failure for DeRL algorithms.
DeDQN achieves low returns in the Hallway environment compared to both on-policy DeA2C and DePPO despite its off-policy training.
Also, signiﬁcant variance can be observed for baselines and DeRL algorithms in harder
DeepSea and most Hallway tasks.
Off-policy optimisation is theoretically independent of the policy generating training samples and in DeA2C and DePPO IS weight correction is applied to correct for the off-policy training data. However, we believe distribution shift (Fujimoto et al., 2018) is causing instabilities when optimising the exploitation policy from data generated by π β .
Figure 4 visualises unstable
IS weights for DeA2C in the DeepSea task with N =averaged over three seeds.
These appear to correlate with some of the noticeable drops in returns throughout training indicating the negative impact of divergence of exploration and exploitation policies on RL training of π e .
Even when applying Retrace( λ ) (Munos et al., 2016) to clip IS weights, similar results could be observed.
6. Hyperparameter Sensitivity
In order to investigate the impact of DeRL on the robustness to varying scale and speed of decay of intrinsic rewards, we train all baselines and decoupled algorithms on the combined objective r = r e
+ λr i in DeepSea N = 10 and
Hallway N l = N r = 10 with varying λ and speeds of decay.
All results with bar plots and tables showing maximum and average returns across varying hyperparameter values can be found in Appendix C. Scale of intrinsic rewards:
We consider λ ∈ { 0 . 01 , 0 . 5 , 1 . 0 , 2 . 0 , 4 . 0 , 10 . 0 , 100 . 0 } to analyse the sensitivity to varying scale of intrinsic rewards.
Figure 5 shows average evaluation returns for all values of λ for baselines and DeRL with Dict-Count intrinsic rewards.
Returns are averaged across three seeds before the mean and standard deviation across all 100 evaluation returns is computed. In DeepSea N = 10 , similar sensitivity to varying values of λ can be observed for most algorithms. DeRL, in particular DeA2C and DeDQN, exhibit slightly improved robustness to varying values of λ compared to the baselines. In Hallway, all algorithms exhibit larger variance for varying λ with no signiﬁcant learning being observed for large or small values of λ with DeA2C and DePPO demonstrating slightly more robustness.
Overall, these results indicate the importance of careful selection of hyperparameters and dependence of all these methods on λ . Even small deviations of λ can make the difference between learning and not learning at all. Decay of intrinsic rewards: We also investigate the sensitivity of intrinsically motivated baselines and DeRL algorithms to the speed of decay of intrinsic rewards in both Hallway and DeepSea. For count-based intrinsic rewards, the speed of decay can be determined by the increment of the state count N ( s ) .
By default, the count is incremented by 1 for each state occurrence. For a sensitivity analysis, we consider increments { 0 . 01 , 0 . 1 , 0 . 2 , 1 . 0 , 5 . 0 , 10 . 0 , 100 . 0 } . For deep predictionbased intrinsic rewards the speed of decay is determined by their learning rates. We consider learning rates
[Equation] [Equation] − 7 , 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 } .
Figure 6 shows average evaluation returns for all considered count increments for baselines and DeRL with Dict-Count intrinsic rewards for both DeepSea N = 10 and Hallway N l = N r = 10 . A2C appears slightly more robust to varying speed of decay in DeepSea N = 10 compared to
PPO.
DePPO demonstrates very similar returns for all decay values, but DeA2C and DeDQN are again shown to be the most robust algorithms exhibiting high evaluation returns for all considered values. Similarly as for λ sensitivity, we observe very signiﬁcant dependency on the speed of decay in the Hallway task with DeA2C exhibiting the most robustness to varying values. However, all DeRL algorithms and baselines appear highly dependent on careful selection of these hyperparameters.
7. Future Work
This work serves as an initial investigation into the feasibility of decoupling exploration and exploitation in RL.
However, there are two signiﬁcant additions which we identiﬁed but did not address yet and require further investigation.
Minimising Distribution Shift: Training π e using onpolicy RL with IS correction still seems to suffer from signiﬁcant variability. Based on our experiments with varying
[Table]
[Figure] [Figure]
[Figure]
[Figure]
[Figure] [Table] [Figure] IS weights (see Figure 4), we believe that distribution shift of π β and π e is a cause for such variability.
Therefore, we believe that the application of divergence constraints, proposed in the literature of ofﬂine RL (Levine et al., 2020), might be a valuable technique to minimise distribution shift.
These auxiliary objectives are introduced to the optimisation and would enforce π e and π β to not diverge signiﬁcantly by introducing a term αD ( π e ( ·| S t ) , π β ( ·| S t )) to the optimisation loss.
This term is based on a distance measure D between the distribution of policies π β and π e and some weighting hyperparameter α .
Commonly-used distance measures are the maximum mean discrepancy (MMD) (Gretton et al., 2012), computed based on drawn samples from both policies, and the Kullback-Leibler (KL) divergence, which has been applied in ofﬂine RL (Jaques et al., 2019).
The resulting regulariser can be written as
[Equation]
For more distance measure candidates between two policies, see Wu et al. (2019) which also found most these metrics to perform comparably. Such divergence constraints have signiﬁcant parallels to trust region optimisation (Schulman et al., 2015; 2017) as applied in DePPO for the optimisation of π e and outlined in Section 5 which computes an approximation of the KL-divergence of π β and π e .
In DeRL, divergence constraints can be directly applied to the optimisation of either the exploration π β or exploitation policy π e .
We can choose to keep π β close to π e and likewise can enforce π e to stay close to π β .
We hypothesise, that adding such a constraining regularisation to the optimisation of π β to enforce the exploration policy to stay close to the exploitation policy might be ideal as it does not interfere with the exploitation policy itself while still leveraging improved exploration of π β .
However, both directions as well as a combination of both constraints should be considered.
Only intrinsically rewarded exploration policy: The ex- [Figure] a i n i n g r [Figure]
t
[Figure]
r n
[Figure]
exploitation policies, respectively.
ploration policy π β is solely trained to generate training data
D for the optimisation of π e .
We hypothesise that it might be beneﬁcial to optimise π β using only intrinsic rewards.
Such optimisation would likely lead to more robustness to hyperparameters of intrinsic rewards as they would not be combined with extrinsic rewards of the environment. However, optimisation of π β without extrinsic rewards might cause further divergence of π β and π e .
We conducted preliminary experiments, training DeA2C with Dict-Count intrinsic rewards in both DeepSea 10 and Hallway N l = N r =for 40,000 and 20,000 episodes, respectively, with the exploration policy being optimised using either the sum of extrinsic and intrinsic rewards or only using intrinsic rewards.
Figure 7 shows that training the exploration policy with only intrinsic rewards does lead to distribution shift with both policies diverging more signiﬁcantly as seen in IS weights and the KL divergence D KL ( π e || π β ) . Training of the exploitation policy appears to suffer from such differences in the more challenging Hallway task, but in DeepSea the exploitation policy was successfully trained to solve the task despite the exploration policy never reaching high returns. These results indicate the feasibility of training π β using only intrinsic rewards but also highlight remaining challenges due to diverging policies.
We hypothesise that the addition of divergence constraints to the optimisation of the exploitation and exploration policies might address these challenges and make optimisation of π β using only intrinsic rewards effective beyond simple tasks.
8. Conclusion
In this work, we propose Decoupled RL (DeRL) which separates exploration and exploitation into two separate policies.
The exploration policy is optimised with additional intrinsic rewards to incentivise exploration and the exploitation policy is trained using only extrinsic rewards from data generated by the exploration policy.
We believe such decoupling is primarily beneﬁcial when tuning the exploration is challenging or heavy exploration is required which would typically dominate any extrinsic rewards received by the environment.
On-policy and off-policy versions of DeRL are formulated and evaluated in two sparse-reward environments.
We observe that DeRL demonstrates improved sample efﬁciency in some tasks by reaching high returns with fewer interactions in the environment. Our results also demonstrate that intrinsically motivated RL is highly dependent on careful hyperparameter tuning of intrinsic rewards with DeRL exhibiting slightly improved robustness to varying scale and speed of decay of intrinsic rewards.
However, DeRL still seems to suffer from drops in performance and instability which correlate with divergence of the exploration and exploitation policies.
Further research is proposed to address such distribution shift of both policies using divergence constraints similar to trust region optimisation.
Lastly, the feasibility of training the exploration policy only using intrinsic rewards is demonstrated which serves as a second opportunity for future work.
A. Hyperparameter Conﬁguration
A gridsearch was conducted to identify the best hyperparameter conﬁguration for each algorithm in both environments. In order to keep the search feasible, hyperparameter search for DeepSea is conducted in the task with N = 20 and the same conﬁguration is applied in all other DeepSea tasks. Similarly, for the Hallway environment hyperparameter search has been conducted in the task with N l = N r = 10 .
Every gridsearch conﬁguration is evaluated using 3 different random seeds with results being averaged over all seeds.
First, the best hyperparameters for A2C and PPO baselines were identiﬁed by training each conﬁguration in the respective training task of both environments with Dict-Count intrinsic rewards.
We chose to conduct the hyperparameter search with intrinsic rewards as both baselines do perform poorly in most tasks without any intrinsic rewards which make different hyperparameter conﬁgurations hardly distinguishable.
The Dict-Count intrinsic reward deﬁnition was chosen as it does not require any hyperparameter tuning in contrast to other intrinsic reward deﬁnitions.
For all considered hyperparameter combinations, we conduct training in the same way as described in Section 4 with λ = 1 . The best hyperparameter conﬁguration is chosen as the one which leads to highest maximum evaluation returns. If multiple conﬁgurations reach the same maximum evaluation returns then the one with the highest average evaluation returns computed over allevaluations throughout training is considered the best.
Such latter metric considers achieved returns alongside learning speed. Identiﬁed hyperparameters for A2C and PPO in both environments can be found in Table 2 and Table 3.
All considered values of hyperparameters are listed with the best-identiﬁed combination highlighted in bold. [Table] [Table]
A.2. Intrinsic Reward Hyperparameters
Following the hyperparameter search of A2C and PPO with Dict-Count intrinsic rewards, a search over hyperparameters of all parameterised intrinsic reward deﬁnitions was conducted.
The setup of the hyperparameter search is identical to the one described above and the best identiﬁed hyperparameter conﬁguration for the baseline algorithms are used in the gridsearch for intrinsic rewards.
Best identiﬁed hyperparameters and all considered hyperparameter conﬁgurations can be found in Tables 4 to [Table] [Table]
[Table]
[Table] [Table] A.3. Decoupled Reinforcement Learning Hyperparameters Based on the aforementioned hyperparameters, a gridsearch for all DeRL algorithms, DeA2C, DePPO and DeDQN, is conducted using the best identiﬁed A2C
and Dict-Count intrinsic rewards to train the exploration policy.
Identiﬁed conﬁgurations are listed in Tables 8 to 10. Table 11 shows hyperparameters for DeRL algorithms with ICM intrinsic rewards used to train the A2C exploration policy. The same general architecture is used for ICM when applied alongside DeRL as reported in Table
[Table]
[Table] [Table]
[Table] [Table]
In this section
[Figure]
we provide tables containing maximum evaluation returns for all baselines and DeRL algorithms in every task. As described in Section 4, results are averaged across ﬁve random seeds for the best identiﬁed hyperparameter conﬁguration as reported in Appendix A. For maximum evaluation returns, we identify the single evaluation out of all 100 conducted evaluations with the maximum evaluation returns averaged across all seeds and report its value with the standard deviation across all seeds. Within tables, the highest performing algorithms for each task are highlighted in bold together with every algorithm within a single standard deviation of the highest return.
Besides these tables, we also provide learning curves for all baselines and DeRL algorithms in every evaluated task.
Normalised evaluation returns of the highest performing baselines and DeRL algorithms in both environments are shown in Figure 8. For each task of an environment, we normalise returns to be within [0, 1] with 0 and 1 corresponding to the minimum and maximum achievable returns within the task, respectively.
[Figure]
B.1. DeepSea
[Table]
PPO RIDE 0 .
74 ± 0 .0 . 00 ± 0 .0 . 15 ± 0 .0 . 00 ± 0 .0 . 00 ± 0 .
DeA2C Dict-Count 0 . 99 ± 0 .0 . 99 ± 0 .0 . 90 ± 0 .0 .
19 ± 0 .0 .
16 ± 0 . DeA2C ICM 0 . 99 ± 0 .0 . 74 ± 0 .0 .
66 ± 0 .0 .
39 ± 0 .0 . 33 ± 0 . DePPO Dict-Count 0 . 99 ± 0 .0 .
99 ± 0 .0 .
08 ± 0 .0 .
82 ± 0 .0 . 00 ± 0 .
DePPO ICM 0 .
99 ± 0 .0 .
99 ± 0 .0 .
24 ± 0 .0 . 00 ± 0 .0 . 00 ± 0 . DeDQN Dict-Count 0 . 99 ± 0 .0 .
99 ± 0 .0 .
99 ± 0 .0 . 79 ± 0 .0 . 19 ± 0 . DeDQN ICM 0 .
99 ± 0 .0 . 99 ± 0 .0 . 99 ± 0 .0 . 59 ± 0 .0 .
39 ± 0 .
[Figure]
E p i s o d e r e t r n
[Figure] [Figure] [Figure]
p i s o d e r e t u r n
[Figure] [Figure] [Figure]
[Table] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Table]
[Table] [Table] [Figure]
[Table] [Figure] [Table]
[Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Table]
[Figure]
[Table] [Table] [Table]
[Figure]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Figure]
C. Hyperparameter Sensitivity
In this section
[Table]
we provide tables containing maximum and average achieved evaluation returns for both sets of hyerparameter sensitivity experiments.
As described in Section 6, we evaluate both baselines and DeRL algorithms with varying intrinsic reward coefﬁcients, λ , and speed of decay of intrinsic rewards.
Both sets of experiments are done in the DeepSea 10 and the Hallway task with N l = N r = 10 . We provide tables showing the maximum and average evaluation returns as in Appendix B and bar plots indicating the varying average evaluation returns for various parameterisation of intrinsic rewards. Within tables, the highest performing conﬁguration for a single algorithm is highlighted in bold within each row with all conﬁgurations within a single standard deviation of the highest return.
C.1. Intrinsic Reward Scale
First [Table] we evaluate all baselines and DeRL algorithms for various intrinsic reward coefﬁcients λ [Table]
5 , 1 . 0 , 2 . 0 , 4 . 0 , 10 . 0 , 100 . 0 } in DeepSea and Hallway.
C.1.1. D EEP S EA
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
[Figure]
[Figure]
[Figure]
E p i s o d e r e t u r n
[Figure]
E p
i s o d e r e t u r n
[Figure]
C.1.2. H ALLWAY
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Figure] [Figure] C.2. Intrinsic Reward Decay
Second [Table]
we evaluate all baselines and DeRL algorithms with varying speed of decay of intrinsic rewards in DeepSea and
Hallway.
For count-based intrinsic rewards, the speed of decay can be determined by the increment of the state count N ( s ) . By default, the count is incremented by 1 for each state occurrence. For this analysis
[Table]
we consider increments { 0 . 01 , 0 . 1 , 0 . 2 , 1 . 0 , 5 . 0 , 10 . 0 , 100 . 0 } . For deep prediction-based intrinsic rewards, ICM, RND and RIDE, the speed of decay is determined by their learning rates. We consider learning rates { 1 e − 9 , 1 e − 8 , 2 e − 8 , 1 e − 7 , 5 e − 7 , 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 } .
C.2.1. D EEP S EA [Table] [Table]
[Table] [Table] [Table] [Table]
[Figure] [Figure]
p i s o
[Figure]
e r
[Figure]
t u r n
[Figure]
i s
[Figure]
d
[Figure]
r
[Figure]
t u r n
[Figure]
C.2.2. H ALLWAY
[Table]
[Table]
[Table] [Table]
[Table]
[Figure]
[Figure]