Learning Intrinsically Motivated Options to Stimulate Policy Exploration
Abstract
A Reinforcement Learning (RL) agent needs to ﬁnd an optimal sequence of actions in order to maximize rewards.
This requires consistent exploration of states and action sequences to ensure the policy found is optimal.
One way to motivate exploration is through intrinsic rewards, i.e. agentinduced rewards to guide itself towards interesting behaviors.
We propose to learn from such intrinsic rewards through exploration options , i.e. additional temporally-extended actions to call separate policies (or ”Explorer” agents) associated to an intrinsic reward.
We show that this method has several key advantages over the usual method of weighted sum of rewards, mainly task-transfer abilities and scalability to multiple reward functions.
1. Introduction
At the core of the Reinforcement Learning paradigm lies the exploration-exploitation dilemma – the agent wants to maximize its reward, using the knowledge at hand, but also needs to look for better behaviors.
Despite impressive recent breakthroughs, state-of-the-art RL methods in highdimensional environments still rely on randomness in actionor parameter-space to explore; with for example Noisy Nets (Fortunato et al., 2018) in Rainbow (Hessel et al., 2018), entropy maximization in Soft Actor-Critic (Haarnoja et al., 2018), or just (cid:15) -greedy in Deep Q-Network (Mnih et al., 2015).
The latter, in particular, has the most naive exploration procedure possible.
Indeed, it is state-independent , without memory , and the exploration behavior emerging from it is, by construction, single-step and purely random .
We would like ways to consistently explore the environment or in other words, sample-efﬁciently span the state-action space.
In environments where the MDP can be considered from a tabular point-of-view, it is possible to extract powerful
Exploration-Exploitation balances.
Methods that achieve this, e.g. MBIE-EB (Strehl & Littman, 2008), often use an intrinsic reward (Oudeyer & Kaplan, 2009), a reward generated by the agent (as opposed to extrinsic environment rewards) to motivate alternative behaviors that might be useful for the true task learning.
For more complex and higher-dimensional environments, a very wide range of methods have been proposed, such as pseudo-count -based methods (Bellemare et al., 2016; Ostrovski et al., 2017; Fox et al., 2018) or curiosity -based methods (Schmidhuber, 2010; Houthooft et al., 2016; Pathak et al., 2017).
The resulting behavior, from intrinsic reward alone , is a consistent policy that actively tries to ﬁnd surprising elements in the environment (Burda et al., 2019a).
The gap in efﬁciency with a random exploration policy is striking, and motivated this work.
To integrate such intrinsic rewards R i into an RL framework with extrinsic reward R e , the traditional approach is to carefully tune a reward-shaping hyperparameter β in the ﬁnal reward expression R =
R e
+ βR i .
Several drawbacks come with this, such as hard ﬁne-tuning of β , loss of task-transfer potential, constraints on the reward function, and restriction to a single intrinsic reward function.
Our proposed alternative is to decouple the policy into an
Explorer and an Exploiter , and to add an exploration option to the set of actions of the Exploiter (see Fig. 1).
It is an additional action that the Exploiter can use to let the
Explorer act for some time, before taking back control.
The objective and contribution of this work is in:
ﬁrst, shifting the attention towards decoupled agents in intrinsic reward settings; second, opening up the path for a new family of intrinsic reward functions; third, allowing the learning from multiple intrinsic reward functions; and fourth, taking a step towards an agent with several task-independent and learnt policies built-in.
In Section 2, we present related work, mainly intrinsic mo- tivation techniques and existing approaches to learn from such signals.
In Section 3, we introduce in more detail the method, its theoretical advantages, and implementation.
In
Section 4, we provide experiments in a tabular setting to support our claims.
In Section 5, we discuss ways to export the method in function approximation, as well as limitations.
Section 6 concludes the paper.
2. Related Work 2.1. Options and Hierarchical RL
An option (Sutton et al., 1999) is a generalization of actions to cover a sequence of primitive actions or other options.
Options are often linked with Hierarchical RL (Barto &
Mahadevan, 2003), allowing a meta-agent to use multiple controller agents or other meta-agents, which enables reasoning over temporally extended states and actions.
2.2. Exploration without intrinsic motivation
Most recent state-of-the-art works in Deep RL explore without intrinsic motivation and focus on much simpler exploration methods, generally relying on randomness in actionor parameter-space.
Deep Q-Network (Mnih et al., 2015) simply uses (cid:15) -greedy.
An improvement was found in Noisy Networks (Fortunato et al., 2018), which learn perturbations of the network weights to drive exploration, relying on the idea that a single change in weight can yield signiﬁcant changes in policy.
Along with (cid:15) -greedy, they have been found to perform relatively well in comparison to intrinsically motivated agents (Taiga et al., 2020).
However, is it impossible to extract just the exploration policy from the network, diminishing its interest from a task transfer and generalization point-of-view.
Entropy-regularization methods (Williams, 1992) try to prevent the agent from overﬁtting to a task or falling into local minima by solving it while acting as randomly as possible.
Most notably, Soft Actor-Critic (Haarnoja et al., 2018) is an off-policy actor-critic algorithm that achieved excellent and very stable results on hard control tasks.
2.3. Intrinsic Motivation
Intrinsic motivation is of interest because it ”leads organisms to engage in exploration, play, and other behavior driven by curiosity in the absence of externally-supplied rewards” (Barto, 2013).
We describe a number of methods that could be used to generate intrinsic reward to train our Explorer.
Curiosity-based methods use prediction error from a learnt forward model F to generate intrinsic reward, fundamentally R i
= ||
s t +1 − F ( s t , a t )
|| , motivating the search for surprising elements and transitions (Pathak et al., 2017;
Burda et al., 2019a; Kim et al., 2019).
Count-based methods count state-action visits N ( s, a ) and generate reward for states that are not often visited, e.g.
R i =
1 / (cid:112) N
( s, a ) .
This motivates the search for novel states.
To extend to complex state-spaces, recent work uses density models to generate a pseudo -count (Bellemare et al., 2016; Ostrovski et al., 2017; Martin et al., 2017). E -values
(Fox et al., 2018) have been shown to generalize countbased methods by drawing inspiration from optimistic start methods.
A count can be derived from the E -value, naturally generating an intrinsic reward for exploration; but an agent can also be derived directly from the learnt value function.
Intrinsic motivation is not limited to exploration – indeed, a line of work focuses rather on skill learning .
Methods within this branch (Achiam et al., 2018; Eysenbach et al., 2019) often use an intrinsic reward to motivate diversity , i.e. ease of distinction of the skills.
Such methods motivate designing agents equipped with multiple intrinsic reward functions.
A thorough survey and classiﬁcation of intrinsic motivation can be found in (Aubret et al., 2019).
2.4. Options for exploration
Option learning is an active ﬁeld in Reinforcement Learning; in particular, options aimed at exploration and coverage (Machado et al., 2017; Jinnai et al., 2019) can be found using information from the graph Laplacian.
Such options are often learnt using intrinsic rewards to lead from one key state to another.
A recent paper (Dabney et al., 2020) showed that temporally extending (cid:15) -greedy by repeating the sampled action for a random duration can lead to vastly improved exploration.
Both ideas can conveniently directly be used as Explorer modules within our framework.
2.5. Combining Intrinsic and Extrinsic rewards
Although extensive work has been done on the different ways to generate intrinsic rewards, surprisingly little work exists on incorporating such rewards with the environment’s; the traditional approach uses the weighted sum R
= R e + βR
i and ﬁne-tunes β .
In the paper introducing Random Network Distillation (Burda et al., 2019b), a very efﬁcient method for intrinsic motivation based on state reconstruction error, the authors shortly propose to decouple the network’s heads into an exploration and exploitation head.
However, ﬁrst, they limit this idea to using different discount rates; and second, they only apply it to the value function estimator, not to the policy network, effectively merging both policies into one in the process.
In contrast, we decouple the agents all the way to the policies, conserving knowledge and distinct behaviors of both.
The engaged climber method (Perotto, 2015) also proposed to decouple an exploration and exploitation strategy, where the Explorer is trained with a version of a curiosity-based reward.
The focus of the author is on the timing to switch between strategies, presenting a set of conditions to decide whether to switch.
However, our work differs in two main ways: First, their algorithm and hyperparameter choices assume knowledge of the MDP; and second, their switching method relies heavily on the reward sign, and is not learnt; while we use options.
In a preliminary paper empirically studying the success of
Hierarchical RL (Nachum et al., 2019), the authors introduce Explore & Exploit , the closest approach we ﬁnd to ours.
They train an Explorer and Exploiter with respectively intrinsic (from goal speciﬁcation) and extrinsic rewards.
The core differences with our method is that a temporary policy is chosen from a ﬁxed { Explorer : 0 . 2 , Exploiter : 0 . 8 } distribution.
The Exploiter can therefore not deliberately choose to explore, nor can-he choose not to, as the exploration is deliberately not formulated as an option.
The authors only use their algorithm for comparison purposes in a HRL setting, while our focus is in learning from intrinsic motivation.
3. Method
We assume that the agent is equipped with an intrinsic reward function f ir ( s, a, s (cid:48) )
=
R i to generate a pseudoreward R i from the ( s, a, s (cid:48) ) transition.
This function’s purpose is to help the agent consistently explore diverse behaviors to ﬁnd the optimal policy, with examples provided in Section 2.3, and is expected to generate an interesting behavior without extrinsic reward at all.
We focus on how to efﬁciently use this reward as a learning signal to improve sample-efﬁciency and performance.
3.1. Traditional approach drawbacks
The traditional approach, which can be called weighted sum reward , is to ﬁne-tune a β parameter in the total reward expression R =
R e
+ βR i , where R e is the task’s reward, called extrinsic .
It basically comes down to tuning a single hyperparameter to balance out exploration and exploitation.
However, multiple problems arise from this approach.
Constraints on the reward function The weighted sum assumes that R i → 0 , so that the agent can converge towards optimal behavior.
First, this means that it constraints the exploration to the beginning of the learning, while as pointed out in (Sutton & Barto, 2018), “ any method that focuses on the initial conditions in any special way is unlikely to help with the general non-stationary case ”.
Indeed, we constrain our f ir function to disappear asymptotically; however, there is always ground for exploration.
Second, if the reward function is non-stationary or too expressive, it can cause convergence issues: ” when intrinsic rewards evolve over time, the agent generally cannot ﬁnd an optimal stationary policy ” (Aubret et al., 2019).
Loss of knowledge and generalization First, if the reward function disappears asymptotically, the learnt exploratory behavior will be lost – the agent will forget how to explore.
Second, even if it did not, in combining both policies into one, it is impossible to extract either the greedy or exploratory behavior at any time.
If no greedy behavior can be extracted, the performance will be hindered as long as the intrinsic reward does not disappear.
If no exploratory behavior can be extracted, we lose valuable task-independent knowledge about the environment.
Third, in combining the policies, it is unclear when the greedy behavior will win over the exploratory one, which might impede multi-step exploration.
Hyperparameter ﬁne-tuning In the weighted sum expression, ﬁne-tuning β is reward-dependent, non-intuitive, and can therefore be tedious: ” without due care, the optimal policy can be altered or even completely obscured by the intrinsic rewards ” (Fortunato et al., 2018).
We will see that there is not always a good balance.
Extensions to multiple reward functions If multiple intrinsic reward functions f ( j ) ir =
R ( j ) i are available, the grid-search ﬁne-tuning of β ( j ) parameters in the total R =
R e + (cid:80) j β ( j ) R ( j ) i reward expression quickly becomes impractical.
Even with few reward functions, the rewards R ( j ) i are hard to compare intuitively, both in amplitude and decaying rates, so we are likely to lose most of the behaviors in the ﬁne-tuning process.
3.2. Proposed alternative
Our focus is therefore on ﬁnding a way to beneﬁt from an intrinsic motivation function f
ir without such downsides.
The key insight to our approach lies in the fact that a purely intrinsically motivated agent can still learn powerful exploration policies (Burda et al., 2019a).
3.2.1. L EARNING FROM THE INTRINSIC REWARD
Our proposition can be summarized in the term decoupling : training a different agent per reward function available.
We call the agent trained with extrinsic reward r ( s, a, s (cid:48) )
=
R e the Exploiter ; all other agents, trained with intrinsic rewards f ( j ) ir ( s, a, s (cid:48) )
=
R ( j ) i , are called Explorers .
3.2.2. C ALLING THE EXPLORATION POLICY
We would like the agent to learn when to explore, and when to exploit.
For this, we equip the Exploiter with an exploration option , i.e. an additional action to call the Explorer.
For n Explorers, if k = |A| with A the set of possible actions, then the options are a explorer j = a k + j , ∀ j ≤ n .
The overall agent architecture with n = 1 Explorer is shown in
Fig.
1, with additions in green.
When called, an Explorer acts for the next c switch timesteps 1 , before ”switching” back to the Exploiter.
Learning an option termination function is beyond the scope of this paper.
Usage of the option leads to multi-step exploration, which corresponds to an exploratory temporal abstraction and has empirically been shown, in some settings, to be important for Hierarchical RL (Nachum et al., 2019).
3.2.3. T RAINING
The reward ˆ R t +1 = (cid:80) c switch − 1 k = γ k R t + k +1 and bootstrapping discount γ c switch are used for option learning.
All agents –Exploiter, or Explorers– have access to the primitive actions for natural control; only the Exploiter has access to more, with as many options as Explorers available.
The agents are all trained off-policy from the ( S 0 , A 0 , ..., S t , A t , S t +1 ) stream of data; using extrinsic reward r ( S t , A t , S t +1 )
=
R e for the Exploiter, and intrinsic rewards f ( j ) ir ( S t , A t , S t +1 )
=
R ( j ) i for Explorers.
3.2.4. U NDER - SAMPLING PROBLEM
When the action space is large, the options might get drowned in the possibilities, and get sampled less.
To prevent this, under an (cid:15) -greedy Exploiter policy, we can choose a probability p ex to select an exploration option when acting non-greedily (under probability (cid:15) ).
At one extreme, p ex = 1 / ( k + n ) leads back to normal (cid:15)
-greedy action selection (which can therefore be seen as a default hyperparameter choice).
At the other extreme, choosing p ex =means the Exploiter will exclusively use an option and call an Explorer when acting non-greedy.
This would mean that (cid:15) -greedy is now, with probability (cid:15) , causing the agent to use consistent, multi-step exploration – instead of a single step of random action selection.
Note that in this setting, we should make sure that the Explorers explore the whole state-action space, in order to ensure thorough exploration.
3.3. Comparison with the weighted sum method
In this subsection we answer the problems that arise from using the weighted sum reward scheme, R = R e + βR
i , mentioned in subsection 3.1, point-by-point.
Constraints on the reward function Since we decoupled the Exploiter and Explorer, the Explorer is now an independent agent that can learn from any reward function f ir .
It does not matter if the function is non-stationary, or expressive, or does not disappear in time.
Loss of knowledge and generalization
The taskindependent knowledge about the environment –regarding how to explore , or any other skill the Explorers might have learnt– is now contained in the Explorers as usable policies.
In the context of task task-transfer, the Exploiter alone can be switched out for a new Exploiting agent, while keeping the set of f ( j ) ir , options and learnt Explorers intact (in green on Fig. 1).
Hyperparameter tuning The β exploration hyperparameter is removed completely from the method, which means we do not need to ﬁne-tune it to balance out exploitation and exploration.
The agent can decide how much it explores.
Note that in return, we have a whole Explorer agent to take care of.
Extensions to multiple reward functions When using
Explorers and Exploration Options, the extension to multiple reward functions f ( j ) ir is natural and requires little extra work.
4. Experiments 4.1. Environment and Setting
We evaluate our method in a tabular setting, on the Shortcut
Maze environment (Sutton & Barto, 2018), where a shorter path is opened up after N = 30 k steps (Fig 2).
Note that the environment is only interesting if the agent did indeed ﬁnd the ﬁrst longer path, or ﬁnding the shorter path becomes easy.
We implement this environment with a sparse reward of 1 when the agent reaches the goal G , and set γ = 0 .
9 <so the agent looks for the shortest geodesic path.
We use a learning rate of α = 0 .
1 for all learnt agents.
The agent is evaluated in a testing phase, using the greedy behavior (the Exploiter still has access to the options).
During such testing phases –but not in training–, the agent is limited to 100 steps to reach the goal.
We average the results over 50 runs.
4.2. Comparison with baselines
We compare our ExploreOption with other model-free baselines for exploration.
The ﬁrst baseline (referred-to as ”QL”) is a simple (cid:15) -greedy QLearning (Watkins & Dayan, 1992) with (cid:15) annealed from 1 to 0 .
1 over 5 k steps.
A constant (cid:15) = 0 .
1 is used for all other QLearning agents.
The second is an Optimistic QLearning (referred-to as ”QL O”), with Q 1 = 1 .
The third is a QLearning agent with a visit-count reward inspired from MBIE-EB 3 (Strehl & Littman, 2008), R i = 1 / (cid:112)
N
( s (cid:48) )
=:
f ir ( s, a, s (cid:48) ) with total reward R = R e + βR i , which we refer to as QLearning+VisitCounts (”QL+VC”).
Since we want the values of states to go down with visits, we use an optimistic initialization at Q 1 =
β/
(1 − γ ) , which solves the ﬁrst update with target
+ β/ (cid:112) (1) + γQ 1 .
The ExploreOption agent is implemented with a QLearning Exploiter and Explorer ( n = 1 ), and the Explorer uses the same optimistic valuesas QL+VC and is trained with reward R i alone.
Note that the Explorer does not have access to the done ﬂag indicating a terminal state, and sees the terminal G → S transition as any other (with a random action).
This is essential for positive reward functions, as otherwise the agent will be pushed off the goal state, resulting in biased exploration.
The whole agent is referred-to as ”QL+EO”.
The hyperparameters are ﬁxed using the best values of the studies from the following paragraphs.
The results are shown in Fig 3.
As expected, QLearning alone with an annealed (cid:15) does not ﬁnd the short path, having overﬁt to the longer one.
Optimistic values alone do not help much for this task, as the path opens up after the agent’s expectations wear down.
The algorithm is therefore simply slower than QLearning to ﬁnd the ﬁrst path.
Since
QL+VC uses optimistic values, its slow convergence is similar, and even slightly slower to ensure visit-count based exploration – it barely has time to ﬁnd the ﬁrst path before the second opens up.
In comparison, QL+EO ﬁnds the ﬁrst path even faster than QLearning alone, due to proper exploration, while it ﬁnds the second path much faster than the visit-count based approach.
As we decoupled exploration from exploitation, the agent is able to beneﬁt from consistent exploration without harming the greedy behavior.
4.3. c switch and intrinsic reward functions
The c switch hyperparameter controls the number of steps the Explorer acts before the Exploiter takes the control back.
With c switch = 1 , the Explorer has the least time and expressiveness, close to (cid:15) -greedy.
With c switch = ∞ (implemented arbitrarily as c switch = 3600 ), the Explorer plays until the end of the episode.
The hyperparameter can be associated with the β hyperparameter from the weighted sum scheme, as c switch appears when decoupling to get rid of β , and β also controls how much we care about exploration.
In this section we compare these two hyperparameters, as well as the approaches’ sensitivity to f ir , on a set of 3 intrinsic reward functions.
The ﬁrst intrinsic reward function we use is the inverse square-root of the visit-counts from the baselines, giving more reward to less-often visited states: f ir ( s, a, s (cid:48) )
=
1 / (cid:112) N
( s (cid:48) ) , which we refer to as Inverse sqrt .
Based on the same intuition, the second function we use is f ir ( s, a, s (cid:48) )
= − (cid:112) N
( s (cid:48) ) referred to as
Negative sqrt .
This function explodes in time, but we use it to show that decoupling allows for much more expressive reward functions, and functions that do not disappear in time.
Note that regardless of the value of β > 0 , the exploration reward will eventually lead the QL+VC agent.
The third function we use is based on the Successor Representation (SR)(Dayan, 1993), deﬁned as Ψ π ( s, s (cid:48) ) :
= E π,p [ (cid:80) ∞ t =
γ t I { S t = s (cid:48) } | S
= s ] , and can be interpreted as a form of generalization or distance over states, based on the similarities of the states they lead to, i.e. their successors.
The SR can conveniently be learnt with TD(0) and it has been shown recently (Machado et al., 2018) that the norm of the SR can be used as an intrinsic reward signal, which makes it one of the most general intrinsic reward functions available.
The resulting intrinsic reward is then f ir ( s, a, s (cid:48) )
= 1 / (cid:107) Ψ ( s ) (cid:107) 1 , referred to as
Successor Rep .
We use the same optimistic starts as for
Inverse sqrt .
In order to compare the reward functions and hyperparameters, we plot the average time the agent takes to reach the goal during the ﬁrst part of training (before the path opens up; top subplot) and second part of training (after the path opens up; bottom subplot) when varying the hyperparameters c switch and β .
Different reward functions correspond to different curves, as the legend details – the results are shown in Fig.
First, let’s focus on our approach (QL+EO – left plots, ﬁrst three curves of the legend).
We observe that the method is extremely robust to changes in the function: the results are virtually the same, with a slight delay for the
Successor Rep to converge in the ﬁrst phase.
Regarding c switch , as expected, the method fails to ﬁnd the second path, or is exceedingly slow, for the lowest values of c switch (for 1 and 3 ).
In such cases, the Explorer does not have enough time to take the Exploiter out of the sub-optimal trajectory, hence the shorter path is never found, leading to an average (and worst possible) performance of 16 steps to the goal (lower plot).
Remember that without the option, the Exploiter is a normal (cid:15) -greedy QLearning algorithm.
Note that the highest values of c switch take the most time to ﬁnd the ﬁrst path, which is likely due to the agent being taken out the temporary optimal path when the task is straightforward.
Any c switch ≥ 5 ﬁnds the shorter path consistently and very fast.
Even the ﬁrst path is found faster with intermediate values of c switch .
These observations align with the Hierarchical RL study (Nachum et al., 2019): multi-step exploration, i.e. through temporal abstractions, is essential to achieve our performance.
Since most values show good results, we arbitrarily selected c switch = 15 for the baselines.
The robustness of the approach to changes in the intrinsic reward function is striking, especially since
Negative sqrt is so expressive, and Successor Rep does not come with the guarantees that Inverse sqrt possesses.
QL+EO beneﬁts just as well from all of them, as long as they lead to consistent exploration.
In comparison, QL+VC (right plots) has much less attractive results.
Inverse sqrt , as mentioned in the comparison, suffers from a difﬁcult β tuning, where high values lead to very delayed ﬁnding of the ﬁrst path, while lower values simply fail to ﬁnd the second path altogether.
This shows that trying to combine both policies into one leads to balancing out when one or the other takes the lead, rather than beneﬁt- ing from both behaviors.
The Successor Rep ﬁnds the ﬁrst path slightly faster, but no value of β was found to discover the second path at all.
Regarding Negative sqrt , averaging does not display the severe instabilities, and the average time in the second phase is actually often higher than 16 .
The value β = 0 .
001 is unstable at ﬁrst but settles down to the optimal path; however, the visit counts will eventually blow the agent out of the correct trajectory.
4.4. Multiple Explorers
As we have seen in Section 4.2, we might want to learn from multiple reward functions, as they offer a spectrum of behaviors.
Since our ExploreOption approach scales naturally with the number of reward functions available, we can try this idea with all 3 intrinsic functions used before: Inverse sqrt , Negative sqrt and
Successor Rep .
The result are shown in the curve ”All” from Fig.
4 (left).
The outcome is virtually the same as for any of those functions separately, with very close performance to
Inverse sqrt in particular.
Small to no changes to performance compared to the functions alone was to be expected, as they all provided very similar results, all focus on exploration, and the spectrum of behaviors that can emerge from such a simple Gridworld is limited.
However, the point is that the integration of several intrinsic reward functions – and therefore several resulting intrinsically motivated policies – was trivial and did not harm performance.
We believe this is a fundamental improvement on the traditional approach to intrinsic motivation.
4.5. Random Explorer
Learning an additional agent and an intrinsic function can be computationally expensive.
This raises the question of the necessity of learning an exploratory policy: how much would the performance or sample-efﬁciency be affected if the Explorer did not require resources?
We answer this question by replacing the (learnt) QLearning algorithm of the Explorer by a random Explorer agent.
The results are reported on the left of Fig 4, ”Random Explorer” curve.
Note that for c switch = 1 , we fall back to QLearning.
For all higher values, the ﬁrst path is found faster, showing that multi-step exploration, even with a random policy, is valuable.
Interestingly, the agent ﬁnds the short path for the highest values of c switch , even though performance is slower than the learnt agents, and less robust to c switch .
The gap in performance between the agents with a learnt or random Explorer is already big, but it is bound to widen considerably with more complex environments and intrinsic reward functions, where the random behavior can become useless 5 .
4.6. Probability of the ExploreOption.
As introduced in subsection 3.2.4, the p ex hyperparameter allows to sample the options more frequently in an (cid:15) -greedy setting.
Fig 5 provides a study of the p ex hyperparameter.
For the special case p ex = 0 , it is interesting to see that although the method is nearly similar to QLearning, the agent can still select the option when breaking ties randomly after initialization, hence the agent still ﬁnds the ﬁrst path faster than QLearning.
Apart from p ex =
none of the values prevent the agent from ﬁnding the shortest path, and the algorithm is relatively robust to it.
Lower probabilities take slightly longer to ﬁnd the path, since exploration is less frequent.
We used the intuitive balance p ex = 0 .
5 so far.
4.7. Option learning rate
In a tabular setting, each state-action pair ( s, a ) has its own
Q value, and in particular, we learn the Q ( · , a explore ) values for all states.
However, intuitively, proper usage of the option requires function approximation in order to generalize over states.
To test this hypothesis, we tried setting the option learning rate to 0 – the option would only be used through (cid:15) -greedy, and not learnt.
We found that the performance did not change : the agent did not need to learn to use the option to beneﬁt form the Explorer’s behavior off-policy.
In the experiments so far, we used arbitrarily the learning rate α explore = 0 . 001 , but any value α explore ≤ α could work.
This observation is important, and we can summarize what we learnt from these experiments.
The gap in performance of QL+EO with QL and QL+VC in this tabular setting can be attributed to just three main factors: ﬁrst, multistep exploration, i.e. c switch > 1 ; second, an independent
Explorer from which to learn off-policy; and third, a learnt (consistent) Explorer.
5. Discussion 5.1. Limitations
A major ﬂaw of the algorithm presented here is the arbitrary, and potentially hard to tune, c switch parameter.
Several alternatives are possible, such as learning c switch (Lakshminarayanan et al., 2017; Sharma et al., 2017); learning an option termination function (Bacon et al., 2017); or learning a stop action for the Explorer.
The method proved robust to c switch in a tabular setting; future work will study it in more complex environments.
Our method relies heavily on the quality of the f ir functions to generate rewards that would independently lead to a powerful exploration policy.
We should keep in mind that these functions are not perfect, and might not properly span the state-action space.
5.2. Learning from alternative signals
In Horde (Sutton et al., 2011), General Value Functions (GVFs) are a generalization of value-functions to pseudorewards and pseudo-discounts .
Horde is then a collection of multiple GVF learners, called demons .
Control demons learn action-value functions, while prediction demons learn values functions.
This framework generates a massive knowledge representation of the environment based on value functions.
GVFs and Horde share many similarities with the framework of auxiliary task learning (Jaderberg et al., 2017;
Mirowski et al., 2017).
There the agent learns, in addition to the task, auxiliary tasks that can be control tasks (e.g. pixel control, feature control), or prediction tasks (e.g.
Depth or reward prediction).
The weights of the function approximator are shared with all tasks in order to build a powerful and general representation .
From these viewpoints, all our Explorers can be considered control tasks or demons, amassing knowledge about the environment through the reward function they try to maximize.
In addition to this, an important observation from the intrinsic motivation literature is that virtually all methods rely on learning interesting properties about the MDP, e.g. a Forward Model, an Inverse Dynamics Model, the Successor Representation, or density models.
All of these can then be seen as prediction tasks or demons, also gathering task-independent knowledge about the environment.
Therefore, combining everything, the ﬁnal agent is a multiheaded function approximator, learning all Exploiter, Explorers and f ( j ) ir jointly and off-policy.
This agent not only builds a very strong task-independent representation; it also contains a lot of information about the environment - how to explore, or other behaviors; as well as predictive knowledge such as forward models or a Successor Representation.
This agent would be very easily transferred across tasks – we only need to switch in a new Exploiter head.
This will be the object of future work.
6. Conclusion
We introduced a new approach for learning from intrinsic motivation.
We carried out experiments with QLearning agents on the simple non-stationary Shortcut Maze environment, where exploration is needed to ﬁnd a shortcut.
We demonstrated empirically that its sample-efﬁciency, adaptability and exploration performance vastly surpass that of the traditional approach, a weighted sum of rewards.
We showed that:
Exploration through temporal abstractions over several steps, especially with a learnt Explorer, lead to much better sample-efﬁciency and exploration; 2.
Our method adapts much better to miscellaneous intrinsic reward functions than the usual R = R e + βR i ; 3.
Decoupling exploiting and exploring policies improves both exploration and exploitation while preserving knowledge
about the
environment
and scaling easily with several reward functions. We presented a view from GVFs and auxiliary tasks, which vastly broadens the scope of the method for multiple Explorers and reward functions, giving glimpses of an agent with a strong representation, world knowledge, control knowledge and task-transfer abilities.