T EMPORALLY -E XTENDED (cid:15) -G REEDY E XPLORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work on exploration in reinforcement learning (RL) has led to a series of increasingly complex solutions to the problem.
This increase in complexity often comes at the expense of generality.
Recent empirical studies suggest that, when applied to a broader set of domains, some sophisticated exploration methods are outperformed by simpler counterparts, such as (cid:15) -greedy.
In this paper we propose an exploration algorithm that retains the simplicity of (cid:15) -greedy while reducing dithering.
We build on a simple hypothesis: the main limitation of (cid:15) - greedy exploration is its lack of temporal persistence, which limits its ability to escape local optima.
We propose a temporally extended form of (cid:15) -greedy that simply repeats the sampled action for a random duration.
It turns out that, for many duration distributions, this sufﬁces to improve exploration on a large set of domains.
Interestingly, a class of distributions inspired by ecological models of animal foraging behaviour yields particularly strong performance.
I NTRODUCTION
Exploration is widely regarded as one of the most important open problems in reinforcement learning (RL).
The problem has been theoretically analyzed under simplifying assumptions, providing reassurance and motivating the development of algorithms (Brafman and Tennenholtz, 2002; Asmuth et al., 2009; Azar, Osband, and Munos, 2017).
Recently, there has been considerable progress on the empirical side as well, with new methods that work in combination with powerful function approximators to perform well on challenging large-scale exploration problems (Bellemare et al., 2016;
Ostrovski et al., 2017; Burda et al., 2018; Badia et al., 2020b).
Despite all of the above, the most commonly used exploration strategies are still simple methods like (cid:15) -greedy, Boltzmann exploration and entropy regularization (Peters, Mulling, and Altun, 2010;
Sutton and Barto, 2018).
This is true for both work of a more investigative nature (Mnih et al., 2015) and practical applications (Levine et al., 2016; Li et al., 2019).
In particular, many recent successes of deep RL, from data-center cooling to Atari game playing, rely heavily upon these simple exploration strategies (Mnih et al., 2015; Lazic et al., 2018; Kapturowski et al., 2019).
Why does the RL community continue to rely on such naive exploration methods?
There are several possible reasons.
First, principled methods usually do not scale well.
Second, the exploration problem is often formulated as a separate problem whose solution itself involves quite challenging steps.
Moreover, besides having very limited theoretical grounding, practical methods are often complex and have signiﬁcantly poorer performance outside a small set of domains they were speciﬁcally designed for.
This last point is essential, as an effective exploration method must be generally applicable.
Naive exploration methods like (cid:15) -greedy, Boltzmann exploration and entropy regularization are general because they do not make strong assumptions about the underlying domain.
As a consequence, they are also simple , not requiring too much implementation effort or per-domain tuning.
This makes them appealing alternatives even when they are not as efﬁcient as some more complex variants.
Perhaps there is a middle ground between simple yet inefﬁcient exploration strategies and more complex, though efﬁcient, methods.
The method we propose in this paper represents such a compromise.
We ask the following question: how can we deviate minimally from the simple exploration strategies adopted in practice and still get clear beneﬁts?
In more pragmatic terms, we want a simple-to-implement algorithm that can be used in place of naive methods and lead to improved exploration.
In order to achieve our goal we propose a method that can be seen as a generalization of (cid:15) -greedy, perhaps the simplest and most widely adopted exploration strategy.
As is well known, the (cid:15) -greedy algorithm selects an exploratory action uniformly at random with probability (cid:15) at each time step.
Besides its simplicity, (cid:15) -greedy exploration has two properties that contribute to its universality: 1 It is stationary , i.e. its mechanics do not depend on learning progress.
Stationarity is important for stability, since an exploration strategy interacting with the agent’s learning dynamics results in circular dependencies that can in turn limit exploration progress.
In simple terms: bad exploratory decisions can hurt the learned policy which can lead to more bad exploration.
2 It provides full coverage of the space of possible trajectories.
All sequences of states, actions and rewards are possible under (cid:15) -greedy exploration, albeit some with exceedingly small probability.
This guarantees, at least in principle, that no solutions are excluded from consideration.
Convergence results for RL algorithms rely on this sort of guarantee (Singh et al., 2000).
This may also explain sophisticated exploration methods’ use of (cid:15) -greedy exploration (Bellemare et al., 2016).
However, (cid:15) -greedy in its original form also comes with drawbacks.
Since it does not explore persistently, the likelihood of deviating more than a few steps off the default trajectory is vanishingly small.
This can be thought of as an inductive bias (or “prior”) that favors transitions that are likely under the policy being learned (it might be instructive to think of a neighbourhood around the associated stationary distribution).
Although this is not necessarily bad, it is not difﬁcult to think of situations in which such an inductive bias may hinder learning.
For example, it may be very difﬁcult to move away from a local maximum if doing so requires large deviations from the current policy.
The issue above arises in part because (cid:15) -greedy provides little ﬂexibility to adjust the algorithm’s inductive bias to the peculiarities of a given problem.
By tuning the algorithm’s only parameter, (cid:15) , one can make deviations more or less likely, but the nature of such deviations is not modiﬁable.
To see this, note that all sequences of exploratory actions are equally likely under (cid:15) -greedy, regardless of the speciﬁc value used for (cid:15) .
This leads to a coverage of the state space that is largely deﬁned by the current (“greedy”) policy and the environment dynamics (see Figure 1 for an illustration).
In this paper we present an algorithm that retains the beneﬁcial properties of (cid:15) -greedy while at the same time allowing for more control over the nature of the induced exploratory behavior.
In order to achieve this, we propose a small modiﬁcation to (cid:15) -greedy: we replace actions with temporallyextended sequences of actions, or options (Sutton, Precup, and Singh, 1999).
Options then become a mechanism to modulate the inductive bias associated with (cid:15) -greedy.
We discuss how by appropriately deﬁning a set of options one can “align” the exploratory behavior of (cid:15) -greedy with a given environment or class of environments; we then show how a very simple set of domain-agnostic options work surprisingly well across a variety of well known environments.
B ACKGROUND AND N OTATION
Reinforcement learning can be set within the Markov Decision Process (MDP) formalism (Puterman, 1994).
An MDP M is deﬁned by the tuple ( X , A , P , R , γ ), where x ∈ X is a state in the state space, a ∈ A is an action in the action space, P ( x (cid:48) | x, a ) is the probability of transitioning from state x to state x (cid:48) after taking action a , R :
X ×
A → R is the reward function and γ ∈ [0 , 1) is the discount factor.
Let P ( A ) denote the space of probability distributions over actions; then, a policy π : X → P ( A ) assigns some probability to each action conditioned on a given state.
We will denote by π a = 1 a the policy which takes action a deterministically in every state.
The agent attempts to learn a policy π that maximizes the expected return or value in a given state,
[Equation]
where V π , Q π are the value and action-value functions of π .
The greedy policy for action-value function Q takes the action arg max a ∈A Q ( x, a ) , ∀ x ∈ X .
In this work we primarily rely upon methods based on the Q -learning algorithm
(Sutton and Barto, 2018), which attempts to learn the optimal policy by approximating the Bellman optimality operator:
[Equation]
[Figure]
In practice, the state space X is often too large to represent exactly and thus we have Q θ ( x, a )
≈
Q ( x, a ) for a function approximator parameterized by θ .
We will generally use some form of differentiable function approximator Q θ , whether it be linear in a ﬁxed set of basis functions, or an artiﬁcial neural network.
We update parameters θ to minimize a squared or Huber loss between the left- and right-hand sides of equation 1, with the right-hand side held ﬁxed (Mnih et al., 2015).
In addition to function approximation, it has been argued that in order to scale to large problems, RL agents should be able to reason at multiple temporal scales (Dayan and Hinton, 1993; Parr and Russell, 1998; Sutton, Precup, and Singh, 1999; Dietterich, 2000).
One way to model temporal abstraction is via options (Sutton, Precup, and Singh, 1999), i.e. temporally-extended courses of action.
In the most general formulation, an option can depend on the entire history between its initiation time step t and the current time step t + k , h t : t + k ≡ x t a t x t +1 ...
a t + k − 1 x t + k .
Let H be the space of all possible histories; a semi-Markov option is a tuple ω ≡ ( I ω , π ω , β ω ) , where I ω ⊆
X is the set of states where the option can be initiated, π ω : H → P ( A ) is a policy over histories, and β ω : H (cid:55)→ [0 , 1] gives the probability that the option terminates after observing history
h
(Sutton, Precup, and Singh, 1999).
As in this work we will use options for exploration, we will assume that I ω
=
X , ∀ ω .
Once an option ω is selected, the agent takes actions a ∼ π ω ( · | h ) after having observed history h and at each step terminates the option with probability β ω ( h ) .
It is worth emphasizing that semiMarkov options depend on the history since their initiation, but not before.
Also, they are usually deﬁned with respect to a statistic of histories h ∈ H ; for example, by looking at the length of h one can deﬁne an option that terminates after a ﬁxed number of steps.
E XPLORATION IN R EINFORCEMENT L EARNING
At its core, RL presents the twin challenges of temporal credit assignment and exploration.
The agent must accurately, and efﬁciently, assign credit to past actions for their role in achieving some long-term return.
However, to continue improving the policy, it must also consider behaviours it estimates to be sub-optimal.
This leads to the well-known exploration-exploitation trade-off .
Because of its central importance in RL, exploration has been among the most studied topics in the ﬁeld.
In ﬁnite state-action spaces, the theoretical limitations of exploration, with respect to sample complexity bounds, are fairly well understood (Azar, Osband, and Munos, 2017; Dann, Lattimore, and Brunskill, 2017).
However, these results are of limited practical use for two reasons.
First, they bound sample complexity by the size of the state-action space and horizon, which makes their immediate application in large-scale or continuous state problems difﬁcult.
Second, these algorithms tend to be designed based on worst-case scenarios, and can be inefﬁcient on problems of actual interest.
Bayesian RL methods for exploration address the explore-exploit problem integrated with the estimation of the value-function itself (Kolter and Ng, 2009).
Generally such methods strongly depend upon the quality of their priors, which can be difﬁcult to set appropriately.
Thompson sampling methods (Thompson, 1933; Osband, Russo, and Van Roy, 2013) estimate the posterior distribution of value-functions, and act greedily according to a sample from this distribution.
As with other methods which integrate learning and exploration into a single estimation problem, this creates non-stationary, but temporally persistent, exploration.
Other examples of this type of exploration strategy include randomized prior functions (Osband, Aslanides, and Cassirer, 2018), uncertainty Bellman equations (O’Donoghue et al., 2018), NoisyNets (Fortunato et al., 2017), and successor uncertainties (Janz et al., 2019).
Although quite different from each other, they share key commonalities: non-stationary targets, temporal persistence, and exploration based on the space of value functions.
At the other end of the spectrum, there have recently been successful attempts to design algorithms with speciﬁc problems of interest in mind.
Certain games from the Atari-57 benchmark (e.g. M ON -
TEZUMA ’ S R EVENGE , P ITFALL !, P RIVATE E YE ) have been identiﬁed as ‘hard exploration games’ (Bellemare et al., 2016), attracting the attention of the research community, leading to signiﬁcant progress in terms of performance (Ecoffet et al., 2019; Burda et al., 2018).
On the downside, these results have been usually achieved by algorithms with little or no theoretical grounding, adopting specialized inductive biases, such as density modeling of images (Bellemare et al., 2016; Ostrovski et al., 2017), error-seeking intrinsic rewards (Pathak et al., 2017; Badia et al., 2020a), or perfect deterministic forward-models (Ecoffet et al., 2019).
Generally, such algorithms are evaluated only on the very domains they are designed to perform well on, raising questions of generality.
Recent empirical analysis showed that some of these methods perform similarly to each other on hard exploration problems and signiﬁcantly under-perform (cid:15) - greedy otherwise (Ali Ta¨ıga et al., 2020).
One explanation is that complex algorithms tend to be more brittle and harder to reproduce, leading to lower than expected performance in follow-on work.
However, these results also suggest that much of the recent work on exploration is over-ﬁtting to a small number of domains.
T EMPORALLY -E XTENDED E XPLORATION
There are many ways to think about exploration: curiosity, experimentation, reducing uncertainty, etc.
Consider viewing exploration as a search for undiscovered rewards or shorter paths to known rewards.
In this context, the behaviour of (cid:15) -greedy appears shortsighted because the probability of moving consistently in any direction decays exponentially with the number of exploratory steps.
In
Figure 1a we visualize the behaviour of uniform (cid:15) -greedy in an open gridworld, where the agent starts at the center-top and the greedy policy moves straight down.
Observe that for (cid:15) ≤ 0 .
5 the agent is exceedingly unlikely to reach states outside a narrow band around the greedy policy.
Even the purely exploratory policy ( (cid:15) = 1 . 0 ) requires a large number of steps to visit the bottom corners of the grid.
This is because, under the uniform policy, the probability of moving consistently in any direction decays exponentially (see Figure 1a).
By contrast, a method that explores persistently with a directed policy leads to more efﬁcient exploration of the space at various values of (cid:15) (Figure 1b).
The importance of temporally-extended exploration has been previously highlighted (Osband et al., 2016), and in general, count-based (Bellemare et al., 2016) or curiosity-based (Burda et al., 2018) exploration methods are inherently temporally-extended due to integrating exploration and exploitation into the greedy policy.
Here our goal is to leverage the beneﬁts of temporally-extended exploration without modifying the greedy policy.
There has been a wealth of research on learning options (McGovern and Barto, 2001; Stolle and
Precup, 2002; S¸ims¸ek and Barto, 2004; Bacon, Harb, and Precup, 2017; Harutyunyan et al., 2019), speciﬁcally for exploration (Machado, Bellemare, and Bowling, 2017; Machado et al., 2018b; Jinnai et al., 2019; 2020; Hansen et al., 2020).
These methods use options for exploration and to augment the action-space, adding learned options to actions available at states where they can be initiated.
In the remainder of this work, we argue for temporally-extended exploration, using options to encode a set of inductive biases to improve sample-efﬁciency.
This fundamental message is found throughout the existing work on exploration with options, but producing algorithms that are empirically effective on large environments remains a challenge for the ﬁeld.
In the next section, we discuss in more detail how the options’ policy π ω and termination β ω can be used to induce different types of exploration.
Temporally-Extended (cid:15) -Greedy
A temporally-extended (cid:15) -greedy exploration strategy depends on choosing an exploration probability (cid:15) , a set of options Ω , and a sampling distribution p with support Ω .
On each step the agent follows the current policy π for one step with probability 1 − (cid:15) , or with probability (cid:15) samples an option ω ∼ p (Ω) and follows it until termination.
Standard (cid:15) -greedy has three desirable properties that help explain its wide adoption in practice: it is simple , stationary , and promotes full coverage of the state-action space in the limit (guaranteeing convergence to the optimal policy under the right conditions).
We now discuss to what extent the proposed algorithm retains these properties.
Although somewhat subjective, it seems fair to call temporally-extended (cid:15) -greedy a simple method.
It is also stationary when the set of options Ω and distribution p are ﬁxed, for in this case its mechanics are not inﬂuenced by the collected data.
Finally, it is easy to deﬁne conditions under which temporally-extended (cid:15) -greedy covers the entire state-action space, as we discuss next.
Obviously, the exploratory behavior of temporally-extended (cid:15) -greedy will depend on the set of options Ω .
Ideally we want all actions a ∈ A to have a nonzero probability of being executed in all states x ∈ X regardless of the greedy policy π .
This is clearly not the case for all sets Ω .
In fact, this may not be the case even if for all ( x, a ) ∈ X
×
A there is an option ω ∈ Ω such that π ω ( a | hx ) > 0 , where hx represents all histories ending in x .
To see why, note that, given a ﬁxed Ω and (cid:15) > 0 , it may be impossible for an option ω ∈ Ω to be “active” in state x
(that is, either start at or visit x ).
For example, if all options in Ω terminate after a ﬁxed number of steps that is a multiple of k , temporally-extended (cid:15) -greedy with (cid:15) = 1 will only visit states of an unidirectional chain whose indices are also multiples of k .
Perhaps even subtler is that, even if all options can be active at state x , the histories hx ∈ H associated with a given action a may themselves not be realizable under the combination of Ω and the current π .
It is clear then that the coverage ability of temporally-extended (cid:15) -greedy depends on the interaction between π , Ω , (cid:15) , and the dynamics P ( ·| x, a ) of the MDP.
One way to reason about this is to consider that, once ﬁxed, these elements induce a stochastic process which in turn gives rise to a well-deﬁned distribution over the space of histories H .
Property 1 (Full coverage) .
Let M be the space of all MDPs with common state-action spaces
X , A , and Ω a set of options deﬁned over this state-action space.
Then, Ω has full coverage for M if ∀ M ∈ M , (cid:15) > 0 , and π , the semi-Markov policy µ :=
(1 − (cid:15) )
π
+ (cid:15)π ω , where ω is a random variable uniform over Ω , visits every state-action pair with non-zero probability.
Note that µ is itself a random variable and not an average policy.
We can then look for simple conditions that would lead to having Property 1.
For example, if the options’ policies only depend on the last state of the history, π ω ( ·| hx )
= π ω ( ·| x ) (i.e. they are Markov, rather than semi-Markov policies), we can get the desired coverage by having π ω ( a | x ) > 0 for all x ∈ X and all a ∈ A .
The coverage of X × A also trivially follows from having all primitive actions a ∈ A as part of Ω .
Note that if the primitive actions are the only elements of Ω we recover standard (cid:15) -greedy, and thus coverage of X × A .
Of course, in these and similar cases, temporally-extended (cid:15) -greedy allows for convergence to the optimal policy under the same conditions as its precursor.
This view of temporally-extended (cid:15) -greedy as inducing a stochastic process also helps to understand its differences with respect to its standard counterpart.
Since the induced stochastic process deﬁnes a distribution over histories we can also talk about distributions over sequences of actions.
With standard (cid:15) -greedy, every sequence of k exploratory actions has a probability of occurrence of exactly ( (cid:15)/ |A| ) k , where |A| is the size of the action space.
By changing (cid:15) one can uniformly change the probabilities of all length- k sequences of actions, but no sequence can be favored over the others.
Temporally-extended (cid:15) -greedy provides this ﬂexibility through the deﬁnition of Ω ; speciﬁcally, by deﬁning the appropriate set of options one can control the temporal correlation between actions.
This makes it possible to control how quickly the algorithm converges, as we discuss next.
Efﬁcient exploration
For sample-efﬁciency we want to cover the state-action space quickly .
Deﬁnition 1.
The cover time of an RL algorithm is the number of steps needed to visit all state-action pairs at least once with probability 0 .
5 starting from the initial state distribution.
Even-Dar and Mansour (2003) show that the sample efﬁciency of Q-learning can be bounded in terms of the cover time of the exploratory policy used.
Liu and Brunskill (2018) provide an upper bound for the cover time of a random exploratory policy based on properties of the MDP.
Putting these results together, we have the characterization of a class of MDPs for which Q-learning plus (cid:15) -greedy
exploration is sample efﬁcient (that is, it converges in polynomial time).
Normally, the efﬁciency of (cid:15) -greedy Q-learning is completely determined by the MDP: given an MDP, either the algorithm is efﬁcient or it is not.
We now discuss how by replacing (cid:15) -greedy exploration with its temporally-extended counterpart we can have efﬁcient exploration on a much broader class of
MDPs.
To understand why this is so, note that the deﬁnition of the set of options Ω can be seen as the deﬁnition of a new MDP in which histories play the role of states and options play the role of actions.
Hence, by appropriately deﬁning Ω , we can have an MDP in which random exploration is efﬁcient.
We now formalize this notion by making explicit properties of Ω that lead to efﬁcient exploration:
Assumption 1.
For an MDP M and set of options Ω , there exists n max ∈ N such that ∀ x, y ∈ X ,
∃ ω ∈ Ω leading to E π ω
[ t | x 0 = x, x t = y ] ≤ n max .
Theorem 1.
For any irreducible MDP, let Ω be a set of options satisfying Assumption 1 with n max ≤ Θ( |X||A| ) .
Then, temporally-extended (cid:15) -greedy with sampling distribution p satisfyingρ ( ω ) ≤ Θ( |X||A| ) , ∀ ω ∈ Ω , has polynomial sample complexity.
In many cases it is easy to deﬁne options that satisfy Assumption 1, as we will discuss shortly.
But even when this is not the case, one can learn options deliberately designed to have this property.
For example, Jinnai et al.
(2019; 2020) learn point-options (transitioning from one state to one other state) that explicitly minimize cover time.
The approach proposed by Machado et al.
(2017; 2018b) also leads to options with a small cover time.
Alternatively, Whitney et al.
(2020) learn an embedding of action sequences such that sequences with similar representations also have similar future state distributions.
They observe that sampling uniformly in this abstracted action space yields action sequences whose future state distribution is nearly uniform over reachable states.
We can interpret such an embedding, coupled with a corresponding decoder back into primitive actions, as an appealing approach to learning open-loop options for temporally-extended (cid:15) -greedy exploration.
Next, we propose a concrete form of temporally-extended (cid:15) -greedy which requires neither learning Ω nor speciﬁc domain knowledge.
These options encode a commonly held inductive bias: actions have (largely) consistent effects throughout the state-space.
(cid:15)z -Greedy
We begin with the options ω a ≡ ( X , π a , β ) ,
where π a ( h ) =
1 a and β ( h )
=
for all h ∈ H , and consider a single modiﬁcation, temporal persistence.
Let ω an ≡ ( X , π a , β ( h ) =|
h | =
= n ) be the option which takes action a for n steps and then terminates.
Our proposed algorithm, is to let Ω = { ω an } a ∈A ,n ≥ 1 and p to be uniform over actions with durations distributed according to some distribution z .
Intuitively, we are proposing the set of semi-Markov options made up of all “action-repeat” policies for all combinations of actions and repeat durations, with a parametric sampling distribution on durations.
This exploration algorithm is described by two parameters, (cid:15) dictating when to explore, and z dictating the degree of persistence.
Notice that when z puts all mass on n = 1 , this is standard (cid:15) -greedy; more generally this combination of distributions forms a composite distribution with support [0 , ∞ ) , which is to say that with some probability the agent explores for n = 0 steps, corresponding to following its usual policy, and for all other n > 0 the agent explores following an action-repeat policy.
A natural question arises: what distribution over durations should we use?
To help motivate this question, and to help understand the desirable characteristics, consider Figure 6 (in Appendix) which shows a modiﬁed chain MDP with two actions.
Taking the ‘down’ action immediately terminates with the speciﬁed reward, whereas taking the ‘right’ action progresses to the next state in the chain.
Similar to other exploration chain-like MDPs, (cid:15) -greedy performs poorly here because the agent must move consistently in one direction for an arbitrary number of steps (determined by the discount) to reach the optimal reward.
Instead, we consider the effects of three classes of duration distribution: exponential ( z ( n ) ∝ λ n − 1 ), uniform ( z ( n ) = 1 n ≤ N /N ), and zeta ( z ( n ) ∝ n − µ ).
Figure 6b shows the average return achieved by these distributions as their hyper-parameters are varied.
This problem illustrates that, without prior knowledge of the MDP, it is important to support long durations, such as with a heavy-tailed distribution like zeta.
Why not simply allow uniform over an extremely large support?
Doing so would effectively force ‘pure’ exploration without any exploitation, because this form of ballistic exploration would simply continue exploring indeﬁnitely.
Indeed, we can see in Figure 6 that the uniform distribution leads to poor performance (the same is true for the zeta distribution as µ → 1 , which also leads to ballistic exploration).
On the other hand, short durations lead to frequent switching and vanishingly small probabilities of reaching larger rewards at all.
This trade-off leads to the existence of an optimal value of µ for the zeta distribution that can vary by domain (Humphries et al., 2010), and is illustrated by the inverted U-curve in Figure 6.
A class of ecological models for animal foraging known as
L´evy ﬂights follow a similar pattern of choosing a direction uniformly at random, and following that direction for a duration sampled from a heavy-tailed distribution.
Under certain conditions, this has been shown to be an optimal foraging strategy, a form of exploration for a food source of unpredictable location (Viswanathan et al., 1996; 1999).
Thus, in the remainder of this work we will use the zeta distribution, with µ = 2 unless otherwise speciﬁed, and call this combination of (cid:15) chance to explore and zeta-distributed durations, (cid:15)z -Greedy exploration 1 .
[Figure]
E XPERIMENTAL R ESULTS
We have emphasized the importance of simplicity, generality (via convergence guarantees), and stationarity of exploration strategies.
We proposed a simple temporally-extended (cid:15) -greedy algorithm, (cid:15)z -greedy, and saw that a heavy-tailed duration distribution yielded the best trade-off between temporal persistence and sample efﬁciency.
In this section, we present empirical results on tabular, linear, and deep RL settings, pursuing two objectives:
The ﬁrst is to demonstrate the generality of our method in applying it across domains as well as across multiple value-based reinforcement learning algorithms (Q-learning, SARSA, Rainbow, R2D2).
Second, we make the point that exploration comes at a cost, and that (cid:15)z -greedy improves exploration with signiﬁcantly less loss in efﬁciency on dense-reward domains compared with existing exploration algorithms.
Small-Scale: Tabular & Linear RL
We consider four small-scale environments (DeepSea, GridWorld, MountainCar, CartPole swingup-sparse), conﬁgured to be challenging sparse-reward exploration problems.
Full details for each are given in the Appendix.
DeepSea and GridWorld use a tabular representation while MountainCar and CartPole use linear function approximation on an order 5 and 7 Fourier basis respectively (Konidaris, Osentoski, and Thomas, 2011).
In Figure 2 we present results comparing (cid:15) -greedy and (cid:15)z -greedy on these four domains.
Unless otherwise speciﬁed, the hyper-parameters and training settings for these two methods are identical.
For each domain we show (i) learning curves showing average return against training episodes 2 , (ii) average ﬁrst-visit times on states during pure ( (cid:15)
= 1 . 0 ) exploration for (cid:15) -greedy and (iii) (cid:15)z -greedy.
The results show that (cid:15)z -greedy provides signiﬁcantly improved performance on these domains, and the ﬁrst-visit times indicate signiﬁcantly better state-space coverage compared to (cid:15) -greedy.
Atari-57:
Deep RL
Motivated by the results in tabular and linear settings, we now turn to deep
RL and evaluate performance on 57 Atari 2600 games in the Arcade Learning Environment (ALE) (Bellemare et al., 2013).
To demonstrate the generality of the approach, we apply (cid:15)z -greedy to two state-of-the-art deep RL agents, Rainbow (Hessel et al., 2018) and R2D2 (Kapturowski et al., 2019).
We compare with baseline performance as well as the performance of a recent intrinsic motivationbased exploration algorithm: CTS-based pseudo-counts (Bellemare et al., 2016) in Rainbow and
RND (Burda et al., 2018) in R2D2, each tuned for performance comparable with published results.
Finally, in R2D2 experiments we also compare with a Bootstrapped DQN version of R2D2 (Osband et al., 2016), providing an exploration baseline without intrinsic rewards.
We include pseudo-code and hyper-parameters in the Appendix, though the implementation of (cid:15)z -greedy in each case is trivial, hyper-parameters are mostly identical to previous work, and we ﬁx µ = 2 for results in this section.
Our ﬁndings (see Figure 3) show that (cid:15)z -greedy improves performance on the hard
[Figure]
o r m a l i
z e d
S c
o r e
H u m a n - n o r m a l i
z e d
S c
o r e
[Figure]
exploration tasks with little to no loss in performance on the rest of the suite.
By comparison, we observe that the intrinsic motivation methods often (although not always) outperform (cid:15)z -greedy on the hard exploration tasks, but at a signiﬁcant loss of performance on the rest of the benchmark.
The results in Figure 3 show median human-normalized score over the 57 games and the human-gap, measuring how much the agent under-performs humans on average (see Appendix D for details).
We consider the median to indicate overall performance on the suite and the human-gap to illustrate gains on the hard exploration games where agents still under-perform relative to humans, with full per-game and mean performance given in the Appendix.
Table 1 gives the ﬁnal performance of each of the agents in terms of these summary statistics.
Figure 4 shows representative examples of per-game performance for the R2D2-based agents.
These per-game results make a strong point, that even on the hard exploration games the inductive biases of intrinsic motivation methods may be poorly aligned, and that outside a small number of games these methods signiﬁcantly hurt performance, whereas our proposed method improves exploration while avoiding this signiﬁcant loss elsewhere.
To demonstrate that the effectiveness of our method does not crucially depend on evaluation on deterministic domains, in Appendix E we additionally show a similar comparison of the Rainbowbased agents on a stochastic variant of the Atari-57 benchmark using ‘sticky actions’ (Machado et al., 2018a). The results are qualitatively similar: while all agent variants do somewhat worse on the stochastic compared to the deterministic case, (cid:15)z -greedy improves over the baseline and (cid:15) -greedy on the hardest exploration domains while not substantially affecting performance in others [Table] and coming out on top in terms of mean and median human-normalized performance as well as human-gap (Table 5, Figures 9, 10, 21).
[Table]
Bootstrap
[Table]
D ISCUSSION AND C ONCLUSIONS
We have proposed temporally-extended (cid:15) -greedy, a form of random exploration performed by sampling an option and following it until termination, with a simple instantiation which we call (cid:15)z -greedy.
We showed, across domains and algorithms spanning tabular, linear and deep reinforcement learning that (cid:15)z -greedy improves exploration and performance in sparse-reward environments with minimal loss in performance on easier, dense-reward environments.
Further, we showed that compared with other exploration methods (pseudo-counts, RND, Bootstrap), (cid:15)z -greedy has comparable performance averaged over the hard-exploration games in Atari, but without the signiﬁcant loss in performance on the remaining games.
Although action-repeats have been a part of deep RL algorithms since DQN, and have been considered as a type of option (Schoknecht and Riedmiller, 2002; 2003; Braylan et al., 2015; Lakshminarayanan, Sharma, and Ravindran, 2017; Sharma, Lakshminarayanan, and Ravindran, 2017), their use for exploration with sampled durations does not appear to have been studied before.
Generality and Limitations.
Both (cid:15) - and (cid:15)z -greedy are guaranteed to converge in the ﬁnite stateaction case, but they place probability mass over exploratory trajectories very differently, thus encoding different inductive biases.
We expect there to be environments where (cid:15)z -greedy signiﬁcantly under-performs (cid:15) -greedy.
Indeed, these are easy to imagine: DeepSea with action effects randomized per-state (see Appendix Figure 14), GridWorld with many obstacles that immediately end the episode (‘mines’), a maze changing direction every few steps, etc.
More generally, the limitations of (cid:15)z -greedy are: (i) Actions may not homogeneously (over states) correspond to a natural notion of shortest-path directions in the MDP.
(ii) Action spaces may be biased (e.g. many actions have the same effect), so uniform action sampling may produce undesirable biased drift through the MDP.
(iii) Obstacles and dynamics in the MDP can cause long exploratory trajectories to waste time (e.g. running into a wall for thousands of steps), or produce uninformative transitions (e.g. end of episode, death).
In Appendix F we report on a series of experiments investigating (cid:15)z -Greedy’s sensitivities to such modiﬁcations of the Gridworld domain and ﬁnd that its performance degrades gracefully overall.
These limitations are precisely where we believe future work is best motivated.
How can an agent learn stationary, problem-speciﬁc notions of direction and explore in that space efﬁciently?
How to avoid wasteful long trajectories, perhaps by truncating early?
This form of exploration bears similarity to the L´evy-ﬂights model of foraging, where an animal will abruptly end foraging as soon as food is within sight.
Could we use discrepancies in value along a trajectory to similarly truncate exploration early?
Recent work around learning action representations appear to be promising directions (Tennenholtz and Mannor, 2019; Chandak et al., 2019)
APPENDICES A C OVER T IME A NALYSIS
Assumption 1.
For an MDP M and set of options Ω , there exists n max ∈ N such that ∀ x, y ∈ X ,
∃ ω ∈ Ω leading to E π ω
[ t | x 0 = x, x t = y ] ≤ n max .
Theorem 1.
For any irreducible MDP, let Ω be a set of options satisfying Assumption 1 with n max ≤ Θ( |X||A| ) .
Then, temporally-extended (cid:15) -greedy with sampling distribution p satisfyingρ ( ω ) ≤ Θ( |X||A| ) , ∀ ω ∈ Ω , has polynomial sample complexity.
Proof.
Liu and Brunskill (2018) establish a PAC RL bound, leading to polynomial sample complexity, for random exploration using primitive actions when 1 min x φ ( x ) and 1 h are polynomial in states and actions, with steady-state distribution φ and Cheeger constant h .
However, their result does not require actions to be primitive, and here we show how temporally-extended actions in the form of exploratory options can be used for a similar result.
We begin by bounding the steady-state probability for any state x ∈ X , where x 0 :=
arg max x φ ( x ) and n max the maximum expected path distance between two states.
We can understand this as bounding the steady-state probability of x by the product of (1) the maximal steady-state probability over states, (2) the probability of choosing an option from x 0 that reaches x .
Let ω be any option satisfying Assumption 1 for starting in state x 0 and reaching x ,
[Equation]
Next, we can similarly bound the Cheeger constant.
Recall from (Liu and Brunskill, 2018) that h = inf U
[Equation]
[Equation]
( U ) ,F (¯ U ) } where ¯ U denotes the set of states not in U ,
[Equation]
Let U (cid:48) =
{ x 0 } , then, the Cheeger constant can be bounded by,
[Equation]
E XAMPLE : C HAIN MDP
Theorem 1 clariﬁes the conditions under which temporally-extended (cid:15) -greedy is efﬁcient.
Given an
MDP, this will depend on two factors: the options and the sampling distribution.
To illustrate this point, we use the well known Chain MDP, for which (cid:15)z -Greedy satisﬁes Assumption 1.
Speciﬁcally, the requirement on z is satisﬁed by the zeta distribution ( z ( n ) = n − µ ζ ( µ ) ) but not by the geometric distribution (exponential decay).
This implies that (cid:15)z -greedy on the Chain MDP will have polynomial sample complexity when z is zeta distributed, but not when exponentially distributed.
We can see this by observing that in a Chain MDP of size N any state can be reached from the starting state within at most N steps, yielding n max ≤ |X| .
The sampling distribution ρ is uniform over actions meaning that ρ ( ω ) ≥ z ( n max ) / |A| .
Finally, we consider the speciﬁc form of the duration distribution z .
When given by a zeta distribution, z ( n ) = n − µ /ζ ( µ ) , we have
[Equation]
thus satisfying our assumption.
On the other hand, if we let the duration distribution be geometric, z ( n ) = λ (1 − λ ) n − 1 , we have
[Equation]
As 1 / (1 − λ ) > 1 and n max is only bounded by the number of states, this results in an upper bound that is exponential in the number of states and therefore does not satisfy the assumptions.
B D OMAIN SPECIFICATIONS
DeepSea (Osband, Aslanides, and Cassirer, 2018)
Parameterized by problem size N , this environment can be viewed as the lower triangle of an N × N gridworld with two actions: “down-left” and “down-right” which move diagonally down either left or right.
There is a single goal state in the far bottom-right corner, which can only be reached through a single action-sequence.
The goal reward is 1 . 0 , and there is a per-step reward of − 0 . 01 /N .
Finally, all episodes end after exactly N steps, once the agent reaches the bottom.
Therefore, the maximum possible undiscounted return is 0 . 99 .
An example with N = 20 is shown in Figure 5a.
Average ﬁrst-passage times are shown for a problem size of N = 20 in Figure 2a, and unlike other plots are logarithmically scaled, log( E [ fpt ] + 1) with contour levels in the range [0 , 16] .
In this work we use the deterministic variant of DeepSea; however, the standard stochastic version randomizes the action effects at every state.
That is, “down-left” may correspond to action indexin one state and 1 in another, and these assignments are performed randomly for each training run (consistently across episodes).
We brieﬂy mention this variant in our conclusions as an example in which our proposed method should be expected to perform poorly.
Indeed, in Figure 14 we show that such an adversarial modiﬁcation reduces (cid:15)z -greedy’s performance back to that of (cid:15) -greedy.
For experiments, we used Q-learning with a tabular function approximator, learning rate α = 1 . 0 , and (cid:15)
= 1 . 0 / ( N + 1) for problem size N .
Experiment results are averages over 30 random seeds.
GridWorld Shown in Figure 5b, this is an open single-room gridworld with four actions (“up”, “down”, “left”, and “right”), and a single non-zero reward at the goal state.
The initial state is in the top center of the grid (offset from the wall by one row), and the goal state is diagonally across from it at the other end of the room.
Notice that if the goal were in the same row or column, as well as if it were placed directly next to a wall, this could be argued to advantage an action-repeat exploration method.
Instead, the goal location was chosen to be harder for (cid:15)z -greedy to ﬁnd (offset from wall, far from and not in same row/column as start state).
For experiments, we used Q-learning with a tabular function approximator, learning rate α = 0 .
1 , (cid:15) = 0 .
1 , and maximum episode length 1000 .
Experiment results are averages over 30 random seeds.
Figure 1 shows average ﬁrst-passage times on a similar gridworld, but with a ﬁxed greedy policy which takes the “down” action deterministically.
MountainCar (Sutton and Barto, 2018)
This environment models an under-powered car stuck in the valley between two hills.
The agent must build momentum in order to reach the top of one hill and obtain the goal reward.
In this version of the domain all rewards are zero except for the goal, which yields reward of 1 . 0 .
There are two continuous state variables, corresponding to the agent location, x , and velocity, ˙ x .
The dense-reward version of this environment can be solved reliably in less than a dozen episodes using linear function approximation on top of a low-order Fourier basis (Konidaris, Osentoski, and
Thomas, 2011).
In our experiments using the sparse-reward variant of the environment, we used SARSA( λ ) with a linear function approximation on top of an order
Fourier basis.
We used learning rate α = 0 .
005 , (cid:15) = 0 .
05 , γ = 0 . 99 , and λ = 0 .
The maximum episode length was set to 5000 .
Experiment results are averages over 30 random seeds.
A near-optimal policy, given this discount and (cid:15) , but without confounds due to function-approximation, should reach approximately 0 .
29 episodic discounted return.
CartPole (Barto, Sutton, and Anderson, 1983)
We use the “swingup sparse” variant as implemented in Tassa et al.
In this sparse reward version of the environment, the agent receives zero reward unless | x | < 0 .
25 and cos( θ ) > 0 .
995 , for the cart location x and pole angle θ .
All episodes run for 1000 steps, and observations are 5 -dimensional continuous observation.
For experiments, we used SARSA( λ ) with a linear function approximation on top of an order
Fourier basis.
We used learning rate α = 0 .
0005 , (cid:15) = 0 .
01 , γ = 0 .
99 , and λ = 0 .
The maximum episode length was 1000 .
Weights were initialized randomly from a mean-zero normal distribution with variance 0 .
Experiment results are averages over 30 random seeds.
Atari-57 (Bellemare et al., 2013), is a benchmark suite of 57 Atari 2600 games in the Arcade
Learning Environment (ALE).
Observations are 210 × 160 color images (following Mnih et al.
, in many agents these are down-scaled to 84 × 84 and converted to grayscale).
Throughout this work we use the original ALE version of Atari 2600 games, which does not include subsequently added games (beyond the 57) or features such as “sticky actions”.
Many existing results on Atari-57 report performance of the best agent throughout training, or simply the maximum evaluation performance attained during training.
We do not report this metric in the main text because it does not reﬂect the true learning progress of agents and tends to reﬂect an over estimate.
However, for comparison purposes, “best” performance is included later in the Appendix.
In the next section, alongside other agent details, we will give hyper-parameters used in the Atariexperiments.
An example frame from the game P RIVATE E YE is shown in Figure 5e.
C A GENT AND A LGORITHM D ETAILS
Except for ablation experiments on the duration distribution, all (cid:15)z -greedy experiments use a duration distribution z ( n ) ∝ n − µ with µ = 2 . 0 .
These durations were capped at n ≤ 10000 for all experiments except for the Rainbow-based agents which were limited to n ≤ 100 , but in this case no other values were attempted.
P SEUDO - CODE
Algorithm
1: function EZ G REEDY ( Q , (cid:15) , z ) 2: n ←3:
ω ← −4: while True do 5:
Observe state x 6: if n == 0 then 7: if random () ≤ (cid:15) then 8:
Sample duration: n ∼ z 9:
Sample action: ω ∼ U
( A ) 10:
Assign action: a ← ω 11: else 12:
Greedy action: a ← arg max a Q ( x, a ) 13: else 14:
Assign action: a ← ω 15: n ← n −16:
Take action a
N ETWORK A RCHITECTURE .
Rainbow-based agents use an identical network architecture as the original Rainbow agent (Hessel et al., 2018).
In particular, these include the use of NoisyNets (Fortunato et al., 2017), with the exception of Rainbow-CTS, which uses a simple dueling value network like the “no noisy-nets” ablation in Hessel et al.
A preliminary experiment showed this setting with Rainbow-CTS performed slightly better than when NoisyNets were included.
R2D2-based agents use a slightly enlarged variant of the network used in the original R2D2 (Kapturowski et al., 2019), namely a 4-layer convolutional neural network with layers of 32, 64,and 128 feature planes, with kernel sizes of 7, 5, 5 and 3, and strides of 4, 2, 2 and 1, respectively.
These are followed by a fully connected layer with 512 units, an LSTM with another 512 hidden units, which ﬁnally feeds a dueling architecture of size 512 (Wang et al., 2015).
Unlike the original
R2D2, Atari frames are passed to this network without frame-stacking, and at their original resolution of 210 × 160 and in full RGB.
Like the original R2D2, the LSTM receives the reward and one-hot action vector from the previous time step as inputs.
H YPER - PARAMETERS AND I MPLEMENTATION N OTES
Unless stated otherwise, hyper-parameters for our Rainbow-based agents follow the original implementation in Hessel et al.
(2018), see Table 2.
An exception is the Rainbow-CTS agent, which uses a regular dueling value network instead of the NoisyNets variant, and also makes use of an (cid:15) -greedy policy (whereas the baseline Rainbow relies on its NoisyNets value head for exploration).
The (cid:15) parameter follows a linear decay schedule 1 . 0 to 0 .
over the course of the ﬁrst 4 M frames, remaining constant after that.
Evaluation happens with an even lower value of (cid:15) = 0 .
The same (cid:15) -schedule is used in Rainbow+ (cid:15) -greedy
and Rainbow+ (cid:15)z -greedy, on top of Rainbow’s regular
NoisyNets-based policy.
The CTS-based intrinsic reward implementation follows Bellemare et al.
(2016), with the scale of intrinsic rewards set to a lower value of 0 . 0005 .
This agent was informally tuned for better performance on hard-exploration games: Instead of the “mixed Monte-Carlo return” update rule from
Bellemare et al.
(2016), Rainbow-CTS uses an n -step Q-learning rule with n = 5 (while the baseline
Rainbow uses n = 3 ), and differently from the baseline does not use a target network.
All of our R2D2-based agents are based on a slightly tuned variant of the published R2D2 agent (Kapturowski et al., 2019) with hyper-parameters unchanged, unless stated otherwise - see Table 3.
Instead of an n -step Q-learning update rule, our R2D2 uses expected SARSA( λ ) with λ = 0 .(Van
Seijen et al., 2009).
It also uses a somewhat shorter target network update period of 400 update steps and the higher learning rate of 2 × 10 − 4 .
For faster experimental turnaround, we also use a slightly larger number of actors ( 320 instead of 256 ).
This tuning was performed on the vanilla R2D in order to match published results.
The RND agent is a modiﬁcation of our baseline R2D2 with the addition of the intrinsic reward generated by the error signal of the RND network from Burda et al.
The additional networks (“predictor” and “target” in the terminology of Burda et al.
(2018)) are small convolutional neural networks of the same sizing as the one used in Mnih et al.
(2015), followed by a single linear layer with output size 128 .
The predictor is trained on the same replay batches as the main agent network, using the Adam optimizer with learning rate 0 . 0005 .
The intrinsic reward derived from its loss is normalized by dividing by its variance, utilizing running estimates of its empirical mean and variance.
Note, the RND agent includes the use of (cid:15) -greedy exploration.
The Bootstrapped R2D2 agent closely follows the details of Osband et al.
The R2D2 network is extended to have k = 8 action-value function heads which share a common convolutional and
LSTM network, but with distinct fully-connected layers on top (each with the same dimensions as in R2D2).
During training, each actor samples a head uniformly at random
[Table]
and follows that action-value function’s (cid:15) -greedy policy for an entire episode.
Each step, a mask is sampled
[Table]
and added to replay, with probability p = 0 .
5 indicating which heads will be trained on that step of experience.
During evaluation, we compute the average of each head’s (cid:15) -greedy policy to form an ensemble policy that is followed.
[Table]
+ (cid:15) / (cid:15)z -greedy, Rainbow+CTS (cid:15) (training
[Table]
[Table] [Table]
D E XPERIMENT D ETAILS
First-visit visualizations These results (e
[Table]
g. see Figure 1) are intended to illustrate the differences in state-visitation patterns between (cid:15) -greedy and (cid:15)z -greedy.
These are generated with some ﬁxed (cid:15) , often (cid:15) = 1 . 0 for pure-exploration independent of the greedy policy, and are computed using
Monte-Carlo rollouts with each state receiving an integer indicating the ﬁrst step at which that state was visited on a given trial.
States that are never seen in a trial receive the maximum step count, and we then average these over many trials.
For continuous-state problems we discretize the state-space and count any state within a small region for the purposes of visitation.
We give these precise values in Table
[Table]
Atari experiments The experimental setup for the Rainbow-based and R2D2-based agents each match those used in their respective baseline works.
In particular, Rainbow-based agents perform a mini-batch gradient update every 4 steps and every 1 M environment frames learning is frozen and the agent is evaluated for 500 K environment frames.
In the R2D2-based agents, acting, learning, and evaluating all occur simultaneously and in parallel, as in the baseline R2D2.
In the Atari-57 experiments, all results for Rainbow agents are averaged over 5 random seeds, while results for R2D2-based agents are averages over 3 random seeds.
Atari-57 is most often used with a built-in number ( 4 ) of action-repeats for every primitive action taken (Mnih et al., 2015).
We did not change this environment parameter, which means that an exploratory action-repeat policy of length n will, in the actual game, produce 4 × n low-level actions.
Similarly, DQN-based agents typically use frame-stacking, while agents such as R2D2 which use an
RNN do not.
The robustness of our results across these different algorithms suggests that (cid:15)z -greedy is not greatly impacted by the presence or absence of frame-stacking.
Atari metrics
The human-normalized score is deﬁned as
[Equation]
where agent, random, and human are the per-game scores for the agent, a random policy, and a human player respectively.
The human-gap is deﬁned as the average, over games, performance difference
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
/98jppXVb9WrX2UKvUb/I6iugUnaEL5KMrVEf3qIGaiKIJekav6M2ZOS/Ou/OxHC04eeYE/YHz+QPuhJUG</latexit> between human-
[Equation]
Computational Resources Small-scale experiments were written in Python and run on commodity hardware using a CPU.
Rainbow-based agents were implemented in Python using JAX, with each conﬁguration (game, algorithm, hyper-parameter setting) run on a single V100 GPU.
Such experiments generally required less than a week of wall-clock time.
R2D2-based agents were implemented in Python using Tensorﬂow, with each conﬁguration run on a single V100 GPU and a number of actors (speciﬁed above) each run on a single CPU.
These agents were trained with a distributed training regime, described in the R2D2 paper (Kapturowski et al., 2019), and required approximately 3 days to complete.
E E XPERIMENTAL R ESULTS : S TOCHASTICITY
In this section we present a series of experiments focused on the effects of stochasticity and nondeterminism in the environment on the performance of (cid:15)z -Greedy
. S MALL -S CALE E NVIRONMENTS
We consider stochastic versions of two of our small-scale environments: GridWorld, and MountainCar.
In the Gridworld domain the “noise scale” is the probability of the action transitioning to a random neighboring state.
In MountainCar, where actions apply a force in − 1 , 0 , 1 to the car, we add mean zero Gaussian noise to this force with variance given by the speciﬁed noise scale.
Finally, note that these forces are clipped to be within the original range of [ − 1 , 1] .
Figure 7a shows the discounted return, averaged over the 100 episode training runs, as a function of the noise scale.
Figure 7b gives example learning curves for the agents ( (cid:15) -Greedy and (cid:15)z -Greedy) for four levels of noise.
We ﬁrst note that for near-deterministic settings we replicate the original ﬁndings in the main text, but that the performance of the (cid:15) -Greedy agent actually improves for small amounts of transition noise, while both agents degrade performance as this noise becomes larger.
We interpret this to suggest that the (cid:15) -Greedy agent is initially beneﬁting from the increased exploration, whereas (cid:15)z -Greedy was already exploring more and begins to degrade slightly sooner.
Figure 8 similarly shows the two agent’s performance on MountainCar as we increase the scale of the transition noise.
Here we see both agents generally suffer reduced performance as the level of noise increases and is maximal for both in the deterministic case, unlike in Gridworld.
A TARI -57 WITH S TICKY -A CTIONS
The Arcade Learning Environment (Bellemare et al., 2013, ALE), supports a form of non-determinism called sticky actions where with some probability ζ (typically set to 0 .
25 ) the agent’s action is ignored and instead the previously executed action is repeated (Machado et al., 2018a).
This is not exactly
[Figure] [Figure]
equivalent to simpler transition noise, such as we used in the small-scale experiments above, because there is now a non-Markovian dependence on the previous action executed in the environment.
Two additional details are important to keep in mind.
First, the agent does not observe the action that was executed, and instead only observes the action that it intended to take.
Second, the sticky-action effect applies at every step of the low-level ALE interaction.
That is, it is standard practice to use an action-repeat (usually 4 ) in Atari, such that each action selected by the agent is repeated some number of times in the low-level interaction.
Thus, sticky actions apply at every step of this low-level interaction.
Because of the non-Markovian effects and the relatively fast decay in probability of action-repeats, sticky actions do not provide similar exploration beneﬁts as seen in (cid:15)z -Greedy.
Unlike in exploration using action-repeats the agent is unable to learn about the actions that were actually executed, making the underlying learning problem more challenging.
In some environments, where precise control is needed for high performance, sticky actions tend to signiﬁcantly degrade performance of agents.
We note that sticky actions are a modiﬁcation usually applied to the problem, not the agent, and usually make the problem harder.
The reason for this is that sticky actions increase the level of stochasticity of the environment, but do so with a non-Markovian dependency on the previously executed action.
Given the superﬁcial similarity with action-repeats, one might ask whether using sticky actions might beneﬁt exploration on hard exploration games similar to (cid:15)z -Greedy.
In order to answer this question we compare our Rainbow-based (cid:15) -Greedy and (cid:15)z -Greedy agents on the Ataribenchmark with sticky actions enabled ( ζ = 0 . 25 ).
Figure 9 gives the median, mean, and human-gap values for the human-normalized scores.
We observe that much of the gap in performance in mean and median have disappeared, and that even for human-gap the performance beneﬁts of (cid:15)z -Greedy over (cid:15) -Greedy have been partially reduced.
Nonetheless, we continue to see signiﬁcant improvements in performance on the same set of harder exploration games as in the non-sticky-action results (see Figure 10). Finally, we give the numeric values for the ﬁnal performance of these agents in Table 5. [Table] [Figure]
[Figure]
R e t u r n
F E XPERIMENTAL R ESULTS : L IMITATIONS
To further study the limitations of (cid:15)z -Greedy, and motivate work on learned options for exploration, we consider the effect on performance of adding “obstacles” and “traps” in the Gridworld domain.
For our purposes, an obstacle is an interior wall in the gridworld, such that if the agent attempts to occupy the location of the obstacle the result is a no-op action with no state transition.
On the other hand, a “trap” results in the immediate end of an episode, with zero reward, if the agent attempts to occupy the same location as the trap.
Figure 11 shows a set of example gridworlds with obstacles and traps generated at varying target densities.
We generate the environments by ﬁlling a 20 ×gridworld with either obstacles or traps (not both), where each state has some probability of being so ﬁlled (given by the target density).
We then identify the largest connected component of open cells and select the goal and start states randomly without replacement from the states in this component.
This ensures that all gridworlds can be solved.
Note that increasing the density of obstacles and traps has two opposite effects: on one hand it reduces the overall state space, thus making the problem easier, on the other hand it also makes exploration more difﬁcult, since obstacles result in a sparser transition graph and traps result in many absorbing states.
Based on our observations, the latter effect tends to be stronger than the former.
Our experiments focus on studying the performance of (cid:15) -Greedy and (cid:15)z -Greedy as we scale the density of the obstacles or traps.
Figure 12 & 13 show our results in these two sets of experiments.
In both cases we observe that the gap in performance between (cid:15)z -Greedy and (cid:15) -Greedy decreases with the density of the problematic states.
Interestingly, we see that (cid:15) -Greedy is not impacted as seriously.
We believe this is partly to be expected because (cid:15) -Greedy is exploring more locally and more densely around the start state, making navigating around obstacles somewhat easier.
Note that we have increased the number of episodes from 100 (used in the main text) to 1000 in order to increase the likelihood of both agents solving a given problem.
Additionally, unlike in the main text, every trial of these experiments is performed on a randomly generated gridworld with randomly selected start and goal locations; although we do ensure that each agent is trained on the same environment the environment itself is different for each seed.
This is reﬂected in the larger variances in performance indicated by the shaded regions in the ﬁgures.
G F URTHER E XPERIMENTAL R ESULTS
In this section, we include several additional experimental results that do not ﬁt into the main text but may be helpful to the reader.
In the conclusions we highlight a limitation of (cid:15)z -greedy which [Figure] occurs when the effects of actions differ signiﬁcantly between states.
In Figure 14 we present results for such an adversarial setting in the DeepSea environment, where the action effects are randomly permuted for every state.
We observe, as expected, that in this setting (cid:15)z -greedy no longer provides more efﬁcient exploration than (cid:15) -greedy.
In Figure 15 we compare with RMax (Brafman and Tennenholtz, 2002) on the Gridworld domain.
We consider two values for the threshold for a state-action being marked as known : N = 1 , effectively encoding an assumption that the domain is deterministic, and N = 10 which is a more generally reasonable value.
We observe that, if tuned aggressively, RMax can signiﬁcantly outperform (cid:15)zGreedy, as should be expected.
However, we note that unlike RMax (cid:15)z -Greedy does not assume access to a tabular representation and can scale to large-scale problems.
Next, in Figures 16 & 17 we report the per-game percentage of relative improvement, using humannormalized scores, over an (cid:15) -greedy baseline for (cid:15)z -greedy in both the Rainbow and R2D2 agents.
In both cases we give the corresponding results for improvement of CTS, for Rainbow, and RND, for R2D2, over the same baselines.
Additionally, we give these results for both the ﬁnal agent performance and the performance averaged over training.
The percent relative improvement of a score over a baseline is computed as 100 × score − baseline baseline .
[Figure]
[Figure]
Note that we limit the maximum vertical axis in such a way that improvement in some games is cut off.
This is because for some games these relative improvements are so large that it becomes difﬁcult to see the values for other games on the same scale.
In the main text we give summary learning curves on Atari-57 for Rainbow- and R2D2-based agents in terms of median human-normalized score and human-gap.
In Figure 18 we show these as well as the mean human-normalized score learning curves.
In Figures 19 & 20 we give full, per-game results for Rainbow- and R2D2-based agents respectively.
These results offer additional context on those we reported in the main text, demonstrating more concretely the nature of the performance trade-offs being made by each algorithm.
Finally, in Table 1 we give mean and median human-normalized scores and the human-gap on
Atari-57 for the ﬁnal trained agents.
However, this is a slightly different evaluation method than is often used (Mnih et al., 2015; Hessel et al., 2018), in which only the best performance for each game, over training, is considered.
For purposes of comparison we include these results in Table 6.
Algorithm (@ 30 B)
[Table]
R2D
[Table]
[Table]
[Table]
[Table]
[Table]
R2D2+RND
[Table]
[Table]
[Table]
[Table]
[Table]
R2D2+Bootstrap
[Table]
[Table]
[Table]
[Table]
[Table]
R2D2+ (cid:15)z -greedy 16.
[Table]
Algorithm (@ 200 M)
Rainbow
[Table]
[Table]
[Table]
[Table]
[Table]
Rainbow+ (cid:15) -Greedy
[Table]
[Table]
[Table]
[Table]
[Table]
Rainbow+CTS
[Table]
Rainbow+ (cid:15)z -Greedy
[Table]
[Table]
[Figure]
D i
[Figure] c o un t e d r e t u r n [Figure] Greedy -Greedy
RMax(1) RMax(10) [Figure] [Figure]
[Figure] H u m a n - n o r m a l i z e d S c o r e
[Figure]
H u m a n - n o r m a l i z e d S c o r e
[Figure] [Figure] [Figure] [Table]