R EGRET M INIMIZATION FOR P ARTIALLY O BSERVABLE D EEP R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep reinforcement learning algorithms that estimate state and state-action value functions have been shown to be effective in a variety of challenging domains, including learning control strategies from raw image pixels.
However, algorithms that estimate state and state-action value functions typically assume a fully observed state and must compensate for partial or non-Markovian observations by using ﬁnite-length frame-history observations or recurrent networks.
In this work, we propose a new deep reinforcement learning algorithm based on counterfactual regret minimization that iteratively updates an approximation to a cumulative clipped advantage function and is robust to partially observed state.
We demonstrate that on several partially observed reinforcement learning tasks, this new class of algorithms can substantially outperform strong baseline methods: on Pong with single-frame observations, and on the challenging Doom (ViZDoom) and
Minecraft (Malm ¨ o) ﬁrst-person navigation benchmarks.
I NTRODUCTION
Many reinforcement learning problems of practical interest have the property of partial observability, where observations of state are generally non-Markovian.
Despite the importance of partial observation in the real world, value function-based methods such as Q-learning (Mnih et al., 2013; 2015) generally assume a Markovian observation space.
On the other hand, Monte Carlo policy gradient methods do not assume Markovian observations, but many practical policy gradient methods such as A3C (Mnih et al., 2016) introduce the Markov assumption when using a critic or state-dependent baseline in order to improve sample efﬁciency.
For example, consider deep reinforcement learning methods that learn a state or state-action value function.
One common workaround for the problem of partial observation is to learn value functions on the space of ﬁnite-length frame-history observations, under the assumption that frame-histories of sufﬁcient length will give the environment the approximate appearance of full observability.
When learning to play Atari 2600 games from images, deep Q-learning algorithms (Mnih et al., 2013; 2015) concatenate the last 4 observed frames of the video screen buffer as input to a state-action value convolutional network.
Not all non-Markovian tasks are amenable to ﬁnite-length framehistories; recurrent value functions can be used to incorporate longer and potentially inﬁnite histories (Hausknecht & Stone, 2017; Foerster et al., 2016), but at the cost of solving a substantially harder optimization problem.
Can we develop methods that learn a variant of the value function that is more robust to partial observability?
Our contribution is a new model-free deep reinforcement learning algorithm based on the principle of regret minimization which does not require access to a Markovian state.
Our method learns a policy by estimating a cumulative clipped advantage function , which is an approximation to a type of regret that is central to two partial information game-solving algorithms from which we draw our primary inspiration: counterfactual regret minimization (CFR) (Zinkevich et al., 2007) and CFR+ (Tammelin, 2014).
Hence we call our algorithm “advantage-based regret minimization” (ARM).
We evaluate our approach on three visual reinforcement learning domains: Pong with varying framehistory lengths (Bellemare et al., 2013), and the ﬁrst-person games Doom (Kempka et al., 2016) and Minecraft (Johnson et al., 2016).
Doom and Minecraft exhibit a ﬁrst-person viewpoint in a 3dimensional environment and should appear non-Markovian even with frame-history observations.
We ﬁnd that our method offers substantial improvement over prior methods in these partially observable environments: on both Doom and Minecraft, our method can learn well-performing policies within about 1 million simulator steps using only visual input frame-history observations.
R ELATED W ORK
Deep reinforcement learning algorithms have been demonstrated to achieve excellent results on a range of complex tasks, including playing games (Mnih et al., 2015;
Oh et al., 2016) and continuous control (Schulman et al., 2015; Lillicrap et al., 2016; Levine et al., 2016).
Prior deep reinforcement learning algorithms either learn state or state-action value functions (Mnih et al., 2013), learn policies using policy gradients (Schulman et al., 2015), or perform a combination of the two using actor-critic architectures (Mnih et al., 2016).
Policy gradient methods typically do not need to assume a Markovian state, but tend to suffer from poor sample complexity, due to their inability to use off-policy data.
Methods based on learning Q-functions can use replay buffers to include off-policy data, accelerating learning (Lillicrap et al., 2016).
However, learning Q-functions with Bellman error minimization typically requires a Markovian state space.
When learning from observations such as images, the inputs might not be Markovian.
Prior methods have proposed to mitigate this issue by using recurrent critics and Q-functions (Hausknecht & Stone, 2017;
Oh et al., 2016; Mnih et al., 2016; Heess et al., 2015), and learning Q-functions that depend on entire histories of observations.
Heuristics such as concatenation of short observation sequences have also been used (Mnih et al., 2015).
However, all of these changes increase the size of the input space, increasing variance, and make the optimization problem more complex.
Our method instead learns cumulative advantage functions that depend only on the current state, but can still handle non-Markovian problems.
The form of our advantage function update resembles positive temporal difference methods (Peng et al., 2016; van Hasselt & Wiering, 2007).
Additionally, our update rule for a modiﬁed cumulative Q-function resembles the average Q-function (Anschel et al., 2017) used for variance reduction in Q-learning.
In both cases, the theoretical foundations of our method are based on cumulative regret minimization, and the motivation is substantively different.
Previous work by Ross et al.
(2011); Ross & Bagnell (2014) has connected regret minimization to reinforcement learning, imitation learning, and structured prediction, although not with counterfactual regret minimization.
Regression regret matching (Waugh et al., 2015) is based on a closely related idea, which is to directly approximate the regret with a linear regression model, however the use of a linear model is limited in representation compared to deep function approximation.
A DVANTAGE - BASED R EGRET M INIMIZATION
In this section, we provide background on CFR and CFR+, describe ARM in detail, and give some intuition for why ARM works.
3. C OUNTERFACTUAL REGRET MINIMIZATION (CFR)
In this section we review the algorithm of counterfactual regret minimization (Zinkevich et al., 2007).
We closely follow the version of CFR as described in the Supplementary Material of Bowling et al.
(2015), except that we try to use the notation of reinforcement learning where appropriate.
Consider the setting of an extensive game.
There are N players numbered i = 1 , . . .
, N .
An additional player may be considered a “chance” player to simulate random events.
At each time step of the game, one player chooses an action a ∈ A i .
Deﬁne the following concepts and notation:
• Sequences: A sequence speciﬁcally refers to a sequence of actions starting from an initial game state.
(It is assumed that a sequence of actions, including actions of the “chance” player, is sufﬁcient for deﬁning state within the extensive game.)
Let H be the space of all sequences, and let Z be the space of terminal sequences.
• Information sets: Let I be the space of information sets; that is, for each I ∈
I , I is a set of sequences h ∈
I which are indistinguishable
to the current player.
Information sets are a represention of partial observability.
• Strategies: Let π i ( a | I ) be the strategy of the i -th player, where π i ( a | I ) is a probability distribution over action a conditioned on information set I .
Let π =
( π 1 , . . .
, π N ) denote the strategy proﬁle for all players, and let π − i =
( π 1 , . . .
π i − 1 , π i +1 , . .
. , π N ) denote the strategy proﬁle for all players except the i -th player.
• Sequence probabilities: Let ρ π ( h ) be the probability of reaching the sequence h when all players follow π .
Additionally, let ρ π ( h, h (cid:48) ) be the probability of reaching s (cid:48) conditioned on h having already been reached.
Similarly, deﬁne ρ πi and ρ π − i to contain the contributions of respectively only the i -th player or of all players except the i -th.
• Values: Let u i ( z ) be the value of a terminal sequence z to the i -th player.
Let the expected value of a strategy proﬁle π to the i -th player be J i
( π ) =
(cid:80)
z ∈
Z ρ π ( z ) u i ( z ) .
Deﬁne the counterfactual value Q CF π,i of all players following strategy π , except the i -th player plays to reach information set I and to then take action a :
[Equation]
The notation h (cid:64) h (cid:48) denotes that h is a preﬁx of h (cid:48) , while π | I → a denotes that action a is to be performed when I is observed.
The counterfactual value Q CF π,i ( I, a ) is a calculation that assumes the i -th player reaches any h ∈ I , and upon reaching
any h ∈
I it always chooses a .
Consider a learning scenario where at the t -th iteration the players follow a strategy proﬁle π t .
The i -th player’s regret after T iterations is deﬁned in terms of the i -th player’s optimal strategy π ∗ i :
[Equation]
The average regret is the average over learning iterations: (1 /T )
R Ti .
Now deﬁne the counterfactual regret of the i -th player for taking action a at information set I :
[Equation]
The counterfactual regret (Equation (3)) can be shown to majorize the regret (Equation (2))
(Theorem 3, Zinkevich et al.
CFR can then be described as a learning algorithm where at iteration t + 1 , the strategy is updated using regret matching (Hart & Mas-Colell, 2000) applied to the counterfactual regret of the previous iteration t :
[Equation]
If all players follow the CFR regret matching strategy (Equation (5)), then at the T -th iteration the players’ average regrets are bounded by O ( T − 1 / 2 ) (Theorem 4, Zinkevich et al.
3. CFR+
CFR+ (Tammelin, 2014) consists of a modiﬁcation to CFR, in which instead of calculating the full counterfactual regret as in (4), instead the counterfactual regret is recursively positively clipped to yield the clipped counterfactual regret:
[Equation]
Comparing Equation (4) with Equation (6), one can see that the only difference in CFR is that the previous iteration’s counterfactual regret is positively clipped in the recursion.
The one-line change of CFR+ turns out to yield a large practical improvement in the performance of the algorithm (Bowling et al., 2015), and there is also an associated regret bound for CFR+ that is as strong as the bound for CFR (Tammelin et al., 2015).
3. F ROM CFR AND CFR+ TO ARM
CFR and CFR+ are formulated for imperfect information extensive-form games, so they are naturally generalized to partially observed stochastic games since a stochastic game can always be represented in extensive form.
A 1-player partially observed stochastic game is simply a POMDP with observation space
O (Littman, 1994).
By mapping information sets I ∈ I to observations o ∈ O , we may rewrite the counterfactual value as a kind of stationary observation-action value
Q CF π,i ( I, a ) ≡ Q (stat) π | o
(cid:55)→ a ( o, a ) that assumes the agent follows the policy π except on observing o , after which the action a is always performed (Bellemare et al., 2016).
We posit that the approximation
Q (stat) π | o (cid:55)→ a ( o, a ) ≈ Q π ( o, a ) , where Q π is the usual action value function, is valid when observations are rarely seen more than once in a trajectory.
By approximating Q (stat) π | o (cid:55)→ a ( o, a )
≈ Q π ( o, a ) , we get a recurrence in terms of more familiar value functions:
[Equation]
where ¯ A + t ( o, a ) is the cumulative clipped advantage function, and A π t ( o, a ) is the ordinary advantage function evaluated at policy π t .
Advantage-based regret minimization (ARM) is the resulting reinforcement learning algorithm that updates the policy to regret match on the cumulative clipped advantage function:
[Equation]
if (cid:80) a (cid:48)
[Equation]
max(0 , ¯ A + t ( o k , a (cid:48) ))
>1 |A| otherwise.
Equation (9) suggests an outline of a batch-mode deep reinforcement learning algorithm.
At the t -th sampling iteration, a batch of data is collected by sampling trajectories using the current policy π t , followed by two processing steps: (a) ﬁt ¯ A + t , then (b) set the next iteration’s policy π t +1 using
Equation (10).
3. I MPLEMENTATION OF ARM
To implement Equation (9) with deep function approximation, we deﬁne two value function approximations, V π t ( o k ; θ t ) and ¯ Q + t
( o k , a k ; ω t ) , as well as a target value function V (cid:48)
( o k ; ϕ ) , where θ t , ω t , and ϕ are the learnable parameters.
The cumulative clipped advantage function is represented as ¯
A + t ( o k , a k ) =
¯ Q + t
( o k , a k ; ω t ) − V π t ( o k ; θ t ) .
Within each sampling iteration, the value functions are ﬁtted using stochastic gradient descent by sampling minibatches and performing gradient steps.
The state-value function V π t ( o k ; θ t ) is ﬁt to minimize an n -step temporal difference loss with a moving target V (cid:48)
( o k + n ; ϕ ) , essentially using the estimator of the deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016).
In the same minibatch, ¯ Q + t
o k , a k ; θ t ) is ﬁt to a similar loss, but with an additional target reward bonus that incorporates the previous iteration’s cumulative clipped advantage, max(0 , ¯ A + t − 1 ( o k , a k )) .
The regression targets v ( o k ; ϕ ) and ¯ q
+ ( o k , a k ; ϕ ) are deﬁned in terms of the n -step returns
[Equation]
Altogether, each minibatch step of the optimization subproblem consists of the following three parameter updates in terms of the regression targets v ( o k ; ϕ ) and ¯ q + (
[Equation]
k , a k ; ϕ ) :
[Equation]
The overall advantage-based regret minimization algorithm is summarized in Algorithm 1.
Algorithm initialize π 0 ← uniform, θ − 1 , ω − 1 ← arbitrary for t in 0 , . . .
do collect batch of trajectory data D t ∼ π t initialize θ t ← θ t − 1 , ω t ← ω t − 1 , ϕ ← θ t − 1 for (cid:96) in 0 , . .
do sample transitions ( o k , a k , r k , . . .
, o k + n − 1 , a k + n − 1 , r k + n − 1 , o k + n ) ∼ D t calculate n -step returns g nk = (cid:80) k
+ n − 1 k (cid:48)
=
k γ k (cid:48) − k r k (cid:48) set δ ← I [ o k + n is terminal ] if t = 0 then set φ k ←else set φ k ← max(0 , ¯ Q + t − 1 ( o k , a k ; ω t − 1 ) − V π t − 1 ( o k ; θ t − 1 )) end if set v ( o k ) ← g nk + γ n (1 − δ ) V (cid:48)
( o k + n ; ϕ ) set ¯ q
+ ( o k , a k ) ← φ k + r k + γg n − 1 k +1 + γ n (1 − δ ) V (cid:48)
( o k + n ; ϕ ) update θ t with step size α and targets v ( o k )
(Equation (14)) update ω t with step size α and targets ¯ q
+ ( o k , a k ) (Equation (15)) update ϕ with moving average step size τ (Equation (16)) end for set π t +1 ( a | o )
∝ max(0 , ¯ Q + t
( o, a ; ω t ) − V π t ( o ; θ t )) end for
3. W HAT IS THE INTUITION BEHIND ARM?
One way to understand CFR is that the algorithm achieves its bound of O
( T − 1 / 2 ) on the average regret while only accessing partially observed information sets.
ARM’s effectiveness on partially observable domains can be seen as inherited from CFR.
The mechanics of the ARM updates are similar to on-policy value function estimation, but ARM learns a modiﬁed on-policy Q-function from transitions with the added reward bonus max(0 , ¯ A + t − 1 ( o k , a k ))
(Equation (13)).
This reward bonus is a kind of optimistic “inertia” in favor of some observation-action pairs.
A second view of ARM comes from comparing the convergence rates of reinforcement learning algorithms.
CFR and CFR+ have average regret bounds of O ( T − 1 / 2 ) that are ultimately derived from regret matching (Hart & Mas-Colell, 2000).
Szepesv´ari proved that the convergence rate of
Q-learning in the L ∞ -norm, assuming a ﬁxed exploration strategy, depends on a condition number C , which is the ratio of the minimum to maximum state-action occupation frequencies (Szepesv ´ ari, 1998), and which describes how “balanced” the exploration strategy is.
If partial observability leads to imbalanced exploration due to confounding of states from perceptual aliasing (McCallum, 1997), then Q-learning should be negatively affected.
To our knowledge, there is no known regret bound for Q-learning.
The policy gradient method with a suitable baseline has a learning rate η -dependent regret bound derived from the stochastic gradient method; by setting the learning rate η ∝ T − 1 / 2 , and taking into account the second moment of the gradient estimates G 2 , policy gradient has an average regret bound of O
( T − 1 / 2 ) (Dick, 2015).
Comparing these algorithms, we see that policy gradient methods can be as efﬁcient as CFR and CFR+ but only for a tuned learning rate schedule.
Additionally, Q-learning has bounds on the Q -function L ∞ -norm which are strictly worse than the other two algorithms’ average regret bounds.
See Table 1 for a summary.
We note that there remains a gap between ARM as implemented and the theory of CFR: the use of (a) function approximation and sampling over tabular enumeration; (b) the “ordinary” Q-function instead of the “stationary” Q-function; and (c) n -step bootstrapped values instead of full returns for value function estimation.
Waugh et al.
(2015) address CFR with function approximation via a noisy version of a generalized Blackwell’s condition (Cesa-Bianchi & Lugosi, 2003).
Even the original implementation of CFR used sampling in place of enumeration (Zinkevich et al., 2007).
We refer the reader to Bellemare et al.
for a more in-depth discussion of the stationary Q-function,
Although only the full returns are guaranteed to be unbiased in non-Markovian settings, it is quite common for practical RL algorithms to trade off strict unbiasedness in favor of lower variance by using n -step returns or variations thereof (Schulman et al., 2016;
Gu et al., 2017).
[Table]
E XPERIMENTS
Because we hypothesize that ARM should perform well in partially observable reinforcement learning environments, we conduct our experiments on visual domains that naturally provide partial observations of state.
All of our evaluations use feedforward convnets with frame-history observations.
We are interested in comparing ARM with methods that assume Markovian observations, namely double deep Q-learning (van Hasselt et al., 2016), as well as methods that can handle non-Markovian observations, primarily TRPO (Schulman et al., 2015; 2016), and to a lesser extent A3C (Mnih et al., 2016)
whose critic assumes Markovian observations.
We are also interested in controlling for the advantage structure of ARM by comparing with other advantage-structured methods, which include dueling networks (Wang et al., 2016), as well as policy gradient methods that estimate an empirical advantage using a baseline state-value function or critic (e.g. TRPO, A3C).
4. L EARNING TO PLAY P ONG WITH A SINGLE FRAME
Atari games consist of a small set of moving sprites with ﬁxed shapes and palettes, and the motion of sprites can be highly deterministic, so that with only 4 recently observed frames as input one can predict hundreds of frames into the future on some games using only a feedforward model (Oh et al., 2015).
To increase the difﬁculty of learning to play Atari games, one may artiﬁcially limit the amount of frame history fed as input to the networks (Hausknecht & Stone, 2017).
As a proof of concept of ARM, we trained agents to play Pong via the Arcade Learning Environment (Bellemare et al., 2013) when the frame-history length is varied between 4 (the default) and 1.
We found that the performance of double deep Q-learning degraded noticeably when the frame-history length was reduced from 4 to 1, whereas performance of ARM was not affected nearly as much.
Our results on
Pong are summarized in Figure 1.
[Figure]
4. L EARNING TO NAVIGATE IN V I ZD OOM
We evaluated ARM on the task of learning ﬁrst-person navigation in the ViZDoom (Kempka et al., 2016) domain based on the game of Doom.
Doom is a substantially more complex domain than
Atari, featuring an egocentric viewpoint, 3D perspective, and complex visuals.
We expect that
Doom exhibits a substantial degree of partial observability and therefore serves as a more difﬁcult evaluation of reinforcement learning algorithms’ effectiveness on partially observable domains.
We performed our evaluation on two standard ViZDoom navigation benchmarks, “HealthGathering” and “MyWayHome.”
In “HealthGathering,” the agent is placed in a toxic room and continually loses life points, but can navigate toward healthkit objects to prolong its life; the goal is to survive for as long as possible.
In “MyWayHome,” the agent is randomly placed in a small maze and must ﬁnd a target object that has a ﬁxed visual appearance and is in a ﬁxed location in the maze; the goal is to reach the target object before time runs out.
Figure 2 (top row) shows example observations from the two ViZDoom scenarios.
Unlike previous evaluations which augmented the raw pixel frames with extra information about the game state, e.g. elapsed time ticks or remaining health (Kempka et al., 2016;
Dosovitskiy &
Koltun, 2017), in our evaluation we forced all networks to learn using only visual input.
Despite this restriction, ARM is still able to quickly learn policies with minimal tuning of hyperparameters and reach close to the maximum score in under 1 million steps.
On “HealthGathering,” we observed that
ARM very quickly learns a policy that can achieve close to the maximum episode return of 2100.
Double deep Q-learning learns a more consistent policy on “HealthGathering” compared to ARM and TRPO, but we believe this to be the result of evaluating double DQN’s (cid:15) -greedy policy with small (cid:15) compared to the truly stochastic policies learned by ARM and TRPO.
On “MyWayHome,” we observed that ARM generally learned a well-performing policy more quickly than other methods.
Additionally, we found that ARM is able to take advantage of an off-policy replay memory when learning on ViZDoom by storing the trajectories of previous sampling batches and applying an importance sampling correction to the n -step returns; please see Section 6.2 in the Appendix for details.
Our Doom results are in Figure 3.
[Figure]
4. L EARNING TO NAVIGATE IN M INECRAFT
We ﬁnally evaluated ARM on the task of learning ﬁrst-person navigation in the Malm¨o domain based on the game of Minecraft (Johnson et al., 2016).
Minecraft has similar visual complexity to Doom and should possess a comparable degree of partial observability, but Minecraft has the potential to be more difﬁcult than Doom due to the diversity of possible Minecraft environments that can be generated.
Our evaluation on Minecraft is adapted from the teacher-student curriculum learning protocol (Matiisen et al., 2017), which consists of 5 consecutive “levels” that successively increase the difﬁculty of completing the simple task of reaching a target block: the ﬁrst level (“L1”) consists of a single room; the intermediate levels (“L2”–“L4”) consist of a corridor with lava-bridge and wall-gap obstacles; and the ﬁnal level (“L5”) consists of a 2 × 2 arrangement of rooms randomly separated by lava-bridge or wall-gap obstacles.
Figure 2 (bottom row) shows example observations from the ﬁve Minecraft levels.
We performed our Minecraft experiments using ﬁxed curriculum learning schedules to evaluate the sample efﬁciency of different algorithms: the agent is initially placed in the ﬁrst level (“L1”), and the agent is advanced to the next level whenever a preselected number of simulator steps have elapsed, until the agent reaches the last level (“L5”).
We found that ARM and dueling double DQN both were able to learn on an aggressive “fast” schedule of only 62500 simulator steps between levels.
TRPO required a “slow” schedule of 93750 simulator steps between levels to reliably learn.
ARM was able to consistently learn a well performing policy on all of the levels, whereas double DQN learned more slowly on some of the intermediate levels.
ARM also more consistently reached a high score on the ﬁnal, most difﬁcult level (“L5”).
Our Minecraft results are shown in Figure 4.
[Figure]
D ISCUSSION
In this paper, we presented a novel deep reinforcement learning algorithm based on counterfactual regret minimization (CFR).
We call our method advantage-based regret minimization (ARM).
Similarly to prior methods that learn state or state-action value functions, our method learns a cumulative clipped advantage function of observation and action.
However, in contrast to these prior methods,
ARM is well suited to partially observed or non-Markovian environments, making it an appealing choice in a number of difﬁcult domains.
When compared to baseline methods, including deep
Q-learning and TRPO, on non-Markovian tasks such as the challenging ViZDoom and Malm¨o ﬁrstperson navigation benchmarks, ARM achieves substantially better results.
This illustrates the value of ARM for partially observable problems.
In future work, we plan to further explore applications of ARM to more complex tasks, including continuous action spaces.
A PPENDIX 6. E XPERIMENTAL DETAILS 6.1. P ONG (A RCADE L EARNING E NVIRONMENT )
We use the preprocessing and convolutional network model of (Mnih et al., 2013).
Speciﬁcally, we view every 4th emulator frame, convert the raw frames to grayscale, and perform downsampling to generate a single observed frame.
The input observation of the convnet is a concatenation of the most recent frames (either 4 frames or 1 frame).
The convnet consists of an 8 × 8 convolution with stride 4 and 16 ﬁlters followed by ReLU, a 4 × 4 convolution with stride 2 and 32 ﬁlters followed by ReLU, a linear map with 256 ﬁlters followed by ReLU, and a linear map with |A| ﬁlters where |A| is the action space cardinality ( |A| = 6 for Pong).
We used Adam with a constant learning rate of α = 10 − 4 , a minibatch size of 32 , and the moment decay rates set to their defaults β 1 = 0 .
9 and β 2 = 0 .
We ran ARM with the hyperparameters: sampling batch size of 12500 , 4000 / 3000 minibatches of
Adam for the ﬁrst/subsequent sampling iterations respectively, and target update step size τ = 0 .
Double DQN uses the tuned hyperparameters (van Hasselt et al., 2016).
Note that our choice of
ARM hyperparameters yields an equivalent number of minibatch gradient updates per sample as used by DQN and double DQN, i.e. 1 minibatch gradient update per 4 simulator steps.
6.1. D OOM (V I ZD OOM )
We used a convolutional network architecture similar to those of (Kempka et al., 2016) and (Dosovitskiy & Koltun, 2017).
The Doom screen was rendered at a resolution of 160 × 120 and downsized to 84 × 84 .
Only every 4th frame was rendered, and the input observation to the convnet is a concatenation of the
4 rendered frames for a of 12 input contains with each is size
8 with stride 4, second
4 with third by
For “HealthGathering” only, we scaled rewards by a factor of 0 .
We did not scale rewards for “MyWayHome.”
We used Adam with a constant learning rate rate of α = 10 − 5 and a minibatch size of 32 to train all networks (except
For “HealthGathering” we set β 1 = 0 .
95 , whereas for “MyWayHome” we set β
We set β
999 for both scenarios.
Double DQN and dueling double DQN: n = 5 step returns; update interval 30000 ; replay memory size 240000 ; exploration period 240000 ; with ﬁnal exploration rate (cid:15)
= 0 .
A3C: 16 workers; n = 20 steps for “HealthGathering” and n = 40 steps for “MyWayHome”; negentropy regularization β
= 0 .
01 ; and gradient norm clip
TRPO: sampling batch size 12500 ; KL-divergence step size α = 0 .
01 ; 10 conjugate gradient iterations; and Fisher information/Gauss-Newton damping coefﬁcient λ = 0 .
ARM: n = 5 step returns; sampling batch size
4000 Adam minibatches in the ﬁrst sampling iteration, 3000 Adam minibatches in all subsequent sampling iterations; target update step size τ = 0 .
Again, our choice of ARM hyperparameters yields an equivalent number of minibatch gradient updates per sample as used by DQN and double DQN, For “HealthGathering” only, because
ARM converges so quickly we annealed the Adam learning rate to α
= 2 . 5 × 10 − 6 afterelapsed simulator steps.
Off-policy ARM: n = 5 step returns; sampling batch size 1563 , replay cache sample size 25000 ; 400 Adam minibatches per sampling iteration; target update step size τ = 0 .
01 ; and importance sampling weight clip c =
6.1. M INECRAFT (M ALM ¨ O )
Our Minecraft tasks generally were the same as the ones used by Matiisen et al.
(2017), with a few differences.
Instead of using a continuous action space, we used a discrete action space with 4 move and turn actions.
To aid learning on the last level (“L5”), we removed the reward penalty upon episode timeout and we increased the timeout on “L5” from 45 seconds to 75 seconds.
We scaled rewards for all levels by 0 .
We use the same convolutional network architecture for Minecraft as we used for ViZDoom in
Section 4.2.
The Minecraft screen was rendered at a resolution of 320 × 240 and downsized to 84 × 84 .
Only every 5th frame was rendered, and the input observation of the convnet is a concatenation of the last 4 rendered RGB frames for a total of 12 input channels.
Double DQN and dueling double DQN: n = 5 step returns; update interval 12500 ; replay memory size 62500 ; exploration period 62500 ; with ﬁnal exploration rate (cid:15)
= 0 .
TRPO: sampling batch size 6250 ; KL-divergence step size α = 0 .
01 ; 10 conjugate gradient iterations; and Fisher information/Gauss-Newton damping coefﬁcient λ = 0 .
ARM: n = 5 step returns; sampling batch size 12500 ; 4000 Adam minibatches in the ﬁrst sampling iteration, 3000 Adam minibatches in all subsequent sampling iterations; target update step size τ = 0 .
6. O FF - POLICY ARM VIA IMPORTANCE SAMPLING
Our current approach to running ARM with off-policy data consists of applying an importance sampling correction directly to the n -step returns.
Given the behavior policy µ under which the data was sampled, the current policy π t under which we want to perform estimation, and an importance sampling weight clip c for variance reduction, the corrected n -step return we use is:
[Equation]
[Equation]
where the truncated importance weight w µ (cid:107) π t ( a | o ) is deﬁned (Ionides, 2008)
[Equation]
Our choice of c = 1 in our experiments was inspired by Wang et al.
We found that c =worked well but note other choices for c may also be reasonable.
When applying our importance sampling correction
[Equation]
we preserve all details of the ARM algorithm except for two aspects: the transition sampling strategy (a ﬁnite memory of previous batches are cached and uniformly sampled) and the regression targets for learning the value functions.
Specifically, the regression targets v ( o k ; ϕ ) , q ( o k , a k ; ϕ )
[Equation]
and ¯ q + ( o k , a k ; ϕ ) (Equations (11)–(13)) are modiﬁed to the following:
[Equation]
Note that the target value function V (cid:48)
( o k + n ; ϕ ) does not require an importance sampling correction because V (cid:48) already approximates the on-policy value function V π t ( o k + n ; θ t ) .
6. A DDITIONAL EXPERIMENTS 6.3. A TARI GAMES
Although our primary interest is in partially observable reinforcement learning domains, we also want to check that ARM works in fully observable, nearly Markovian environments, such as Atari 2600 games.
We consider two baselines: double deep Q-learning, and double deep ﬁtted Q-iteration which is a batch counterpart to double DQN.
We ﬁnd that double deep Q-learning is a strong baseline for learning to play Atari games, although ARM still successfully learns interesting policies.
One major beneﬁt of Q-learning-based methods is the ability to utilize a large off-policy replay memory.
Our results on a suite of Atari games are in Figure 5.
[Figure]