Original: A NALYZING THE R OLE OF T EMPORAL D IFFERENCING IN D EEP R EINFORCEMENT L EARNING
Revised: A NALYZING THE R OLE OF T EMPORAL D IFFERENCING IN D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Wide adoption of deep networks as function approximators in modern reinforcement learning (RL) is changing the research environment, both with regard to best practices and application domains.
Revised: Wide adoption of deep networks as function approximators in modern reinforcement learning (RL) is changing the research environment, both with regard to best practices and application domains.

Original: Yet, our understanding of RL methods has been shaped by theoretical and empirical results with tabular representations and linear function approximators.
Revised: Yet, our understanding of RL methods has been shaped by theoretical and empirical results with tabular representations and linear function approximators.

Original: These results suggest that RL methods using temporal differencing (TD) are superior to direct Monte Carlo (MC) estimation.
Revised: These results suggest that RL methods using temporal differencing (TD) are superior to direct Monte Carlo (MC) estimation.

Original: In this paper, we re-examine the role of TD in modern deep RL, using specially designed environments that each control for a speciﬁc factor that affects performance, such as reward sparsity, reward delay or the perceptual complexity of the task.
Revised: In this paper, we re-examine the role of TD in modern deep RL, using specially designed environments that each control for a speciﬁc factor that affects performance, such as reward sparsity, reward delay or the perceptual complexity of the task.

Original: When comparing TD with inﬁnite horizon MC, we are able to reproduce the results from the past in modern settings characterized by perceptual complexity and deep nonlinear models.
Revised: When comparing TD with inﬁnite horizon MC, we are able to reproduce the results from the past in modern settings characterized by perceptual complexity and deep nonlinear models.

Original: However, we also ﬁnd that ﬁnite horizon MC methods are not inferior to TD, even in sparse or delayed reward tasks, making
Revised: However, we also ﬁnd that ﬁnite horizon MC methods are not inferior to TD, even in sparse or delayed reward tasks, making

Original: MC a viable alternative to TD.
Revised: MC a viable alternative to TD.

Original: We discuss the role of perceptual complexity in reconciling these ﬁndings with classic empirical results.
Revised: We discuss the role of perceptual complexity in reconciling these ﬁndings with classic empirical results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The use of deep networks as function approximators has signiﬁcantly expanded the range of problems that can be successfully tackled with reinforcement learning (RL).
Revised: The use of deep networks as function approximators has signiﬁcantly expanded the range of problems that can be successfully tackled with reinforcement learning (RL).

Original: However, there is little understanding of when and why certain deep reinforcement learning (DRL) algorithms work well.
Revised: However, there is little understanding of when and why certain deep reinforcement learning (DRL) algorithms work well.

Original: Theoretical results are mainly based on tabular environments or linear function approximators (Sutton & Barto, 2017).
Revised: Theoretical results are mainly based on tabular environments or linear function approximators (Sutton & Barto, 2017).

Original: Their assumptions do not cover the typical application domains of DRL, which feature extremely high input dimensionality (typically in the tens of thousands) and the use of nonlinear function approximators.
Revised: Their assumptions do not cover the typical application domains of DRL, which feature extremely high input dimensionality (typically in the tens of thousands) and the use of nonlinear function approximators.

Original: Thus, our understanding of DRL is based primarily on empirical results, and these empirical results guide the design of DRL algorithms.
Revised: Thus, our understanding of DRL is based primarily on empirical results, and these empirical results guide the design of DRL algorithms.

Original: One such design decision common to the vast majority of existing value-based DRL methods is the use of temporal difference (TD) learning – training predictive models by bootstrapping based on their own predictions.
Revised: One such design decision common to the vast majority of existing value-based DRL methods is the use of temporal difference (TD) learning – training predictive models by bootstrapping based on their own predictions.

Original: This design decision is primarily based on evidence from the pre-DRL era (Sutton, 1988; 1995).
Revised: This design decision is primarily based on evidence from the pre-DRL era (Sutton, 1988; 1995).

Original: The results of those experimental studies are well-known and clearly demonstrate that simple supervised learning, also known as Monte Carlo (MC) prediction, is outperformed by pure TD learning, which, in turn, is outperformed by TD( λ ) – a method that can be seen as a mixture of TD with MC (Sutton, 1988).
Revised: The results of those experimental studies are well-known and clearly demonstrate that simple supervised learning, also known as Monte Carlo (MC) prediction, is outperformed by pure TD learning, which, in turn, is outperformed by TD( λ ) – a method that can be seen as a mixture of TD with MC (Sutton, 1988).

Original: However, recent research has shown (Dosovitskiy & Koltun, 2017) that an algorithm based on Monte
Revised: However, recent research has shown (Dosovitskiy & Koltun, 2017) that an algorithm based on Monte

Original: Carlo prediction can outperform TD-based methods on complex sensorimotor control tasks in threedimensional environments.
Revised: Carlo prediction can outperform TD-based methods on complex sensorimotor control tasks in threedimensional environments.

Original: These results suggest that the classic understanding of the relative performance of TD and MC may not hold in modern settings.
Revised: These results suggest that the classic understanding of the relative performance of TD and MC may not hold in modern settings.

Original: This evidence is not conclusive: the algorithm proposed by Dosovitskiy & Koltun (2017) involves custom components such as parametrized goals and decomposed rewards, and therefore cannot be directly compared to TD-based baselines.
Revised: This evidence is not conclusive: the algorithm proposed by Dosovitskiy & Koltun (2017) involves custom components such as parametrized goals and decomposed rewards, and therefore cannot be directly compared to TD-based baselines.

Original: In this paper, we perform a controlled experimental study aiming at better understanding the role of temporal differencing in modern deep reinforcement learning, characterized by essentially inﬁnitedimensional state spaces, extremely high observation dimensionality, and deep nonlinear models used as function approximators.
Revised: In this paper, we perform a controlled experimental study aiming at better understanding the role of temporal differencing in modern deep reinforcement learning, characterized by essentially inﬁnitedimensional state spaces, extremely high observation dimensionality, and deep nonlinear models used as function approximators.

Original: We focus on environments with visual inputs and discrete action sets, and algorithms that involve prediction of value or action-value functions.
Revised: We focus on environments with visual inputs and discrete action sets, and algorithms that involve prediction of value or action-value functions.

Original: This is in contrast to value-free policy optimization algorithms (Schulman et al., 2015; Levine & Koltun, 2013) and tasks with continuous action spaces and low-dimensional vectorial state representations that have been extensively benchmarked by Duan et al.
Revised: This is in contrast to value-free policy optimization algorithms (Schulman et al., 2015; Levine & Koltun, 2013) and tasks with continuous action spaces and low-dimensional vectorial state representations that have been extensively benchmarked by Duan et al.

Original: (2016) and Henderson et al.
Revised: (2016) and Henderson et al.

Original: We base our study on deep Q -learning
Revised: We base our study on deep Q -learning

Original: (Mnih et al., 2015), where the Q -function is learned either via temporal differencing or via a ﬁnite-horizon Monte Carlo method.
Revised: (Mnih et al., 2015), where the Q -function is learned either via temporal differencing or via a ﬁnite-horizon Monte Carlo method.

Original: To ensure that our conclusions are not limited to pure value-based methods, we additionally evaluate asynchronous advantage actor-critic (A3C), which combines temporal differencing with a policy gradient method (Mnih et al., 2016).
Revised: To ensure that our conclusions are not limited to pure value-based methods, we additionally evaluate asynchronous advantage actor-critic (A3C), which combines temporal differencing with a policy gradient method (Mnih et al., 2016).

Original: Our main focus is on performing controlled experiments, both in terms of algorithm conﬁgurations and environment properties.
Revised: Our main focus is on performing controlled experiments, both in terms of algorithm conﬁgurations and environment properties.

Original: This is in contrast to previous works, which typically benchmark a number of existing algorithms on a set of standard environments.
Revised: This is in contrast to previous works, which typically benchmark a number of existing algorithms on a set of standard environments.

Original: While proper benchmarking is crucial for tracking the progress of the ﬁeld, it is not always sufﬁcient for understanding the reasons behind good or poor performance of the algorithms.
Revised: While proper benchmarking is crucial for tracking the progress of the ﬁeld, it is not always sufﬁcient for understanding the reasons behind good or poor performance of the algorithms.

Original: In this work, we ensure that the algorithms are comparable by implementing them in a common software framework.
Revised: In this work, we ensure that the algorithms are comparable by implementing them in a common software framework.

Original: By varying the parameters such as the balance between TD and MC in the learning update or the prediction horizon, we are able to clearly isolate the effect of these parameters on learning.
Revised: By varying the parameters such as the balance between TD and MC in the learning update or the prediction horizon, we are able to clearly isolate the effect of these parameters on learning.

Original: Moreover, we designed a series of controlled scenarios that focus on speciﬁc characteristics of RL problems: reward sparsity, reward delay, perceptual complexity, and properties of terminal states.
Revised: Moreover, we designed a series of controlled scenarios that focus on speciﬁc characteristics of RL problems: reward sparsity, reward delay, perceptual complexity, and properties of terminal states.

Original: Results in these environments shed light on strengths and weaknesses of algorithms under investigation.
Revised: Results in these environments shed light on strengths and weaknesses of algorithms under investigation.

Original: Our ﬁndings in modern DRL scenarios both support and contradict previous results on merits of
Revised: Our ﬁndings in modern DRL scenarios both support and contradict previous results on merits of

Original: TD.
Revised: TD.

Original: On the one hand, value-based inﬁnite-horizon methods perform best in the regime which is a mixture of TD and MC, similar to the TD ( λ ) results of Sutton (1988).
Revised: On the one hand, value-based inﬁnite-horizon methods perform best in the regime which is a mixture of TD and MC, similar to the TD ( λ ) results of Sutton (1988).

Original: On the other hand, in sharp contrast with previous belief, we observe that Monte Carlo algorithms can perform very well on challenging RL tasks.
Revised: On the other hand, in sharp contrast with previous belief, we observe that Monte Carlo algorithms can perform very well on challenging RL tasks.

Original: This is made possible by simply limiting the prediction to a ﬁnite horizon.
Revised: This is made possible by simply limiting the prediction to a ﬁnite horizon.

Original: Surprisingly, ﬁnite-horizon Monte Carlo training is successful in dealing with sparse and delayed rewards, which are generally assumed to impair this class of methods.
Revised: Surprisingly, ﬁnite-horizon Monte Carlo training is successful in dealing with sparse and delayed rewards, which are generally assumed to impair this class of methods.

Original: Monte Carlo training is also more stable to noisy rewards and is particularly robust to perceptual complexity and variability.
Revised: Monte Carlo training is also more stable to noisy rewards and is particularly robust to perceptual complexity and variability.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We work in a standard reinforcement learning setting of an agent acting in an environment over discrete time steps.
Revised: We work in a standard reinforcement learning setting of an agent acting in an environment over discrete time steps.

Original: At each time step t , the agent receives an observation o t and selects an action a t .
Revised: At each time step t , the agent receives an observation o t and selects an action a t .

Original: We assume partial observability: the observation o t need not carry complete information about the environment and can be seen as a function of the environment’s “true state”.
Revised: We assume partial observability: the observation o t need not carry complete information about the environment and can be seen as a function of the environment’s “true state”.

Original: We assume an episodic setup, where an episode starts with time step 0 and concludes at a terminal time step T .
Revised: We assume an episodic setup, where an episode starts with time step 0 and concludes at a terminal time step T .

Original: We denote by s t the tuple of all observations collected by the agent from the beginning of the episode: s t = (cid:104) o 0 , . . .
Revised: We denote by s t the tuple of all observations collected by the agent from the beginning of the episode: s t = (cid:104) o 0 , . . .

Original: , o t (cid:105) .
Revised: , o t (cid:105) .

Original: (In practice we will only include a set of recent observations in s .)
Revised: (In practice we will only include a set of recent observations in s .)

Original: The objective is to ﬁnd a policy π ( a t | s t ) that maximizes the expected return – the sum of all future rewards through the remainder of the episode:
Revised: The objective is to ﬁnd a policy π ( a t | s t ) that maximizes the expected return – the sum of all future rewards through the remainder of the episode:

Original: [Equation]
Revised: [Equation]

Original: This sum can become arbitrarily large for long episodes.
Revised: This sum can become arbitrarily large for long episodes.

Original: To avoid divergence, temporally distant rewards can be discounted.
Revised: To avoid divergence, temporally distant rewards can be discounted.

Original: This is typically done in one of two ways: by introducing a discount factor γ or by truncating the sum after a ﬁxed number of steps (horizon) τ .
Revised: This is typically done in one of two ways: by introducing a discount factor γ or by truncating the sum after a ﬁxed number of steps (horizon) τ .

Original: [Equation]
Revised: [Equation]

Original: The parameters γ and τ regulate the contribution of temporally distant rewards to the agent’s objective.
Revised: The parameters γ and τ regulate the contribution of temporally distant rewards to the agent’s objective.

Original: In what follows ˆ R t stands for R γt or R τt .
Revised: In what follows ˆ R t stands for R γt or R τt .

Original: For a given policy π , the value function and the action-value function are deﬁned as expected returns that are conditioned, respectively, on the observation or the observation-action pair:
Revised: For a given policy π , the value function and the action-value function are deﬁned as expected returns that are conditioned, respectively, on the observation or the observation-action pair:

Original: [Equation]
Revised: [Equation]

Original: Optimal value and action-value functions are deﬁned as the maxima over all possible policies:
Revised: Optimal value and action-value functions are deﬁned as the maxima over all possible policies:

Original: [Equation]
Revised: [Equation]

Original: In value-based, model-free reinforcement learning, the value or action value are estimated by a function approximator V with parameters θ .
Revised: In value-based, model-free reinforcement learning, the value or action value are estimated by a function approximator V with parameters θ .

Original: The function approximator is typically trained by minimizing a loss between the current estimate and a target value:
Revised: The function approximator is typically trained by minimizing a loss between the current estimate and a target value:

Original: [Equation]
Revised: [Equation]

Original: The learning procedure for the action-value function is analogous.
Revised: The learning procedure for the action-value function is analogous.

Original: Hence, we focus on the value function in the remainder of this section.
Revised: Hence, we focus on the value function in the remainder of this section.

Original: Reinforcement learning methods differ in how the target value is obtained.
Revised: Reinforcement learning methods differ in how the target value is obtained.

Original: The most straightforward approach is to use the empirical return as target: i.e., V target =
Revised: The most straightforward approach is to use the empirical return as target: i.e., V target =

Original: R γt or V target =
Revised: R γt or V target =

Original: R τt .
Revised: R τt .

Original: This is referred to as Monte Carlo (MC) training, since the empirical loss becomes a Monte Carlo estimate of the expected loss.
Revised: This is referred to as Monte Carlo (MC) training, since the empirical loss becomes a Monte Carlo estimate of the expected loss.

Original: Using the empirical return as target requires propagating the environment forward before a training step can take place – by τ steps for ﬁnite-horizon return R τt or until the end of the episode for discounted return R γt .
Revised: Using the empirical return as target requires propagating the environment forward before a training step can take place – by τ steps for ﬁnite-horizon return R τt or until the end of the episode for discounted return R γt .

Original: This increases the variance of the target value for long horizons and large discount factors.
Revised: This increases the variance of the target value for long horizons and large discount factors.

Original: An alternative to Monte Carlo training is temporal difference (TD) learning (Sutton, 1988).
Revised: An alternative to Monte Carlo training is temporal difference (TD) learning (Sutton, 1988).

Original: The idea is to estimate the return by bootstrapping from the function approximator itself, after acting for a ﬁxed number of steps n :
Revised: The idea is to estimate the return by bootstrapping from the function approximator itself, after acting for a ﬁxed number of steps n :

Original: [Equation]
Revised: [Equation]

Original: TD learning is typically used with inﬁnite-horizon returns.
Revised: TD learning is typically used with inﬁnite-horizon returns.

Original: When the rollout length n approaches inﬁnity (or, in practice, maximal episode duration T max ), TD becomes identical to Monte Carlo training.
Revised: When the rollout length n approaches inﬁnity (or, in practice, maximal episode duration T max ), TD becomes identical to Monte Carlo training.

Original: TD learning applied to the action-value function is known as Q
Revised: TD learning applied to the action-value function is known as Q

Original: -learning
Revised: -learning

Original: (Watkins, 1989;
Revised: (Watkins, 1989;

Original: Watkins & Dayan, 1992; Peng & Williams, 1996; Mnih et al., 2015).
Revised: Watkins & Dayan, 1992; Peng & Williams, 1996; Mnih et al., 2015).

Original: An alternative to value-based methods are policy-based methods, which directly parametrize the policy π ( a | s ; θ ) .
Revised: An alternative to value-based methods are policy-based methods, which directly parametrize the policy π ( a | s ; θ ) .

Original: An approximate gradient of the expected return is computed with respect to the policy parameters, and the return is maximized using gradient ascent.
Revised: An approximate gradient of the expected return is computed with respect to the policy parameters, and the return is maximized using gradient ascent.

Original: Williams (1992) has shown that an unbiased estimate of the gradient can be computed as ∇ θ log π ( a | s ; θ ) ( R t − b t ( s t )) , where the function b t ( s t ) is called a baseline and can be chosen so as to decrease the variance of the estimator.
Revised: Williams (1992) has shown that an unbiased estimate of the gradient can be computed as ∇ θ log π ( a | s ; θ ) ( R t − b t ( s t )) , where the function b t ( s t ) is called a baseline and can be chosen so as to decrease the variance of the estimator.

Original: A common choice for the baseline is the value function: b t ( s t ) =
Revised: A common choice for the baseline is the value function: b t ( s t ) =

Original: V π ( s t ) .
Revised: V π ( s t ) .

Original: A combination of policy gradient with a baseline value function learned via TD is referred to as an actor-critic method, with policy π being the actor and the value function estimator being the critic.
Revised: A combination of policy gradient with a baseline value function learned via TD is referred to as an actor-critic method, with policy π being the actor and the value function estimator being the critic.

Original:  E XPERIMENTAL S ETUP 3. A LGORITHMS
Revised:  E XPERIMENTAL S ETUP 3. A LGORITHMS

Original: In our analysis of temporal differencing we focus on three key characteristics of RL algorithms.
Revised: In our analysis of temporal differencing we focus on three key characteristics of RL algorithms.

Original: The ﬁrst is the balance between TD and MC in the learning update.
Revised: The ﬁrst is the balance between TD and MC in the learning update.

Original: The second is the prediction horizon, in particular inﬁnite versus ﬁnite horizon.
Revised: The second is the prediction horizon, in particular inﬁnite versus ﬁnite horizon.

Original: The third is the use of pure value-based learning versus an actor-critic approach which includes an explicitly parametrized policy.
Revised: The third is the use of pure value-based learning versus an actor-critic approach which includes an explicitly parametrized policy.

Original: To study the ﬁrst aspect, we use asynchronous n-step Q-learning (n-step Q )
Revised: To study the ﬁrst aspect, we use asynchronous n-step Q-learning (n-step Q )

Original: (Mnih et al., 2016).
Revised: (Mnih et al., 2016).

Original: In this algorithm, an action-value function is learned with n-step TD (Eq.
Revised: In this algorithm, an action-value function is learned with n-step TD (Eq.

Original: (6)), and actions are selected greedily according to this function.
Revised: (6)), and actions are selected greedily according to this function.

Original: By varying the rollout length n , we can smoothly interpolate between pure TD and pure MC updates.
Revised: By varying the rollout length n , we can smoothly interpolate between pure TD and pure MC updates.

Original: In order to analyze the second aspect, we implemented a ﬁnite-horizon Monte Carlo version of n-step Q , which we call Q MC .
Revised: In order to analyze the second aspect, we implemented a ﬁnite-horizon Monte Carlo version of n-step Q , which we call Q MC .

Original: This algorithm can be seen as a simpliﬁed version of Direct Future Prediction (Dosovitskiy & Koltun, 2017).
Revised: This algorithm can be seen as a simpliﬁed version of Direct Future Prediction (Dosovitskiy & Koltun, 2017).

Original: Finally, we select asynchronous advantage actor-critic (A3C) (Mnih et al., 2016) to study the third aspect.
Revised: Finally, we select asynchronous advantage actor-critic (A3C) (Mnih et al., 2016) to study the third aspect.

Original: In A3C, the value function estimate is learned with n-step TD, and a policy is trained with policy gradient.
Revised: In A3C, the value function estimate is learned with n-step TD, and a policy is trained with policy gradient.

Original: This allows us to evaluate the interplay of TD learning and policy gradient learning.
Revised: This allows us to evaluate the interplay of TD learning and policy gradient learning.

Original: To ensure that the comparison is fully controlled and fair, we implemented all algorithms in the asynchronous training framework proposed by Mnih et al.
Revised: To ensure that the comparison is fully controlled and fair, we implemented all algorithms in the asynchronous training framework proposed by Mnih et al.

Original: Multiple actor threads are running in parallel and send the weight updates asynchronously to a parameter server.
Revised: Multiple actor threads are running in parallel and send the weight updates asynchronously to a parameter server.

Original: For A3C and n-step Q , we use the algorithms as described by Mnih et al.
Revised: For A3C and n-step Q , we use the algorithms as described by Mnih et al.

Original: Q MC is the n-step Q algorithm where the n-step TD targets are replaced by ﬁnite-horizon MC targets.
Revised: Q MC is the n-step Q algorithm where the n-step TD targets are replaced by ﬁnite-horizon MC targets.

Original: Note that switching to ﬁnite horizon necessitates a small additional change in the Q MC algorithm.
Revised: Note that switching to ﬁnite horizon necessitates a small additional change in the Q MC algorithm.

Original: In practice, in n-step Q each parameter update is not just an n -step TD update, but a sum of all updates for rollouts from 1 to n .
Revised: In practice, in n-step Q each parameter update is not just an n -step TD update, but a sum of all updates for rollouts from 1 to n .

Original: This improves the stability of training.
Revised: This improves the stability of training.

Original: In Q MC such accumulation of updates is impossible, since predictions for different horizons are not compatible.
Revised: In Q MC such accumulation of updates is impossible, since predictions for different horizons are not compatible.

Original: We therefore always predict several Q -values corresponding to different horizons, similar to Dosovitskiy &
Revised: We therefore always predict several Q -values corresponding to different horizons, similar to Dosovitskiy &

Original: Koltun (2017).
Revised: Koltun (2017).

Original: Speciﬁcally, for horizon τ = 2 K , we additionally predict Q -values for horizons { 2 k } 0 ≤ k<K .
Revised: Speciﬁcally, for horizon τ = 2 K , we additionally predict Q -values for horizons { 2 k } 0 ≤ k<K .

Original: This design choice is further explained and supported with experiments in the supplement.
Revised: This design choice is further explained and supported with experiments in the supplement.

Original: Apart from this, there is no difference between n-step Q and Q MC .
Revised: Apart from this, there is no difference between n-step Q and Q MC .

Original: 3. E NVIRONMENTS
Revised: 3. E NVIRONMENTS

Original: To calibrate our implementations against results available in the literature, we begin by conducting experiments on several standard benchmark environments: ﬁve Atari games from the Arcade
Revised: To calibrate our implementations against results available in the literature, we begin by conducting experiments on several standard benchmark environments: ﬁve Atari games from the Arcade

Original: Learning Environment (Bellemare et al., 2013) and two environments based on ﬁrst-person-view 3D simulation in the ViZDoom framework (Kempka et al., 2016).
Revised: Learning Environment (Bellemare et al., 2013) and two environments based on ﬁrst-person-view 3D simulation in the ViZDoom framework (Kempka et al., 2016).

Original: We used a set of Atari games commonly analyzed in the literature: Space Invaders, Pong, Beam Rider, Sea Quest, and Frostbite (Mnih et al., 2015; Schulman et al., 2015; Lake et al., 2016).
Revised: We used a set of Atari games commonly analyzed in the literature: Space Invaders, Pong, Beam Rider, Sea Quest, and Frostbite (Mnih et al., 2015; Schulman et al., 2015; Lake et al., 2016).

Original: For the ViZDoom environments, we used the Navigation, Battle and Battle2 scenarios from Dosovitskiy & Koltun (2017).
Revised: For the ViZDoom environments, we used the Navigation, Battle and Battle2 scenarios from Dosovitskiy & Koltun (2017).

Original: Our main experiments are on sequences of specialized environments.
Revised: Our main experiments are on sequences of specialized environments.

Original: Each sequence is designed such that a single factor of variation is modiﬁed in a controlled fashion.
Revised: Each sequence is designed such that a single factor of variation is modiﬁed in a controlled fashion.

Original: This allows us to study the effect of this factor.
Revised: This allows us to study the effect of this factor.

Original: Factors of variation include: reward sparsity, reward delay, reward type, and perceptual complexity.
Revised: Factors of variation include: reward sparsity, reward delay, reward type, and perceptual complexity.

Original: These environments form the heart of our analysis and will be publicly released for reproducibility.
Revised: These environments form the heart of our analysis and will be publicly released for reproducibility.

Original: For the controlled environments, we used the ViZDoom platform.
Revised: For the controlled environments, we used the ViZDoom platform.

Original: This platform is compatible with existing map editors with built-in scripting, which allows for ﬂexible and controlled speciﬁcation of different scenarios.
Revised: This platform is compatible with existing map editors with built-in scripting, which allows for ﬂexible and controlled speciﬁcation of different scenarios.

Original: In comparison to Atari games, ViZDoom offers a more realistic setting with a three-dimensional environment and partially observed ﬁrst-person navigation.
Revised: In comparison to Atari games, ViZDoom offers a more realistic setting with a three-dimensional environment and partially observed ﬁrst-person navigation.

Original: We now brieﬂy describe the tasks.
Revised: We now brieﬂy describe the tasks.

Original: Further details are provided in the supplement.
Revised: Further details are provided in the supplement.

Original: Basic health gathering.
Revised: Basic health gathering.

Original: The basis for our controlled scenarios is the health gathering task.
Revised: The basis for our controlled scenarios is the health gathering task.

Original: In this scenario, the agent’s aim is to collect health kits while navigating through a maze using visual input.
Revised: In this scenario, the agent’s aim is to collect health kits while navigating through a maze using visual input.

Original: Figure 1(b) shows a typical image observed by the agent.
Revised: Figure 1(b) shows a typical image observed by the agent.

Original: The agent’s health level is constantly declining.
Revised: The agent’s health level is constantly declining.

Original: Health kits add to the health level.
Revised: Health kits add to the health level.

Original: The goal is to survive and maintain as much health as possible by collecting health kits.
Revised: The goal is to survive and maintain as much health as possible by collecting health kits.

Original: To be precise, the agent loses 6 health units every 8 steps, and obtains 20 health units when collecting a health pack.
Revised: To be precise, the agent loses 6 health units every 8 steps, and obtains 20 health units when collecting a health pack.

Original: The agent’s total health cannot exceed 100 .
Revised: The agent’s total health cannot exceed 100 .

Original: The reward is +1 when the agent collects a health kit and 0 otherwise.
Revised: The reward is +1 when the agent collects a health kit and 0 otherwise.

Original: There are 16 health kits in the labyrinth at any given time.
Revised: There are 16 health kits in the labyrinth at any given time.

Original: When the agent collects one of them, a new one appears at a random location.
Revised: When the agent collects one of them, a new one appears at a random location.

Original: An episode is terminated after 525 steps, which is equivalent to 1 minute of in-game time.
Revised: An episode is terminated after 525 steps, which is equivalent to 1 minute of in-game time.

Original: Terminal states.
Revised: Terminal states.

Original: To test the effect of terminal states on the performance of the algorithms, we modiﬁed the health gathering scenario so that each episode terminates after m health kits are collected.
Revised: To test the effect of terminal states on the performance of the algorithms, we modiﬁed the health gathering scenario so that each episode terminates after m health kits are collected.

Original: For m = 1 , all useful training signals come from the terminal state.
Revised: For m = 1 , all useful training signals come from the terminal state.

Original: With larger m , the importance of terminal states diminishes.
Revised: With larger m , the importance of terminal states diminishes.

Original: Delayed rewards.
Revised: Delayed rewards.

Original: In this sequence of scenarios we introduce a delay between the act of collecting a health kit and its effect – an increase in health and a reward of 1 .
Revised: In this sequence of scenarios we introduce a delay between the act of collecting a health kit and its effect – an increase in health and a reward of 1 .

Original: We have set up environments with delays of 2 , 4 , 8 , 16 , and 32 steps.
Revised: We have set up environments with delays of 2 , 4 , 8 , 16 , and 32 steps.

Original: Sparse rewards.
Revised: Sparse rewards.

Original: To examine the effect of reward sparsity, we varied the number of available health kits on the map.
Revised: To examine the effect of reward sparsity, we varied the number of available health kits on the map.

Original: We created two variations of the basic health gathering environment with increasingly sparse rewards.
Revised: We created two variations of the basic health gathering environment with increasingly sparse rewards.

Original: In the ‘Sparse’ setting, there are 4 health kits in the labyrinth – four times fewer than in the basic setting.
Revised: In the ‘Sparse’ setting, there are 4 health kits in the labyrinth – four times fewer than in the basic setting.

Original: In the ‘Very Sparse’ setting, only 2 health kits are in the labyrinth – eight times fewer than in the basic setting.
Revised: In the ‘Very Sparse’ setting, only 2 health kits are in the labyrinth – eight times fewer than in the basic setting.

Original: In order to isolate the effect of sparsity and keep the general difﬁculty of the task ﬁxed, we accordingly adjusted the amount of health the agent loses per time period: 3 in the Sparse conﬁguration and 2 in Very Sparse.
Revised: In order to isolate the effect of sparsity and keep the general difﬁculty of the task ﬁxed, we accordingly adjusted the amount of health the agent loses per time period: 3 in the Sparse conﬁguration and 2 in Very Sparse.

Original: In the Very Sparse scenario under random exploration, the agent gathers a health kit on average every 6 , 440 steps.
Revised: In the Very Sparse scenario under random exploration, the agent gathers a health kit on average every 6 , 440 steps.

Original: Reward type.
Revised: Reward type.

Original: In this scenario, we compare the standard binary reward with its more natural but more noisy counterpart.
Revised: In this scenario, we compare the standard binary reward with its more natural but more noisy counterpart.

Original: In the basic scenario above, the reward is +1 for gathering a health kit and 0 otherwise.
Revised: In the basic scenario above, the reward is +1 for gathering a health kit and 0 otherwise.

Original: A more natural measure of success in the health gathering task is the actual change in
Revised: A more natural measure of success in the health gathering task is the actual change in

Original: [Figure]
Revised: [Figure]

Original: health.
Revised: health.

Original: With this reward, the agent would directly aim to maximize its health.
Revised: With this reward, the agent would directly aim to maximize its health.

Original: In this conﬁguration we therefore use a scaled change in health as the reward signal.
Revised: In this conﬁguration we therefore use a scaled change in health as the reward signal.

Original: This reward is more challenging than the basic binary reward due to its noisiness (health is decreased only every eighth step) and the variance in the reward after collecting a health kit due to the total health limit.
Revised: This reward is more challenging than the basic binary reward due to its noisiness (health is decreased only every eighth step) and the variance in the reward after collecting a health kit due to the total health limit.

Original: Perceptual complexity.
Revised: Perceptual complexity.

Original: To analyze the effect of perceptual complexity, we designed variants of the health gathering task with different input representations.
Revised: To analyze the effect of perceptual complexity, we designed variants of the health gathering task with different input representations.

Original: First, to increase the perceptual complexity of the task, we replaced the single maze used in the basic health gathering scenario byrandomly textured versions, some of which are shown in Figure 1(c).
Revised: First, to increase the perceptual complexity of the task, we replaced the single maze used in the basic health gathering scenario byrandomly textured versions, some of which are shown in Figure 1(c).

Original: The labyrinth’s texture is changed after each episode during both training and evaluation.
Revised: The labyrinth’s texture is changed after each episode during both training and evaluation.

Original: We also created two variants of the health gathering task with reduced visual complexity.
Revised: We also created two variants of the health gathering task with reduced visual complexity.

Original: These are the only controlled scenarios not using the ViZDoom framework.
Revised: These are the only controlled scenarios not using the ViZDoom framework.

Original: Both are based on a grid world, where the agent is navigating an 8 × 8 room with 5 available actions: wait, up, down, left, and right.
Revised: Both are based on a grid world, where the agent is navigating an 8 × 8 room with 5 available actions: wait, up, down, left, and right.

Original: There are 4 randomly placed health kits in the room, and the aim of the agent is to collect these, with reward +1 for collecting a health kit and 0 otherwise.
Revised: There are 4 randomly placed health kits in the room, and the aim of the agent is to collect these, with reward +1 for collecting a health kit and 0 otherwise.

Original: Each time a health kit is collected a new one appears in a random location.
Revised: Each time a health kit is collected a new one appears in a random location.

Original: The two variants differ in the representation that is fed to the agent.
Revised: The two variants differ in the representation that is fed to the agent.

Original: In one, the agent’s input is a 10 -dimensional vector that concatenates the 2D Cartesian coordinates of the agent itself and the 4 health kits, sorted by their distance to the agent.
Revised: In one, the agent’s input is a 10 -dimensional vector that concatenates the 2D Cartesian coordinates of the agent itself and the 4 health kits, sorted by their distance to the agent.

Original: In the other variant, the agent gets as input an 8 × 8 image of the grid world.
Revised: In the other variant, the agent gets as input an 8 × 8 image of the grid world.

Original: This is a raster RGB image of a map of the grid world, as shown in Figure 1(a).
Revised: This is a raster RGB image of a map of the grid world, as shown in Figure 1(a).

Original: This latter representation is compatible with the input representation used in the ﬁrst-person variants (i.e., an RGB image), but the content of the input image is drastically simpliﬁed.
Revised: This latter representation is compatible with the input representation used in the ﬁrst-person variants (i.e., an RGB image), but the content of the input image is drastically simpliﬁed.

Original: 3. A LGORITHM DETAILS
Revised: 3. A LGORITHM DETAILS

Original: We used identical network architectures for the three algorithms in all experiments.
Revised: We used identical network architectures for the three algorithms in all experiments.

Original: For experiments in Atari and ViZDoom domains we used deep convolutional networks similar to the one used by Mnih et al.
Revised: For experiments in Atari and ViZDoom domains we used deep convolutional networks similar to the one used by Mnih et al.

Original: For gridworld experiments we used fully-connected networks with three hidden layers.
Revised: For gridworld experiments we used fully-connected networks with three hidden layers.

Original: For Q MC and n-step Q we used dueling network architectures, similar to Wang et al.
Revised: For Q MC and n-step Q we used dueling network architectures, similar to Wang et al.

Original: The exact architectures are speciﬁed in the supplement.
Revised: The exact architectures are speciﬁed in the supplement.

Original: For experiments in Atari environments we followed a common practice and fed the 4 most recent frames to the networks.
Revised: For experiments in Atari environments we followed a common practice and fed the 4 most recent frames to the networks.

Original: In all other environments the input was limited to the observation from the current time step.
Revised: In all other environments the input was limited to the observation from the current time step.

Original: In ViZDoom scenarios, in addition to the observed image we fed a vector of measurements to all networks.
Revised: In ViZDoom scenarios, in addition to the observed image we fed a vector of measurements to all networks.

Original: The measurements are the agent’s scalar health in the health gathering scenarios and a three-dimensional vector of the agent’s health, ammo, and frags in the battle scenario.
Revised: The measurements are the agent’s scalar health in the health gathering scenarios and a three-dimensional vector of the agent’s health, ammo, and frags in the battle scenario.

Original: We trained all models with 16 asynchronous actor threads, for a total of 60 million steps.
Revised: We trained all models with 16 asynchronous actor threads, for a total of 60 million steps.

Original: We identiﬁed optimal hyperparameters for each algorithm via a hyperparameter search on a subset of environments and used these ﬁxed hyperparameters for all environments, unless noted otherwise.
Revised: We identiﬁed optimal hyperparameters for each algorithm via a hyperparameter search on a subset of environments and used these ﬁxed hyperparameters for all environments, unless noted otherwise.

Original: [Table]
Revised: [Table]

Original: For evaluation, we trained three models on each task, selected the best-performing snapshot for each training run, and averaged the performance of these three best-performing snapshots.
Revised: For evaluation, we trained three models on each task, selected the best-performing snapshot for each training run, and averaged the performance of these three best-performing snapshots.

Original: Further details are provided in the supplement.
Revised: Further details are provided in the supplement.

Original:  R ESULTS 4. C ALIBRATION
Revised:  R ESULTS 4. C ALIBRATION

Original: We start by calibrating our implementations of the methods against published results reported in the literature.
Revised: We start by calibrating our implementations of the methods against published results reported in the literature.

Original: To this end, we train and test our implementations on standard environments used in prior work.
Revised: To this end, we train and test our implementations on standard environments used in prior work.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: Our implementations perform similarly to corresponding results reported in prior work.
Revised: Our implementations perform similarly to corresponding results reported in prior work.

Original: For A3C the results are signiﬁcantly different only for BeamRider.
Revised: For A3C the results are signiﬁcantly different only for BeamRider.

Original: However, in Mnih et al.
Revised: However, in Mnih et al.

Original: the evaluation used the average over the best 5 out of 50 experiments with different learning rates.
Revised: the evaluation used the average over the best 5 out of 50 experiments with different learning rates.

Original: We used the average over 3 runs with a ﬁxed learning rate.
Revised: We used the average over 3 runs with a ﬁxed learning rate.

Original: Since the results for BeamRider have a high variance even for very small learning rate changes, this explains the difference between the results.
Revised: Since the results for BeamRider have a high variance even for very small learning rate changes, this explains the difference between the results.

Original: On the ViZDoom scenarios, the Q MC implementation performs on par with the DFP algorithm.
Revised: On the ViZDoom scenarios, the Q MC implementation performs on par with the DFP algorithm.

Original: This shows that DFP does not crucially depend on a decomposition of the reward into a vector of measurements, and can perform equally well given a standard RL setup with a scalar reward.
Revised: This shows that DFP does not crucially depend on a decomposition of the reward into a vector of measurements, and can perform equally well given a standard RL setup with a scalar reward.

Original: Our A3C implementation achieves signiﬁcantly better results than those reported by Dosovitskiy &
Revised: Our A3C implementation achieves signiﬁcantly better results than those reported by Dosovitskiy &

Original: Koltun (2017) on the ViZDoom scenarios.
Revised: Koltun (2017) on the ViZDoom scenarios.

Original: We attribute this to (a) using a rollout value of 20 in our experiments instead of 5 as used by Mnih et al.
Revised: We attribute this to (a) using a rollout value of 20 in our experiments instead of 5 as used by Mnih et al.

Original: and Dosovitskiy & Koltun (2017), and (b) providing the measurements as input to the network.
Revised: and Dosovitskiy & Koltun (2017), and (b) providing the measurements as input to the network.

Original: Dosovitskiy & Koltun (2017) have not tested
Revised: Dosovitskiy & Koltun (2017) have not tested

Original: DFP on Atari games.
Revised: DFP on Atari games.

Original: We ﬁnd that in these environments Q MC performs worse overall than 20-step
Revised: We ﬁnd that in these environments Q MC performs worse overall than 20-step

Original: Q and 20-step A3C.
Revised: Q and 20-step A3C.

Original: 4. V ARYING THE ROLLOUT IN TD- BASED ALGORITHMS
Revised: 4. V ARYING THE ROLLOUT IN TD- BASED ALGORITHMS

Original: By changing the rollout length n in n-step Q and A3C, we can smoothly transition between TD and MC training.
Revised: By changing the rollout length n in n-step Q and A3C, we can smoothly transition between TD and MC training.

Original: 1 -step rollouts correspond to pure bootstrapping as used in the standard Bellman equation.
Revised: 1 -step rollouts correspond to pure bootstrapping as used in the standard Bellman equation.

Original: Inﬁnite rollouts (until the terminal state), on the other hand, correspond to pure Monte
Revised: Inﬁnite rollouts (until the terminal state), on the other hand, correspond to pure Monte

Original: Carlo learning of discounted inﬁnite-horizon returns.
Revised: Carlo learning of discounted inﬁnite-horizon returns.

Original: Results on three environments – Basic health gathering, Sparse health gathering, and Battle – are presented in Figure 2.
Revised: Results on three environments – Basic health gathering, Sparse health gathering, and Battle – are presented in Figure 2.

Original: Rollout length of 20 is best on all tasks for n-step Q .
Revised: Rollout length of 20 is best on all tasks for n-step Q .

Original: Both very short and very long rollouts lead to decreased performance.
Revised: Both very short and very long rollouts lead to decreased performance.

Original: These ﬁndings are in agreement with prior results of
Revised: These ﬁndings are in agreement with prior results of

Original: TD( λ ) experiments (Sutton, 1988; 1995), considering that longer rollouts increase the MC portion of the value target, converging to a full MC update for inﬁnite rollout.
Revised: TD( λ ) experiments (Sutton, 1988; 1995), considering that longer rollouts increase the MC portion of the value target, converging to a full MC update for inﬁnite rollout.

Original: A mixture of TD and MC yields the best performance.
Revised: A mixture of TD and MC yields the best performance.

Original: The results for A3C are qualitatively similar, and again the 20 -step rollout is overall near-optimal.
Revised: The results for A3C are qualitatively similar, and again the 20 -step rollout is overall near-optimal.

Original: C ONTROLLED EXPERIMENTS
Revised: C ONTROLLED EXPERIMENTS

Original: We now proceed to a series of controlled experiments on a set of speciﬁcally designed environments and compare TD-based methods to Q MC , a purely Monte Carlo approach.
Revised: We now proceed to a series of controlled experiments on a set of speciﬁcally designed environments and compare TD-based methods to Q MC , a purely Monte Carlo approach.

Original: The motivation is as follows.
Revised: The motivation is as follows.

Original: In the previous section we have seen that very long rollouts lead to deteriorated performance of n-step Q and A3C. This can be attributed to large variance in target values.
Revised: In the previous section we have seen that very long rollouts lead to deteriorated performance of n-step Q and A3C. This can be attributed to large variance in target values.

Original: The variance can be reduced by using a ﬁnite horizon, as is the case in Q MC .
Revised: The variance can be reduced by using a ﬁnite horizon, as is the case in Q MC .

Original: However, the use of a ﬁnite horizon means that rewards that are further away than the horizon will not be part of the value target, resulting in a disadvantage in tasks with sparse or delayed rewards.
Revised: However, the use of a ﬁnite horizon means that rewards that are further away than the horizon will not be part of the value target, resulting in a disadvantage in tasks with sparse or delayed rewards.

Original: In order to evaluate this we run controlled experiments designed to isolate the reward delay, sparsity, and other factors.
Revised: In order to evaluate this we run controlled experiments designed to isolate the reward delay, sparsity, and other factors.

Original: We test 20-step Q and
Revised: We test 20-step Q and

Original: A3C (optimal rollout for TD-based methods), 5-step Q and A3C (more TD in the update), and Q MC (ﬁnite horizon Monte Carlo).
Revised: A3C (optimal rollout for TD-based methods), 5-step Q and A3C (more TD in the update), and Q MC (ﬁnite horizon Monte Carlo).

Original: Reward type.
Revised: Reward type.

Original: We contrast the standard binary reward with the more natural reward signal proportional to the change in the health level of the agent.
Revised: We contrast the standard binary reward with the more natural reward signal proportional to the change in the health level of the agent.

Original: Figure 3 (left) shows that in the scenario with binary reward the performance of Q MC , 20-step Q , and 20-step A3C is nearly identical, within 4% of each other.
Revised: Figure 3 (left) shows that in the scenario with binary reward the performance of Q MC , 20-step Q , and 20-step A3C is nearly identical, within 4% of each other.

Original: However, when trained with the noisier health-based reward, Q MC performs within 1% of the result with binary reward, but the performance of TD-based algorithms decreases significantly, especially for the 5 -step rollouts.
Revised: However, when trained with the noisier health-based reward, Q MC performs within 1% of the result with binary reward, but the performance of TD-based algorithms decreases significantly, especially for the 5 -step rollouts.

Original: These results suggest that Monte Carlo training is more robust to noisy rewards than TD-based methods.
Revised: These results suggest that Monte Carlo training is more robust to noisy rewards than TD-based methods.

Original: m
Revised: m

Original: =m =
Revised: =m =

Original: Q
Revised: Q

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Terminal states.
Revised: Terminal states.

Original: Table 2 shows that in environments where terminal states play a crucial role, Q MC is outperformed by TDbased methods.
Revised: Table 2 shows that in environments where terminal states play a crucial role, Q MC is outperformed by TDbased methods.

Original: This is due to the ﬁnite-horizon nature of Q MC .
Revised: This is due to the ﬁnite-horizon nature of Q MC .

Original: A terminal reward only contributes to a single update per episode, while in TD it contributes to every update in the episode.
Revised: A terminal reward only contributes to a single update per episode, while in TD it contributes to every update in the episode.

Original: If nonterminal rewards are present ( m = 2 ), Q MC approaches the TDbased algorithms, but still does not reach the performance of 20step Q. Difﬁculties with terminal states can partially explain poor performance of Q MC on some Atari games.
Revised: If nonterminal rewards are present ( m = 2 ), Q MC approaches the TDbased algorithms, but still does not reach the performance of 20step Q. Difﬁculties with terminal states can partially explain poor performance of Q MC on some Atari games.

Original: Delayed rewards.
Revised: Delayed rewards.

Original: Figure 3 (middle) shows that the performance of all algorithms declines even with moderate delays in the reward signal.
Revised: Figure 3 (middle) shows that the performance of all algorithms declines even with moderate delays in the reward signal.

Original: A delay of 2 steps, or approximately 0 .
Revised: A delay of 2 steps, or approximately 0 .

Original: 2 seconds of in-game time, already leads to a 8–12% relative drop in performance for Q MC and 20-step TD algorithms and a 30-40% drop for 5-step TD algorithms.
Revised: 2 seconds of in-game time, already leads to a 8–12% relative drop in performance for Q MC and 20-step TD algorithms and a 30–40% drop for 5-step TD algorithms.

Original: With a delay of 8 steps, or approximately 1 second, the performance of Q MC and 20-step TD algorithms drops by 30–70% and 5-step TD agents are essentially unable to survive until the end of an episode.
Revised: With a delay of 8 steps, or approximately 1 second, the performance of Q MC and 20-step TD algorithms drops by 30–70% and 5-step TD agents are essentially unable to survive until the end of an episode.

Original: With a delay of 32 steps, all algorithms degrade to a trivial score.
Revised: With a delay of 32 steps, all algorithms degrade to a trivial score.

Original: Interestingly, the performance of Q MC declines less rapidly than the performance of the other algorithms and Q MC consistently outperforms the other algorithms in the presence of delayed rewards.
Revised: Interestingly, the performance of Q MC declines less rapidly than the performance of the other algorithms and Q MC consistently outperforms the other algorithms in the presence of delayed rewards.

Original: Sparse rewards.
Revised: Sparse rewards.

Original: TD-based inﬁnite-horizon approaches should theoretically be effective at propagating distal rewards, and are therefore supposed to be advantageous in scenarios with sparse rewards.
Revised: TD-based inﬁnite-horizon approaches should theoretically be effective at propagating distal rewards, and are therefore supposed to be advantageous in scenarios with sparse rewards.

Original: The results on the Sparse and Very Sparse scenarios however, do not support this expectation (Figure 3 (right)): Q MC performs on par with 20-step Q , and noticeably better than 20step A3C and 5-step algorithms.
Revised: The results on the Sparse and Very Sparse scenarios however, do not support this expectation (Figure 3 (right)): Q MC performs on par with 20-step Q , and noticeably better than 20step A3C and 5-step algorithms.

Original: We believe the reason for the unexpectedly good performance of
Revised: We believe the reason for the unexpectedly good performance of

Original: Q MC is that Monte Carlo approaches are well suited for training perception systems, as discussed in more detail in Section 4.4.
Revised: Q MC is that Monte Carlo approaches are well suited for training perception systems, as discussed in more detail in Section 4.4.

Original: A video of a Q MC agent trained on the Very Sparse task is available at https://youtu.be/OJ1eBzW7cJ0 .
Revised: A video of a Q MC agent trained on the Very Sparse task is available at https://youtu.be/OJ1eBzW7cJ0 .

Original: Perceptual complexity.
Revised: Perceptual complexity.

Original: We test the algorithms on a series of environments of gradually increasing perceptual complexity.
Revised: We test the algorithms on a series of environments of gradually increasing perceptual complexity.

Original: The results are summarized in Figure 4.
Revised: The results are summarized in Figure 4.

Original: In simple gridworld environments, the locations of the agent and the health kits are given to the agent directly in the form of their coordinates (Grid Vec.) or a map (Grid Map).
Revised: In simple gridworld environments, the locations of the agent and the health kits are given to the agent directly in the form of their coordinates (Grid Vec.) or a map (Grid Map).

Original: In these scenarios, TD-based methods perform well.
Revised: In these scenarios, TD-based methods perform well.

Original: The Grid Vec.
Revised: The Grid Vec.

Original: task is successfully solved by all methods.
Revised: task is successfully solved by all methods.

Original: 5-step unrolling outperforms the 20-step versions and Q MC in both setups.
Revised: 5-step unrolling outperforms the 20-step versions and Q MC in both setups.

Original: However, the situation is completely different in the Basic and Multi-texture setups, in which the perceptual input is much more complex.
Revised: However, the situation is completely different in the Basic and Multi-texture setups, in which the perceptual input is much more complex.

Original: In the Basic setup, all methods perform roughly on par, but 5-step unrolling drops behind the other methods.
Revised: In the Basic setup, all methods perform roughly on par, but 5-step unrolling drops behind the other methods.

Original: In the Multi-texture setup, Q MC outperforms other algorithms.
Revised: In the Multi-texture setup, Q MC outperforms other algorithms.

Original: [Table]
Revised: [Table]

Original: To further analyze the effect of perception on DRL, we conduct an additional experiment where we separate the learning of perception and control.
Revised: To further analyze the effect of perception on DRL, we conduct an additional experiment where we separate the learning of perception and control.

Original: We ﬁrst train two perception systems on the Battle task by predicting Q -values under a ﬁxed policy with 20-step Q or Q MC .
Revised: We ﬁrst train two perception systems on the Battle task by predicting Q -values under a ﬁxed policy with 20-step Q or Q MC .

Original: We then re-initiailize the weights in the top two layers, freeze the weights in the rest of the the networks, and re-train the top two layers on the Battle task with 20-step Q or Q MC .
Revised: We then re-initiailize the weights in the top two layers, freeze the weights in the rest of the the networks, and re-train the top two layers on the Battle task with 20-step Q or Q MC .

Original: Further details are provided in the supplement.
Revised: Further details are provided in the supplement.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: Both Q and Q MC control reach higher score with a perception system trained with Q MC .
Revised: Both Q and Q MC control reach higher score with a perception system trained with Q MC .

Original: This supports the hypothesis that Monte Carlo training is efﬁcient at training deep perception systems from raw pixels.
Revised: This supports the hypothesis that Monte Carlo training is efﬁcient at training deep perception systems from raw pixels.

Original: 4. D ISCUSSION
Revised: 4. D ISCUSSION

Original: Temporal differencing methods are generally considered superior to Monte Carlo methods in reinforcement learning.
Revised: Temporal differencing methods are generally considered superior to Monte Carlo methods in reinforcement learning.

Original: This opinion is largely based on empirical evidence from domains such as gridworlds (Sutton, 1995), cart pole (Barto et al., 1983), and mountain car (Moore, 1990).
Revised: This opinion is largely based on empirical evidence from domains such as gridworlds (Sutton, 1995), cart pole (Barto et al., 1983), and mountain car (Moore, 1990).

Original: Our results agree: in gridworlds and on Atari games we ﬁnd that n-step Q learning outperforms Q MC .
Revised: Our results agree: in gridworlds and on Atari games we ﬁnd that n-step Q learning outperforms Q MC .

Original: We further ﬁnd, similar to the TD( λ ) experiments from the past (Sutton, 1988), that a mixture of MC and TD achieves best results in n-step Q and A3C.
Revised: We further ﬁnd, similar to the TD( λ ) experiments from the past (Sutton, 1988), that a mixture of MC and TD achieves best results in n-step Q and A3C.

Original: However, the situation changes in perceptually complex environments.
Revised: However, the situation changes in perceptually complex environments.

Original: In our experiments in immersive three-dimensional simulations, a ﬁnite-horizon MC method ( Q MC ) matches or outperforms
Revised: In our experiments in immersive three-dimensional simulations, a ﬁnite-horizon MC method ( Q MC ) matches or outperforms

Original: TD-based methods.
Revised: TD-based methods.

Original: Especially interesting are the results of the sparse reward experiments.
Revised: Especially interesting are the results of the sparse reward experiments.

Original: Sparse problems are supposed to be speciﬁcally challenging for ﬁnite-horizon Monte Carlo estimation: in our Very Sparse setting, average time between health kits is 44 time steps when a human is controlling the agent.
Revised: Sparse problems are supposed to be speciﬁcally challenging for ﬁnite-horizon Monte Carlo estimation: in our Very Sparse setting, average time between health kits is 44 time steps when a human is controlling the agent.

Original: This exceeds Q MC ’s ﬁnite prediction horizon of 32 steps, making it seemingly impossible for the algorithm to achieve nontrivial performance.
Revised: This exceeds Q MC ’s ﬁnite prediction horizon of 32 steps, making it seemingly impossible for the algorithm to achieve nontrivial performance.

Original: Yet Q MC is able to keep up with the results of the 20-step Q algorithm and clearly outperforms A3C.
Revised: Yet Q MC is able to keep up with the results of the 20-step Q algorithm and clearly outperforms A3C.

Original: What is the reason for this contrast between classic ﬁndings and our results?
Revised: What is the reason for this contrast between classic ﬁndings and our results?

Original: We believe that the key difference is in the complexity of perception in immersive three-dimensional environments, which was not present in gridworlds and other classic problems, and is only partially present in Atari games.
Revised: We believe that the key difference is in the complexity of perception in immersive three-dimensional environments, which was not present in gridworlds and other classic problems, and is only partially present in Atari games.

Original: In immersive simulation, the agent’s observation is a high-dimensional image that represents a partial view of a large (mostly hidden) three-dimensional environment.
Revised: In immersive simulation, the agent’s observation is a high-dimensional image that represents a partial view of a large (mostly hidden) three-dimensional environment.

Original: The dimensionality of the state space is essentially inﬁnite: the underlying environment is speciﬁed by continuous surfaces in three-dimensional space.
Revised: The dimensionality of the state space is essentially inﬁnite: the underlying environment is speciﬁed by continuous surfaces in three-dimensional space.

Original: Memorizing all possible states is easy and routine in gridworlds and is also possible in some Atari games (Blundell et al., 2016), but is not feasible in immersive three-dimensional simulations.
Revised: Memorizing all possible states is easy and routine in gridworlds and is also possible in some Atari games (Blundell et al., 2016), but is not feasible in immersive three-dimensional simulations.

Original: Therefore, in order to successfully operate in such simulations, the agent has to learn to extract useful representations from the observations it receives.
Revised: Therefore, in order to successfully operate in such simulations, the agent has to learn to extract useful representations from the observations it receives.

Original: Encoding a meaningful representation from rich perceptual input is where Monte Carlo methods are at an advantage due to the reliability of their training signals.
Revised: Encoding a meaningful representation from rich perceptual input is where Monte Carlo methods are at an advantage due to the reliability of their training signals.

Original: Monte Carlo methods train on ground-truth targets, not “guess from a guess”, as TD methods do (Sutton & Barto, 2017).
Revised: Monte Carlo methods train on ground-truth targets, not “guess from a guess”, as TD methods do (Sutton & Barto, 2017).

Original: These intuitions are supported by our experiments.
Revised: These intuitions are supported by our experiments.

Original: Figure 4 shows that increasing the perceptual difﬁculty of the health gathering scenario hurts the performance of Q MC less than it does the TDbased approaches.
Revised: Figure 4 shows that increasing the perceptual difﬁculty of the health gathering scenario hurts the performance of Q MC less than it does the TDbased approaches.

Original: Table 3 shows that Q MC is able to learn a better perception network than 20-step
Revised: Table 3 shows that Q MC is able to learn a better perception network than 20-step

Original: Q .
Revised: Q .

Original: In Figure 3, 20-step TD algorithms perform better than their 5-step counterparts in all tested scenarios.
Revised: In Figure 3, 20-step TD algorithms perform better than their 5-step counterparts in all tested scenarios.

Original: Longer rollouts bring TD closer to MC, in agreement with our hypothesis.
Revised: Longer rollouts bring TD closer to MC, in agreement with our hypothesis.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: For the past 30 years TD-based methods have dominated the ﬁeld of reinforcement learning.
Revised: For the past 30 years TD-based methods have dominated the ﬁeld of reinforcement learning.

Original: Our experiments on a range of complex tasks in perceptually challenging environments show that in deep reinforcement learning ﬁnite-horizon MC can be a viable alternative to TD.
Revised: Our experiments on a range of complex tasks in perceptually challenging environments show that in deep reinforcement learning ﬁnite-horizon MC can be a viable alternative to TD.

Original: We ﬁnd
Revised: We ﬁnd

Original: that while
Revised: that while

Original: TD is at advantage in tasks with simple perception, long planning horizons, or terminal rewards,
Revised: TD is at advantage in tasks with simple perception, long planning horizons, or terminal rewards,

Original: MC training is more robust to noisy rewards, effective for training perception systems from raw sensory inputs, and surprisingly successful at dealing with delayed and sparse rewards.
Revised: MC training is more robust to noisy rewards, effective for training perception systems from raw sensory inputs, and surprisingly successful at dealing with delayed and sparse rewards.

Original: Thus, a key challenge that can be derived from our results is to ﬁnd ways on how to combine the advantages of noise-free supervised MC learning with those of TD.
Revised: Thus, a key challenge that can be derived from our results is to ﬁnd ways on how to combine the advantages of noise-free supervised MC learning with those of TD.

Original: We hope that our results will contribute to a set of best practices for deep reinforcement learning that are consistent with the empirical reality of modern application domains.
Revised: We hope that our results will contribute to a set of best practices for deep reinforcement learning that are consistent with the empirical reality of modern application domains.

Original: S UPPLEMENTARY M ATERIAL S F URTHER RESULTS
Revised: S UPPLEMENTARY M ATERIAL S F URTHER RESULTS

Original: Effect of the rollout length and the prediction horizon In Table 5 of the main paper we have shown that the performance of n-step Q decreases for rollouts larger than 20 .
Revised: Effect of the rollout length and the prediction horizon In Table 5 of the main paper we have shown that the performance of n-step Q decreases for rollouts larger than 20 .

Original: For the Q MC algorithm a similar phenomenon is observed for large horizons, as shown in Table S1.
Revised: For the Q MC algorithm a similar phenomenon is observed for large horizons, as shown in Table S1.

Original: The performance is decreasing for a horizon larger than 32 .
Revised: The performance is decreasing for a horizon larger than 32 .

Original: In both cases, the decrease is likely caused by the high variance of large sums of future rewards.
Revised: In both cases, the decrease is likely caused by the high variance of large sums of future rewards.

Original: The high variance in reward sums increases the variance of the gradients and leads to higher noise when training the value predictions.
Revised: The high variance in reward sums increases the variance of the gradients and leads to higher noise when training the value predictions.

Original: This hinders the action selection process, which relies on ﬁne differences between values of different actions.
Revised: This hinders the action selection process, which relies on ﬁne differences between values of different actions.

Original: Difference between asynchronous n-step Q and Q MC .
Revised: Difference between asynchronous n-step Q and Q MC .

Original: As mentioned in the main paper, apart from the different targets to learn the Q -function there is another difference between the n-step Q and
Revised: As mentioned in the main paper, apart from the different targets to learn the Q -function there is another difference between the n-step Q and

Original: Q MC algorithms.
Revised: Q MC algorithms.

Original: It is caused by the usage of multiple unrolling values in the n-step Q algorithms.
Revised: It is caused by the usage of multiple unrolling values in the n-step Q algorithms.

Original: In n-step Q instead of only using the n-step rollout, multiple values are used within every batch (every value from 1 to n (Mnih et al., 2016)).
Revised: In n-step Q instead of only using the n-step rollout, multiple values are used within every batch (every value from 1 to n (Mnih et al., 2016)).

Original: This results in an increased performance and stability of the n-step Q algorithm.
Revised: This results in an increased performance and stability of the n-step Q algorithm.

Original: It is not directly applicable to Q MC since different unrolling values result in different ﬁnite horizons.
Revised: It is not directly applicable to Q MC since different unrolling values result in different ﬁnite horizons.

Original: Instead in Q MC multiple Q -function heads exist to predict the different ﬁnite horizons (Dosovitskiy & Koltun, 2017).
Revised: Instead in Q MC multiple Q -function heads exist to predict the different ﬁnite horizons (Dosovitskiy & Koltun, 2017).

Original: The difference between the trivial implementation and the multiple unrolling modiﬁcations are shown in Table S1.
Revised: The difference between the trivial implementation and the multiple unrolling modiﬁcations are shown in Table S1.

Original: There are no further differences between the two algorithms.
Revised: There are no further differences between the two algorithms.

Original: They use the same architecture and asynchronous training.
Revised: They use the same architecture and asynchronous training.

Original: Both even perform best under the same hyperparameters like the learning rate.
Revised: Both even perform best under the same hyperparameters like the learning rate.

Original: [Table]
Revised: [Table]

Original: Separate training of perception and control In order to perform the perception freezing experiments we ﬁrst train two perception systems on the Battle task with 20-step Q and Q MC formillion steps ( 1 / 3 of the usual training) by predicting Q values under a ﬁxed policy (we tried using a fully trained Q MC or 20-step Q policy).
Revised: Separate training of perception and control In order to perform the perception freezing experiments we ﬁrst train two perception systems on the Battle task with 20-step Q and Q MC formillion steps ( 1 / 3 of the usual training) by predicting Q values under a ﬁxed policy (we tried using a fully trained Q MC or 20-step Q policy).

Original: Thereafter we freeze the perception and the measurements part of the network.
Revised: Thereafter we freeze the perception and the measurements part of the network.

Original: (The full architecture of the perception and the measurements are shown in Table S3).
Revised: (The full architecture of the perception and the measurements are shown in Table S3).

Original: We then reinitialize the remaining layers and retrain the networks with the frozen perception with Q MC and 20-step Q , both using each of the two available perceptions (for 40 million steps).
Revised: We then reinitialize the remaining layers and retrain the networks with the frozen perception with Q MC and 20-step Q , both using each of the two available perceptions (for 40 million steps).

Original: [Table]
Revised: [Table]

Original: The full results are shown in Table 3.
Revised: The full results are shown in Table 3.

Original: Both Q and Q MC are able to reach higher score with a
Revised: Both Q and Q MC are able to reach higher score with a

Original: Q MC perception, on both of the used initial policies.
Revised: Q MC perception, on both of the used initial policies.

Original: The results in the main paper correspond to perception systems trained under the Q MC policy.
Revised: The results in the main paper correspond to perception systems trained under the Q MC policy.

Original: S A DDITIONAL ALGORITHM AND ENVIRONMENT DETAILS
Revised: S A DDITIONAL ALGORITHM AND ENVIRONMENT DETAILS

Original: Batch size for small rollouts In algorithms with asynchronous n-step TD targets the batch size is usually equal to the unrolling length n.
Revised: Batch size for small rollouts In algorithms with asynchronous n-step TD targets the batch size is usually equal to the unrolling length n.

Original: However decreasing the batch size in A3C could also effect the performance of the policy gradient of the A3C loss.
Revised: However decreasing the batch size in A3C could also effect the performance of the policy gradient of the A3C loss.

Original: To make sure that we only measure the effect of different n-step TD targets and do not alternate the policy gradient part we keep the batch size at the constant value of 20 (for all rollouts smaller than 20 ).
Revised: To make sure that we only measure the effect of different n-step TD targets and do not alternate the policy gradient part we keep the batch size at the constant value of 20 (for all rollouts smaller than 20 ).

Original: This is realized by using multiple n-step rollouts within one batch (e.g. for a 5-step rollout the batch consists of 4 rollouts).
Revised: This is realized by using multiple n-step rollouts within one batch (e.g. for a 5-step rollout the batch consists of 4 rollouts).

Original: Overall those batches lead to improved performance of A3C. For n-step Q using the constant batch size of 20 results in similar performance and signiﬁcantly reduced the execution time.
Revised: Overall those batches lead to improved performance of A3C. For n-step Q using the constant batch size of 20 results in similar performance and signiﬁcantly reduced the execution time.

Original: Therefore we used those batches for both A3C and n-step Q in our experiments.
Revised: Therefore we used those batches for both A3C and n-step Q in our experiments.

Original: Network details
Revised: Network details

Original: In each experiment we used the same network architecture for all algorithms.
Revised: In each experiment we used the same network architecture for all algorithms.

Original: For tasks with visual input – in ViZDoom and ALE – we used a convolutional network with architecture similar to Mnih et al.
Revised: For tasks with visual input – in ViZDoom and ALE – we used a convolutional network with architecture similar to Mnih et al.

Original: For all experiments in the ViZDoom environment, in addition to the image the networks got a vector of measurements as input: agent’s health level and current time step for Health gathering and Navigation, and agent’s health, ammo and frags for Battle.
Revised: For all experiments in the ViZDoom environment, in addition to the image the networks got a vector of measurements as input: agent’s health level and current time step for Health gathering and Navigation, and agent’s health, ammo and frags for Battle.

Original: For Q MC and n-step Q we used the dueling architecture (Wang et al., 2016), splitting value prediction into an action independent expectation E ( s t , θ ) and an action dependent part for the advantage of using a speciﬁc action A ( s t , a , θ ) .
Revised: For Q MC and n-step Q we used the dueling architecture (Wang et al., 2016), splitting value prediction into an action independent expectation E ( s t , θ ) and an action dependent part for the advantage of using a speciﬁc action A ( s t , a , θ ) .

Original: For l actions, the value prediction emitted by the network is computed as:
Revised: For l actions, the value prediction emitted by the network is computed as:

Original: [Equation]
Revised: [Equation]

Original: The architecture of the Q MC Network is shown in Table S3.
Revised: The architecture of the Q MC Network is shown in Table S3.

Original: Q MC is predicting the Q value for multiple ﬁnite horizons at once: 1 , 2 , 4 , 8 , 16 and 32 .
Revised: Q MC is predicting the Q value for multiple ﬁnite horizons at once: 1 , 2 , 4 , 8 , 16 and 32 .

Original: Predictions for all horizons are emitted at once.
Revised: Predictions for all horizons are emitted at once.

Original: Therefore, for l actions, the network has 6 outputs for the expectation values and 6 × l outputs for the action advantages.
Revised: Therefore, for l actions, the network has 6 outputs for the expectation values and 6 × l outputs for the action advantages.

Original: We used greedy action selection according to an objective function which is a linear combination of predictions at different horizons, same as in DFP (Dosovitskiy & Koltun, 2017):
Revised: We used greedy action selection according to an objective function which is a linear combination of predictions at different horizons, same as in DFP (Dosovitskiy & Koltun, 2017):

Original: [Equation]
Revised: [Equation]

Original: The network for n-step Q was identical, except that instead of 6 predictions, a single value function was predicted for each action.
Revised: The network for n-step Q was identical, except that instead of 6 predictions, a single value function was predicted for each action.

Original: The A3C architecture was also identical, except that the network was not split in the last hidden layer like it was for the dueling networks.
Revised: The A3C architecture was also identical, except that the network was not split in the last hidden layer like it was for the dueling networks.

Original: Both the policy and the value output shared the same last hidden layer as in Mnih et al.
Revised: Both the policy and the value output shared the same last hidden layer as in Mnih et al.

Original: The network we are using for A3C is larger than that used by Mnih et al.
Revised: The network we are using for A3C is larger than that used by Mnih et al.

Original: We found that the larger network matches or exceeds the performance of the smaller network used by Mnih et al.
Revised: We found that the larger network matches or exceeds the performance of the smaller network used by Mnih et al.

Original: (2016) on our tasks.
Revised: (2016) on our tasks.

Original: For both gridworlds we used fully connected networks.
Revised: For both gridworlds we used fully connected networks.

Original: For all results reported in the paper the three algorithms used three hidden fully connected layers with size of 512 .
Revised: For all results reported in the paper the three algorithms used three hidden fully connected layers with size of 512 .

Original: [Table]
Revised: [Table]

Original: Training and evaluation details We found that for each of the three asynchronous algorithms the learning rate of 7 × 10 − 4 leads to the best result in most of the tested environments.
Revised: Training and evaluation details We found that for each of the three asynchronous algorithms the learning rate of 7 × 10 − 4 leads to the best result in most of the tested environments.

Original: Further we found that, in general, ViZdoom scenarios are less sensitive to learning rate changes than different
Revised: Further we found that, in general, ViZdoom scenarios are less sensitive to learning rate changes than different

Original: Atari games.
Revised: Atari games.

Original: We decreased the learning rate linearly to zero over the training procedure.
Revised: We decreased the learning rate linearly to zero over the training procedure.

Original: As the optimizer we used shared RMSProp with the same parameters as in Mnih et al. (2016).
Revised: As the optimizer we used shared RMSProp with the same parameters as in Mnih et al. (2016).

Original: In all experiments we used a total of 60 million steps for training.
Revised: In all experiments we used a total of 60 million steps for training.

Original: This means all actor threads together processed 60 million steps.
Revised: This means all actor threads together processed 60 million steps.

Original: We used frame skip of 4 , therefore 60 million frame-skipped steps correspond to 240 million non-frame-skipped environment steps.
Revised: We used frame skip of 4 , therefore 60 million frame-skipped steps correspond to 240 million non-frame-skipped environment steps.

Original: For Q MC each asynchronous agent performed a parameter update every 20 steps with a batch size of 20 .
Revised: For Q MC each asynchronous agent performed a parameter update every 20 steps with a batch size of 20 .

Original: Each time the most recent 20 frames with available value targets were used.
Revised: Each time the most recent 20 frames with available value targets were used.

Original: For A3C and n-step Q the training intervals and batch sizes were equal to the rollout lengths.
Revised: For A3C and n-step Q the training intervals and batch sizes were equal to the rollout lengths.

Original: Every 2 .
Revised: Every 2 .

Original: 5 million steps we evaluated the network over 500 episodes for Vizdoom Environments and over 200 episodes for Atari games.
Revised: 5 million steps we evaluated the network over 500 episodes for Vizdoom Environments and over 200 episodes for Atari games.

Original: For one training run the best result out of all 24 evaluations was considered as its ﬁnal score.
Revised: For one training run the best result out of all 24 evaluations was considered as its ﬁnal score.

Original: For each experiment three runs were performed for each algorithm.
Revised: For each experiment three runs were performed for each algorithm.

Original: The average of the three run scores was considered as the ﬁnal performance of that algorithm on the task.
Revised: The average of the three run scores was considered as the ﬁnal performance of that algorithm on the task.

Original: Additional environment details The Navigation scenario is identical to the “Health Gathering
Revised: Additional environment details The Navigation scenario is identical to the “Health Gathering

Original: Supreme” scenario included in the ViZDoom environment.
Revised: Supreme” scenario included in the ViZDoom environment.

Original: The aim of the agent is to navigate a maze, collect health kits and avoid vials with poison.
Revised: The aim of the agent is to navigate a maze, collect health kits and avoid vials with poison.

Original: A map of the maze is shown in Figure S2.
Revised: A map of the maze is shown in Figure S2.

Original: All other Health gathering scenarios are set up in the same labyrinth, but differ in the presence and the number of objects in the maze: no poison vials, and a different number of health kits depending on the variant of the Health gathering scenario.
Revised: All other Health gathering scenarios are set up in the same labyrinth, but differ in the presence and the number of objects in the maze: no poison vials, and a different number of health kits depending on the variant of the Health gathering scenario.

Original: In each Health gathering scenario a constant number of health kits is present on the map at any given point in time.
Revised: In each Health gathering scenario a constant number of health kits is present on the map at any given point in time.

Original: Once a health kit is gathered, another one is created at a random location in the maze.
Revised: Once a health kit is gathered, another one is created at a random location in the maze.

Original: To make sparse health gathering map results comparable to each other we kept the health d that the agent looses every 8 time steps to be proportional to the density of health kits on the map: d ∝ √ #health kits .
Revised: To make sparse health gathering map results comparable to each other we kept the health d that the agent looses every 8 time steps to be proportional to the density of health kits on the map: d ∝ √ #health kits .

Original: In the Battle scenario we used the same reward as in Dosovitskiy & Koltun (2017).
Revised: In the Battle scenario we used the same reward as in Dosovitskiy & Koltun (2017).

Original: It is a weighted sum of changes in measurements: r = f
Revised: It is a weighted sum of changes in measurements: r = f

Original: + ∆ h/ 60 + ∆ a/ 20 where f are the amount of eliminated monsters, ∆ h the change in health and ∆ a the change in ammunition.
Revised: + ∆ h/ 60 + ∆ a/ 20 where f are the amount of eliminated monsters, ∆ h the change in health and ∆ a the change in ammunition.

Original: For Basic health gathering we either used a binary reward r ∈ { 0 , 1 } , or the change in health: r = ∆ h/ 30 .
Revised: For Basic health gathering we either used a binary reward r ∈ { 0 , 1 } , or the change in health: r = ∆ h/ 30 .

Original: [Table]
Revised: [Table]