Original: Disentangling Generalization in Reinforcement Learning
Revised: Disentangling Generalization in Reinforcement Learning

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Generalization in Reinforcement Learning (RL) is usually measured according to concepts from supervised learning.
Revised: Generalization in Reinforcement Learning (RL) is usually measured according to concepts from supervised learning.

Original: Unlike a supervised learning model however, an RL agent must generalize across states, actions and observations from limited reward-based feedback.
Revised: Unlike a supervised learning model however, an RL agent must generalize across states, actions and observations from limited reward-based feedback.

Original: We propose to measure an RL agent’s capacity to generalize by evaluating it in a contextual decision process that combines a tabular environment with observations from a supervised learning dataset.
Revised: We propose to measure an RL agent’s capacity to generalize by evaluating it in a contextual decision process that combines a tabular environment with observations from a supervised learning dataset.

Original: The resulting environment, while simple, necessitates function approximation for state abstraction and provides ground-truth labels for optimal policies and value functions.
Revised: The resulting environment, while simple, necessitates function approximation for state abstraction and provides ground-truth labels for optimal policies and value functions.

Original: The ground truth labels provided by our environment enable us to characterize generalization in RL across diﬀerent axes: state-space, observation-space and action-space.
Revised: The ground truth labels provided by our environment enable us to characterize generalization in RL across diﬀerent axes: state-space, observation-space and action-space.

Original: Putting this method to work, we combine the MNIST dataset with various gridworld environments to rigorously evaluate generalization of DQN and QR-DQN in state, observation and action spaces for both online and oﬄine learning.
Revised: Putting this method to work, we combine the MNIST dataset with various gridworld environments to rigorously evaluate generalization of DQN and QR-DQN in state, observation and action spaces for both online and oﬄine learning.

Original: Contrary to previous reports about common regularization methods, we ﬁnd that dropout does not improve observation generalization.
Revised: Contrary to previous reports about common regularization methods, we ﬁnd that dropout does not improve observation generalization.

Original: We ﬁnd, however, that dropout improves action generalization.
Revised: We ﬁnd, however, that dropout improves action generalization.

Original: Our results also corroborate recent ﬁndings that QR-DQN is able to generalize to new observations better than DQN in the oﬄine setting.
Revised: Our results also corroborate recent ﬁndings that QR-DQN is able to generalize to new observations better than DQN in the oﬄine setting.

Original: This success does not extend to state generalization, where DQN is able to generalize better than
Revised: This success does not extend to state generalization, where DQN is able to generalize better than

Original: QR-DQN.
Revised: QR-DQN.

Original: These ﬁndings demonstrate the need for careful consideration of generalization in RL, and we hope that this line of research will continue to shed light on generalization claims in the literature.
Revised: These ﬁndings demonstrate the need for careful consideration of generalization in RL, and we hope that this line of research will continue to shed light on generalization claims in the literature.

Original:  Generalization in Reinforcement Learning
Revised:  Generalization in Reinforcement Learning

Original: A Reinforcement Learning (RL) agent perpetually ﬁnds itself in novel states of an environment.
Revised: A Reinforcement Learning (RL) agent perpetually ﬁnds itself in novel states of an environment.

Original: To act intelligently, the agent must generalize its previous experience to new situations.
Revised: To act intelligently, the agent must generalize its previous experience to new situations.

Original: Function approximation helps distill this previous experience into the agent’s learnable parameters, which allows previous experience to be leveraged with new state inputs (Boyan and Moore, 1994; Sutton, 1995).
Revised: Function approximation helps distill this previous experience into the agent’s learnable parameters, which allows previous experience to be leveraged with new state inputs (Boyan and Moore, 1994; Sutton, 1995).

Original: While there is a growing literature of new methods for improving generalization of deep RL algorithms, principled and quantitative methods for evaluating generalization remain lacking.
Revised: While there is a growing literature of new methods for improving generalization of deep RL algorithms, principled and quantitative methods for evaluating generalization remain lacking.

Original: This is due in part to the complexity of the MDP problem formulation and the diﬃculty of disentangling generalization from the performance of RL algorithms in terms of achieving higher expected cumulative reward, i.e. return.
Revised: This is due in part to the complexity of the MDP problem formulation and the diﬃculty of disentangling generalization from the performance of RL algorithms in terms of achieving higher expected cumulative reward, i.e. return.

Original: One common notion of generalization in RL evaluates an agent’s capabilities by checking if it can achieve similar performance in an environment that is similar to, but not exactly the same as, the environment in which it was trained (Whiteson et al., 2011).
Revised: One common notion of generalization in RL evaluates an agent’s capabilities by checking if it can achieve similar performance in an environment that is similar to, but not exactly the same as, the environment in which it was trained (Whiteson et al., 2011).

Original: This can be accomplished through randomization, i.e. by randomizing the parameters underlying the environment, such as wind velocity in helicopter hovering or the mass of objects in a simulator, thereby changing the transition dynamics (Peng et al., 2017).
Revised: This can be accomplished through randomization, i.e. by randomizing the parameters underlying the environment, such as wind velocity in helicopter hovering or the mass of objects in a simulator, thereby changing the transition dynamics (Peng et al., 2017).

Original: Generalization in this sense draws parallels to supervised learning, where classiﬁers are often trained on a ﬁxed dataset, and evaluated on a separate testing dataset under the I.I.D. assumption.
Revised: Generalization in this sense draws parallels to supervised learning, where classiﬁers are often trained on a ﬁxed dataset, and evaluated on a separate testing dataset under the I.I.D. assumption.

Original: The agent is said to generalize well if the diﬀerence between training error and testing error is small.
Revised: The agent is said to generalize well if the diﬀerence between training error and testing error is small.

Original: While these supervised learning concepts are relevant to RL, there are two problems with taking a cross-environment approach to evaluating RL generalization.
Revised: While these supervised learning concepts are relevant to RL [Table] there are two problems with taking a cross-environment approach to evaluating RL generalization.

Original: First, this paradigm does not disentangle the various aspects of generalization that are required for an RL agent to succeed in its task, whether it be the value estimates or the policy, and whether they are robust to variations in states, observations or actions.
Revised: First, this paradigm does not disentangle the various aspects of generalization that are required for an RL agent to succeed in its task, whether it be the value estimates or the policy, and whether they are robust to variations in states, observations or actions.

Original: This is the question of “what” performance criterion should be measured when we discuss generalization.
Revised: This is the question of “what” performance criterion should be measured when we discuss generalization.

Original: In RL, we have function approximators for many quantities: state-transition, reward, state-value, actionvalue and policy.
Revised: In RL, we have function approximators for many quantities: state-transition, reward, state-value, actionvalue and policy.

Original: While generalization of state-value is similar to regression, generalization with quantities related to action, such as policy or action-value, do not have supervised learning analogues and hence require separate consideration.
Revised: While generalization of state-value is similar to regression, generalization with quantities related to action, such as policy or action-value, do not have supervised learning analoguesand hence require (cid:58) . [Table] This [Table]

Original: Second, the paradigm follows practice in supervised learning to impose a strict separation of test and train environments.
Revised: is [Table] because (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]

Revised: have

Revised: [Table]

Revised: as

Revised: [Table]

Revised: many

Revised: [Table]

Original:  This practice eﬀectively focuses on transfer performance across environments but fails to evaluate the agent’s ability to generalize within a single environment.
Revised: necessitates [Table] Second, the paradigm follows practice in supervised learning to impose a strict separation of test and train environments. This practice eﬀectively focuses on transfer performance across environments but fails to evaluate the agent’s ability to generalize within a single environment.

Original: I.e. it fails to answer the question of “how” to measure generalization under a performance criterion.
Revised: I [Table] e. it fails to answer the question of “how” to measure generalization under a performance criterion.

Original: Unlike supervised learning, the agent’s state distribution changes during learning because the policy changes.
Revised: Unlike supervised learning, the agent’s state distribution changes during learning because the policy changes.

Original: Even where we have randomly generated training and testing sets of environments, the environments’ complex dynamics do not admit ground-truth labels to determine optimal actions or values for comparison and evaluation.
Revised: Even where we have randomly generated training and testing sets of environments, the environments’ complex dynamics do not admit ground-truth labels to determine optimal actions or values for comparison and evaluation.

Original: These environments only allow an agent’s generalization capabilities to be measured in terms of Monte-Carlo rollouts of the learned policy, leaving us restricted to the single performance criterion of return and preventing us from speciﬁcally measuring important nuanced diﬀerences in the agent’s ability to generalize.
Revised: These environments only allow an agent’s generalization capabilities to be measured in terms of Monte-Carlo rollouts of the learned policy, leaving us restricted to the single performance criterion of return and preventing us from speciﬁcally measuring important nuanced diﬀerences in the agent’s ability to generalize.

Original: In addition, because the upper bound on return is unknown to the researcher, informed judgments about the quality of the policy is very diﬃcult to make.
Revised: In addition, because the upper bound on return is unknown to the researcher, informed judgments about the quality of the policy is very diﬃcult to make.

Original: Within the RL-oriented literature on generalization, there are two distinct categories of research.
Revised: Within the RL-oriented literature on generalization, there are two distinct categories of research.

Original: The ﬁrst proposes environments and methodologies for measuring RL generalization.
Revised: The ﬁrst proposes environments and methodologies for measuring RL generalization.

Original: Examples of this approach are ProcGen/CoinRun (Cobbe et al., 2020; 2018), randomizedreward CartPole (Zhang et al., 2018a), the GridWorld maze (Zhang et al., 2018b), observation projection  (Song et al., 2019) and the hierarchy of state generalization (Witty et al., 2018).
Revised: Examples of this approach are ProcGen/CoinRun (Cobbe et al., 2020; ? ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (Cobbe et al., 2020; 2019), randomized-reward CartPole (Zhang et al., 2018a), the GridWorld maze (Zhang et al., 2018b), observation projection ( ? ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (Song et al., 2020) and the hierarchy of state generalization (Witty et al., 2018).

Original: These eﬀorts are aimed at the second issue raised above, i.e. “how” generalization should be measured in RL.
Revised: These eﬀorts are aimed at the second issue raised above, i.e. “how” generalization should be measured in RL.

Original: Underlying these approaches is specifying how to split the environment into testing and training scenarios.
Revised: Underlying these approaches is specifying how to split the environment into testing and training scenarios.

Original: Early work by Zhang et al.
Revised: Early work by Zhang et al.

Original: (2018a) proposes using separate seeds.
Revised: (2018a) proposes using separate seeds.

Original: This, however, does not ensure that the states encountered by the agent are truly separate.
Revised: This, however, does not ensure that the states encountered by the agent are truly separate.

Original: Other works, such as those by Cobbe et al. 
Revised: Other works, such as those by Cobbe et al. ; Elsayed et al.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Cobbe et al.

Original: (2020; 2018); Elsayed et al.
Revised: (2020; 2019); Elsayed et al.

Original: (2020), procedurally generate separate environments for testing and training.
Revised: (2020), procedurally generate separate environments for testing and training.

Original: When generalization is measured on truly separate testing and training environments, we are able to determine whether an agent’s policy is generalizing from one environment to another.
Revised: When generalization is measured on truly separate testing and training environments, we are able to determine whether an agent’s policy is generalizing from one environment to another.

Original: Again, this formulation does not allow us to study generalization within a single environment, nor does it allow us to measure generalization of the value functions.
Revised: Again, this formulation does not allow us to study generalization within a single environment, nor does it allow us to measure generalization of the value functions.

Original: The second category proposes or investigates RL methods that improve generalization.
Revised: The second category proposes or investigates RL methods that improve generalization.

Original: These include regularization experiments in Atari (Farebrother et al., 2018) and continuous control  (Liu et al., 2019), contrastive similarity embeddings (Agarwal et al., 2021) and bisimulation metrics  (Zhang et al., 2020).
Revised: These include regularization experiments in Atari (Farebrother et al., 2018) and continuous control ( ? ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (Liu et al., 2021), contrastive similarity embeddings (Agarwal et al., 2021) and bisimulation metrics ( ? ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (Zhang et al., 2021).

Revised: There is also the hypothesis that better, and hence more generalizable, representations arise from auxiliary tasks ( ? )

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: There is also the hypothesis that better, and hence more generalizable, representations arise from auxiliary tasks (Jaderberg et al., 2016), which is also suspected to be the reason for the success of distributional RL  (Dabney et al., 2017; Bellemare et al., 2017) and has recently been investigated in the oﬄine setting (Agarwal et al., 2019b).
Revised: (Jaderberg et al., 2017), which is also suspected to be the reason for the success of distributional RL ( ? Bellemare et al., 2017) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (Dabney et al., 2018; Bellemare et al., 2017) and has recently been investigated in the oﬄine setting ( ? ).

Original:  Previous work closest to ours is using Contextual Decision Processes (CDPs) as  a problem class for theoretical analysis of RL algorithms that use function approximation (Du et al., 2019; Jiang et al., 2017; Dann et al., 2018).
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (Agarwal et al., 2020) (cid:58) . (cid:58) Previous work closest to ours is using Contextual Decision Processes (CDPs) as (cid:58)(cid:58) To (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) understand (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generalization (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) RL, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58) use (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Contextual (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Decision (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Process (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (CDP) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) framework, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58) is (cid:58) a problem class for theoretical analysis of RL algorithms that use function approximation (Du et al., 2019; Jiang et al., 2017; Dann et al., 2018).

Original: The CDP problem formulation renders the states unobservable, but allows the agent to view observations that contain enough information to recover the state.
Revised: The CDP problem formulation renders the states unobservable, but allows the agent to view observations that contain enough information to recover the state.

Original: This formulation enables the study of function approximation as applied to RL and has signiﬁcant implications for RL in general.
Revised: This formulation enables the study of function approximation as applied to RL and has signiﬁcant implications for RL in general.

Original: For example, it helps to  show that there exist algorithms with exponentially more eﬃcient exploration than Q-learning (Du et al., 2019), a result suggesting that RL algorithms should be designed to take advantage of function approximation and its generalization abilities, rather than naively extending tabular algorithms with function approximation.
Revised: For example, it helps to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Du et al. (2019) show that there exist algorithms with exponentially more eﬃcient explo- ration than Q-learning(Du et al., 2019), a result suggesting that RL algorithms should be designed to take advantage of function approximation and its generalization abilities, rather than naively extending tabular algorithms with function approximation.

Original: However, no existing work has leveraged CDPs empirically, despite that it connects supervised learning with RL. 
Revised: However, no existing work has leveraged CDPs empirically, despite that it connects supervised learning with RL. The MNIST gridworld [Table]

Original: The MNIST gridworld environment used by Lee et al.
Revised: 2D [Table] MNIST [Table]

Original: (2018) may be considered a CDP, but is not recognized as such.
Revised: maze environment used by ? (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Lee et al. (2019) may be considered a CDP, [Table]

Revised: simple

Revised: [Table]

Revised: CDP

Original: The work by Song et al.
Revised: [Table] where [Table] the

Revised: [Table]

Revised: observations

Revised: [Table]

Revised: are

Revised: [Table]

Revised: (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) equivalent (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state, but is not recognized as such.

Revised: The work by ?

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Song et al.

Original: (2019) proposed projecting the state of simple control environment and varying this projection between training and testing sets.
Revised: (2020) proposed projecting the state of simple control environment and varying this projection between training and testing sets.

Original: These two works have similar goals to ours.
Revised: These two works have similar goals to ours.

Original: However, the ﬁrst work does not investigate generalization and neither makes use of the ground-truth values to probe generalization rigorously, such as in the label-corruption experiment by (Zhang et al., 2017) that we will extend to RL.
Revised: However, the ﬁrst work does not investigate generalization and neither makes use of the ground-truth values to probe generalization rigorously, such as in the label-corruption experiment by (Zhang et al., 2017) that we will extend to RL . (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Our [Table]

Revised: work (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) uniquely (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) analogous (cid:58)(cid:58) to

Original: A Problem Formulation For Evaluating Generalization in RL
Revised: [Table] should [Table] be

Revised: [Table]

Revised: measured.

Revised: [Table]

Revised: In

Original:  To make generalization in RL concrete, we ﬁrst make clear the Markov Decision Process (MDP) problem formulation that underpins task speciﬁcation in RL  (White, 2016; Lattimore and Szepesvári, 2020).
Revised: [Table]  A Problem Formulation For Evaluating Generalization in RL To make generalization in RL concrete, we ﬁrst make clear the Markov Decision Process (MDP) problem formulation that underpins task speciﬁcation in RL ( ? Lattimore and Szepesvári, 2020) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (White, 2017; Lattimore and Szepesvári, 2020).

Original: An MDP M is deﬁned by the tuple ( S , A , r, T, µ 0 , γ ) , where A denotes the action space, S is the state space, 
Revised: An MDP M is deﬁned by the tuple ( S , A , r, T, µ 0 , γ ) , where A denotes the action space, S is the state space, r : S ×A → R is the reward function that maps a state and an action to a reward,

Original: r : S ×A → R is the reward function that maps a state and an action to a reward, T : S×A×S → [0 , 1] is the state transition function, µ 0 is the initial state distribution and γ ∈ [0 , 1] is the discount factor.
Revised: T : S × A × S → [0 , 1] is the state transition function, µ 0 is the initial state distribution and γ ∈ [0 , 1] is the discount factor.

Original: A policy that interacts with the MDP is deﬁned by a mapping from states to a distribution over actions π : S → ∆( A ) , which includes deterministic policies that concentrate the probability mass on a single action.
Revised: A policy that interacts with the MDP is deﬁned by a mapping from states to a distribution over actions π : S → ∆( A ) , which includes deterministic policies that concentrate the probability mass on a single action.

Original: The discounted state visitation distribution for following the policy π , starting in state s 0 and taking action a 0 , is deﬁned as d πs 0 ,a 0 ( s ) =
Revised: The discounted state visitation distribution for following the policy π , starting in state s 0 and taking action a 0 , is deﬁned as d πs 0 , [Equation] 0 ( s ) =

Original: (1 − γ ) (cid:80) ∞
Revised: [Equation] γ ) (cid:80) ∞

Original: t =

Revised: t =

Original: Pr ( s t = s | s 0 , a 0 , a t +1 ∼ π, s t +1 ∼ T ( s t , a t ))
Revised: Pr ( s t = s | s 0 , a 0 , a t +1 ∼ π, s t +1 ∼ T ( s t , a t ))

Original: (Agarwal et al., 2019a).
Revised: (Agarwal et al., 2019).

Original: If action a 0 is not given, then we deﬁne d π s 0 ( s ) =
Revised: If action a 0 is not given, then we deﬁne d πs 0 ( s ) =

Original: E a 0 ∼ π
Revised: E a 0 ∼ π

Original: [ d s 0 ,a 0 ( s )] .
Revised: [ d s 0 ,a 0 ( s )] .

Original: For any given policy, we deﬁne the state-value and action-value functions as expectations over the discounted state visitation distribution 1 ,
Revised: For any given policy, we deﬁne the state-value and action-value functions as expectations over the discounted state visitation distribution 1 ,

Original: [Equation]
Revised: [Equation]

Original: ∼ π [ r ( s (cid:48) , a (cid:48) )] , q π ( s, a ) = 1 − γ E s (cid:48) ∼ d πs,a ,a (cid:48) ∼ π [ r ( s (cid:48) , a (cid:48) )] .
Revised: [Equation]

Original: We denote the optimal policy that maximizes the expected cumulative reward ( i.e. return) by π ∗ = arg max π E s ∼ µ 

Revised: We denote the optimal policy that maximizes the expected cumulative reward ( i.e. return) by π ∗ = arg max π E s ∼ µ

Original: [ v π ( s )] .
Revised: [ v π ( s )] .

Original: Lastly, we use the shorthand v ∗ ( s ) = v π ∗ ( s ) and q ∗ ( s, a ) =
Revised: Lastly, we use the shorthand v ∗ ( s ) = v π ∗ ( s ) and q ∗ ( s, a ) =

Original: q π ∗ ( s, a ) for the value of the optimal policy (Sutton and Barto, 2018).
Revised: q π ∗ ( s, a ) for the value of the optimal policy (Sutton and Barto, 2018).

Original: 2. The Current Paradigm: Generalization Across MDPs
Revised: 2. The Current Paradigm: Generalization Across MDPs

Original: Studying generalization in RL is challenging in part due to the MDP problem formulation.
Revised: Studying generalization in RL is challenging in part due to the MDP problem formulation.

Original: As an agent interacts and learns in an environment, which is assumed to be an MDP, the agent’s policy changes and it continuously visits new states.
Revised: As an agent interacts and learns in an environment, which is assumed to be an MDP, the agent’s policy changes and it continuously visits new states.

Original: Due to the changing policy, it is not possible to forbid an agent from entering a state that a practitioner would like to use later for testing.
Revised: Due to the changing policy, it is not possible to forbid an agent from entering a state that a practitioner would like to use later for testing.

Original: To evaluate RL agents’ ability to generalize, one can instead use a generalized environment E =
Revised: To evaluate RL agents’ ability to generalize, one can instead use a generalized environment E =

Original: ( Θ , ρ ) that provides a distribution ρ over a collection of environments Θ = {M i } Ni =

Revised: ( Θ , ρ ) that provides a distribution ρ over a collection of environments Θ = {M i } Ni =

Original: (Whiteson et al., 2011).
Revised: (Whiteson et al., 2011).

Original: It is then the responsibility of the environment designer to ensure that there is similarity among the environments M i such that generalization is possible.
Revised: It is then the responsibility of the environment designer to ensure that there is similarity among the environments M i such that generalization is possible.

Original: Yet, the environments must also be diﬀerent enough to avoid inadvertently training on experience that is later tested.
Revised: Yet, the environments must also be diﬀerent enough to avoid inadvertently training on experience that is later tested.

Original: The state and action spaces, for example, should not change between the environments.
Revised: The state and action spaces, for example, should not change between the environments.

Original: Lastly, one can think of the distribution over environments ρ as a distribution over the distribution of start states µ i for each M i .
Revised: Lastly, one can think of the distribution over environments ρ as a distribution over the distribution of start states µ i for each M i .

Original: Even with generalized environments, such as procedurally generated games (Cobbe et al., 2020), we are unable to probe the details of an agent’s generalization ability during learning.
Revised: Even with generalized environments, such as procedurally generated games (Cobbe et al., 2020), we are unable to probe the details of an agent’s generalization ability during learning.

Original: By measuring only the Monte-Carlo return of the learned policy on an unseen environment instance, we are restricting the evaluation criterion to on-policy control in this new environment instance.
Revised: By measuring only the Monte-Carlo return of the learned policy on an unseen environment instance, we are restricting the evaluation criterion to on-policy control in this new environment instance.

Original: Even if we accept that this is the performance criterion that matters, we can only evaluate the relative performance of the policy because we typically do not know the optimal policy.
Revised: Even if we accept that this is the performance criterion that matters, we can only evaluate the relative performance of the policy because we typically do not know the optimal policy.

Original: Lastly, this evaluation protocol cannot evaluate the agent’s ability to generalize in a single environment.
Revised: Lastly, this evaluation protocol cannot evaluate the agent’s ability to generalize in a single environment.

Original: Harnessing generalization within a single environment is as important as transfer to new environments, and was the original goal for combining reinforcement learning with function approximation (Sutton, 1995; Boyan and Moore, 1994;
Revised: Harnessing generalization within a single environment is as important as transfer to new environments, and was the original goal for combining reinforcement learning with function approximation (Sutton, 1995; Boyan and Moore, 1994;

Original: Tesauro, 1995; Murphy, 2005).
Revised: Tesauro, 1995; Murphy, 2005).

Original: 2. Proposed Paradigm: Evaluating generalization using CDPs
Revised: 2. Proposed Paradigm: Evaluating generalization using CDPs

Original: The Contextual Decision Process (CDP) adds an observation space (also referred to as context)
Revised: The Contextual Decision Process (CDP) adds an observation space (also referred to as context)

Original: O to the MDP, and renders the state-space unobservable to the agent (Jiang et al., 2017).
Revised: O to the MDP, and renders the state-space unobservable to the agent (Jiang et al., 2017).

Original: Observations are provided to the agent by an emission function φ : O × S → [0 , 1] that samples an observation o ∈ O at a state s ∈ S .
Revised: Observations are provided to the agent by an emission function φ : O × S → [0 , 1] that samples an observation o ∈ O at a state s ∈ S .

Original: The reward and state transitions are still deﬁned as functions of the unobservable states s ∈ S , and the agent is responsible for learning how to map the observations to states.
Revised: The reward and state transitions are still deﬁned as functions of the unobservable states s ∈ S , and the agent is responsible for learning how to map the observations to states.

Original: We write the set of observations associated with a speciﬁc state s as O s =
Revised: We write the set of observations associated with a speciﬁc state s as O s =

Original: { o : φ − 1 ( o ) = s } , for some decoding function φ − 1 .
Revised: { o : φ − 1 ( o ) = s } , for some decoding function φ − 1 .

Original: In this work, we assume that the observation space is disjoint: each state s ∈ S is associated with a subset of the observations O s ⊂
Revised: In this work, we assume that the observation space is disjoint: each state s ∈ S is associated with a subset of the observations O s ⊂

Original: O where O s ∩ O s (cid:48)
Revised: O where O s ∩ O s (cid:48)

Original: = ∅ for s (cid:54)
Revised: = ∅ for s (cid:54)

Original: =
Revised: =

Original: s (cid:48) .
Revised: s (cid:48) .

Original: The disjoint property ensures the existence of the decoding function φ − 1 that maps each observation to a single state.
Revised: The disjoint property ensures the existence of the decoding function φ − 1 that maps each observation to a single state.

Original: Recent literature introduced the term “Block MDP” to refer to a CDP with the disjoint property (Du et al., 2019).
Revised: Recent literature introduced the term “Block MDP” to refer to a CDP with the disjoint property (Du et al., 2019).

Original: The experiments in this paper only use Block MDPs, but this is to focus on the generalization problem without introducing partial observability.
Revised: The experiments in this paper only use Block MDPs, but this is to focus on the generalization problem without introducing partial observability.

Original: While CDPs have been the subject of theoretical study, no work has investigated its use as an evaluation platform for studying generalization.
Revised: While CDPs have been the subject of theoretical study, no work has investigated its use as an evaluation platform for studying generalization.

Original: To accomplish this, we outline how a CDP can be combined with a supervised learning dataset to allow for generalization across observations.
Revised: To accomplish this, we outline how a CDP can be combined with a supervised learning dataset to allow for generalization across observations.

Original: To generate observations, we choose to use images from a supervised learning dataset.
Revised: To generate observations, we choose to use images from a supervised learning dataset.

Original: Given a ﬁnite state MDP, we use a K-way classiﬁcation dataset { ( x i , y i ) } ni =1 , with x i ∈ R d and labels y i ∈ { 1 , . . .
Revised: Given a ﬁnite state MDP, we use a K-way classiﬁcation dataset { ( x i , y i ) } ni =1 , with x i ∈ R d and labels y i ∈ { 1 , . . .

Original: , K } .
Revised: , K } .

Original: If the total number of classes K is equal to the number of states |S| in the MDP, then each state can be uniquely identiﬁed with a class label, yielding the observation sets O s =
Revised: If the total number of classes K is equal to the number of states |S| in the MDP, then each state can be uniquely identiﬁed with a class label, yielding the observation sets O s =

Original:  x i : y i = s } .
Revised: { x i : y i = s } .

Original: If the states have structure, such as if each state is an xy-coordinate in a gridworld environment s
Revised: If the states have structure, such as if each state is an xy-coordinate in a gridworld environment s

Original: =
Revised: =

Original: ( s (1) , s (2) ) , then the emission function can concatenate the images corresponding to each component,
Revised: ( s (1) , s (2) ) , then the emission function can concatenate the images corresponding to each component,

Original: O s = { [ x i , x j ] : y
Revised: O s = { [ x i , x j ] : y

Original: i =
Revised: i =

Original: s (1) ,
Revised: s (1) ,

Original: y j = s (2) } .
Revised: y j = s (2) } .

Original: Using the provided training and testing split of supervised learning datasets, one can then use training data during learning and testing data during evaluation.
Revised: Using the provided training and testing split of supervised learning datasets, one can then use training data during learning and testing data during evaluation.

Original: With this formulation, generalization to new observations can be studied within a single environment.
Revised: With this formulation, generalization to new observations can be studied within a single environment.

Original: In the next section, we discuss how to extend this idea to states and actions.
Revised: In the next section, we discuss how to extend this idea to states and actions.

Original:  Characterizing Generalization in RL
Revised:  Characterizing Generalization in RL

Original: With the CDP problem formulation, we can now reexamine the diﬀerences between generalization in supervised learning and RL.
Revised: With the CDP problem formulation, we can now reexamine the diﬀerences between generalization in supervised learning and RL.

Original: In supervised learning, measuring generalization requires a training distribution D train , testing distribution D test and performance criterion g which is low if the function approximator f is performant.
Revised: In supervised learning, measuring generalization requires a training distribution D train , testing distribution D test and performance criterion g which is low if the function approximator f is performant.

Revised: (cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) testing (cid:58)(cid:58)(cid:58)(cid:58) set (cid:58)(cid:58)(cid:58)(cid:58) come (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) same (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) population (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution, (cid:58)(cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) maintain (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sample (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) separate (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)

Original:  We say that a function approximator f exhibits good generalization if the generalization gap,
Revised: [Table] distribution. (cid:58) We say that a function approximator f exhibits good generalization if the generalization gap,

Original: [Equation]
Revised: [Equation]

Original: is small.
Revised: is small.

Original: When the training error can be minimized to zero, as in supervised learning with over-parameterized neural networks, generalization can be characterized by the testing performance alone.
Revised: When the training error can be minimized to zero, as in supervised learning with over-parameterized neural networks, generalization can be characterized by the testing performance alone.

Original: As discussed in Section 2.1, generalization in RL is often measured by the diﬀerence in value achieved by a policy π over a distribution 2 of training instances ρ train and testing instances ρ test from a generalized environment E ,
Revised: As discussed in Section 2.1, generalization in RL is often measured by the diﬀerence in value achieved by a policy π over a distribution 2 of training instances ρ train and testing instances ρ test from a generalized environment E ,

Original: [Equation]
Revised: [Equation]

Original: (3) where the expectation and the value are both estimated using Monte-Carlo methods.
Revised: where the expectation and the value are both estimated using Monte-Carlo methods.

Original: Note that a higher v π is more performant, and so we ﬂip the order in which test and train appear in the gap so that G ≥ 0 .
Revised: Note that a higher v π is more performant, and so we ﬂip the order in which test and train appear in the gap so that G ≥ 0 .

Original: While Equations 2 and 3 seem otherwise similar, they have a number of diﬀerences.
Revised: While Equations 2 and 3 seem otherwise similar, they have a number of diﬀerences.

Original: First, the performance criterion in RL has no ground-truth label for comparison.
Revised: First, the performance criterion in RL has no ground-truth label for comparison.

Original: An RL agent attempts to maximize the value, but the true optimal value function v ∗ is unknown in most environments used to study generalization.
Revised: An RL agent attempts to maximize the value, but the true optimal value function v ∗ is unknown in most environments used to study generalization.

Original: A randomly initialized function will exhibit similarly poor performance during testing and training and hence the generalization gap will be small.
Revised: A randomly initialized function will exhibit similarly poor performance during testing and training and hence the generalization gap will be small.

Original: In supervised learning, we can detect that, although the generalization gap is small, the error is high (or performance is low, in the case of value or accuracy).
Revised: In supervised learning, we can detect that, although the generalization gap is small, the error is high (or performance is low, in the case of value or accuracy).

Original: Second, there is a diﬀerence in the distributions that are used in the expectation.
Revised: Second, there is a diﬀerence in the distributions that are used in the expectation.

Original: A supervised learning model learns to produce certain outputs, conditioned on certain inputs.
Revised: A supervised learning model learns to produce certain outputs, conditioned on certain inputs.

Original: The model is then tested on held out inputs.
Revised: The model is then tested on held out inputs.

Original: An RL agent evaluated in the current paradigm produces actions conditioned on states.
Revised: An RL agent evaluated in the current paradigm produces actions conditioned on states.

Original: However, it is then evaluated on entire environments where it chooses actions and produces its own next states as input.
Revised: However, it is then evaluated on entire environments where it chooses actions and produces its own next states as input.

Original: Lastly, there is a potential mismatch in the performance criterion.
Revised: Lastly, there is a potential mismatch in the performance criterion.

Original: The value of the policy is not the explicit objective being optimized by most reinforcement learning algorithms.
Revised: The value of the policy is not the explicit objective being optimized by most reinforcement learning algorithms.

Original: This is not an issue in supervised learning because the optimization objectives are calibrated surrogates (Chen et al., 2019).
Revised: This is not an issue in supervised learning because the optimization objectives are calibrated surrogates (Chen et al., 2019).

Original: The two issues that we explicitly address is the discrepancy in the measuring distributions and a lack of ground-truth labels.
Revised: The two issues that we explicitly address is the discrepancy in the measuring distributions and a lack of ground-truth labels.

Original: These two issues make operationalizing the supervised learning notion of generalization diﬃcult in RL.
Revised: These two issues make operationalizing the supervised learning notion of generalization diﬃcult in RL.

Original: Using the CDP formulation from Section 2, we are able to alleviate these two issues.
Revised: Using the CDP formulation from Section 2, we are able to alleviate these two issues.

Original: First, we are able to sample from any distribution over states, observations and actions, meaning that we do not have to explicitly rollout the policy for evaluation.
Revised: First, we are able to sample from any distribution over states, observations and actions, meaning that we do not have to explicitly rollout the policy for evaluation.

Original: Second, we can use dynamic programming with the unobserved latent states to recover the optimal value and policy, and use those as a comparison to the learned value and policy.
Revised: Second, we can use dynamic programming with the unobserved latent states to recover the optimal value and policy, and use those as a comparison to the learned value and policy.

Original: One question remains: what performance criterion should we use to test an agent’s ability to generalize?
Revised: One question remains: what performance criterion should we use to test an agent’s ability to generalize?

Original: Re-examining the current paradigm for measuring generalization in RL, we see how the CDP approach can help remedy the compromises made when learning in complex environments.
Revised: Re-examining the current paradigm for measuring generalization in RL, we see how the CDP approach can help remedy the compromises made when learning in complex environments.

Original: An RL agent’s task is to maximize return and hence, most RL research reports the returns that an agent receives from interacting with the environment.
Revised: An RL agent’s task is to maximize return and hence, most RL research reports the returns that an agent receives from interacting with the environment.

Original: Due to the the lack of groundtruth labels, the performance criterion g is limited to the Monte-Carlo estimate of v π .
Revised: Due to the the lack of groundtruth labels, the performance criterion g is limited to the Monte-Carlo estimate of v π .

Original: While
Revised: While

Original: Monte-Carlo returns do demonstrate policy performance, it does not probe the agent in its estimates of the actual values being learned, and is only one part of the generalization puzzle.
Revised: Monte-Carlo returns do demonstrate policy performance, it does not probe the agent in its estimates of the actual values being learned, and is only one part of the generalization puzzle.

Original: Given the optimal policy and action-values, an agent can be evaluated with more acuity than Monte-Carlo returns.
Revised: Given the optimal policy and action-values, an agent can be evaluated with more acuity than Monte-Carlo returns.

Original: One possibility is to measure the accuracy of the policy, where accuracy is given by the agreement of the learned policy with the true optimal action.
Revised: One possibility is to measure the accuracy of the policy, where accuracy is given by the agreement of the learned policy with the true optimal action.

Original: Another possibility is to measure the mean squared error between the learned action values and the true optimal action values.
Revised: Another possibility is to measure the mean squared error between the learned action values and the true optimal action values.

Original: While the optimal policy is deﬁned as a function of state, action-values can be evaluated on arbitrary actions.
Revised: While the optimal policy is deﬁned as a function of state, action-values can be evaluated on arbitrary actions.

Original: This means that even if an action was not taken in a particular state, we can evaluate the agent’s estimate of that particular action-value.
Revised: This means that even if an action was not taken in a particular state, we can evaluate the agent’s estimate of that particular action-value.

Original: Thus, we consider three types of generalization: state, observation, and action generalization.
Revised: Thus, we consider three types of generalization: state, observation, and action generalization.

Original: To ease the notation, we deﬁne the joint training distribution over state, observation and action, starting at state s 0 as p π,π (cid:48) s 0 ,tr ( s, o, a )
Revised: To ease the notation, we deﬁne the joint training distribution over state, observation and action, starting at state s 0 as p π,π (cid:48) s 0 ,tr ( s, o, a )

Original: =
Revised: =

Original: d πs 0 ,tr ( s ) φ tr ( o | s ) π (cid:48) tr ( a | o ) , with the testing distribution deﬁned similarly.
Revised: d πs 0 ,tr ( s ) φ tr ( o | s ) π (cid:48) tr ( a | o ) , with the testing distribution deﬁned similarly.

Original: The second policy π (cid:48) is often the same as π , but this second degree of freedom will be needed for action generalization.
Revised: The second policy π (cid:48) is often the same as π , but this second degree of freedom will be needed for action generalization.

Original: Using this shorthand, we deﬁne the generalization gap of a function approximator f with respect to a performance criterion g as,
Revised: Using this shorthand, we deﬁne the generalization gap of a function approximator f with respect to a performance criterion g as,

Original: [Equation] 
Revised: [Equation] [Table]

Original: In the next two subsections, we will investigate each component of this joint distribution and its connection to generalization across each of the three axes: state, observation and action.
Revised: In the next two subsections [Table] we will investigate each component of this joint distribution and its connection to generalization across each of the three axes: state, observation and action.

Original: We will also show examples of performance criteria g that can be minimized, other than the scaled negativereward g ( s (cid:48) , o (cid:48) , a (cid:48) , π )
Revised: We will also show examples of performance criteria g that can be minimized [Table] , other than the scaled negative

Original: = − 11 −
Revised: [Table] g [Table]

Original: γ r ( s (cid:48) , a (cid:48) ) , that may be of interest when studying generalization in RL using CDPs. 
Revised: , that may be of interest when studying generalization in RL using CDPs. [Table] and [Table]

Original:  3. Observation and State-space Generalization
Revised: actions. 3. Observation and State-space Generalization

Original: Observation-space generalization is the most commonly explored notion of generalization in RL (see, i.e.  Song et al.
Revised: Observation-space generalization is the most commonly explored notion of generalization in RL (see, i.e. ? (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Song et al.

Original: Given an observation o , sampled from the test distribution φ test that the agent has not seen before, the agent is evaluated on quantities related to that observation.
Revised: Given an observation o , sampled from the test distribution φ test that the agent has not seen before, the agent is evaluated on quantities related to that observation.

Original: In model-free algorithms that learn action-values parameterized by θ , for example, an agent may be evaluated on the mean squared error of its value of the greedy policy, g mse ( o, s, a, θ )
Revised: In model-free algorithms that learn action-values parameterized by θ , for example, an agent may be evaluated on the mean squared error of its value of the greedy policy, g mse ( o, s, a, θ )

Original: =
Revised: =

Original: ( v ( o ; θ ) − v ∗ ( s )) 2 .
Revised: ( v ( o ; θ ) − v ∗ ( s )) 2 .

Original: We focus on action-unconditional quantities to separate observation-space generalization from action-space generalization that we will discuss in the next section.
Revised: We focus on action-unconditional quantities to separate observation-space generalization from action-space generalization that we will discuss in the next section.

Original: Another example of an action unconditional performance criterion is the accuracy of the greedy policy induced by q parameterized by θ , g acc
Revised: Another example of an action unconditional performance criterion is the accuracy of the greedy policy induced by q parameterized by θ , g acc

Original: ( o, s, a, θ ) =
Revised: ( o, s, a, θ ) =

Original: 1 (arg max a q ( o, a ; θ ) =
Revised: 1 (arg max a q ( o, a ; θ ) =

Original: π ∗ ( s )) where 1 is the indicator function.
Revised: π ∗ ( s )) where 1 is the indicator function.

Original: There is often no distinction between observations and states in the current RL literature because, except in toy problems, the underlying MDP (and hence, the true Markovian state) is not known.
Revised: There is often no distinction between observations and states in the current RL literature because, except in toy problems, the underlying MDP (and hence, the true Markovian state) is not known.

Original: However, there are important diﬀerences.
Revised: However, there are important diﬀerences.

Original: The state transition and reward are functions of the state, hence generalization across states must necessarily generalize to diﬀerent state-transition and reward functions.
Revised: The state transition and reward are functions of the state, hence generalization across states must necessarily generalize to diﬀerent state-transition and reward functions.

Original: For online algorithms, measuring state generalization is not straightforward because the agent does not put weight on states outside of its state visitation distribution d πs 0 ( s ) .
Revised: For online algorithms, measuring state generalization is not straightforward because the agent does not put weight on states outside of its state visitation distribution d πs 0 ( s ) .

Original: Expecting an agent to generalize to an arbitrary state outside of this distribution is similar to the zero-shot generalization problem, and would require a meta-learning approach.
Revised: Expecting an agent to generalize to an arbitrary state outside of this distribution is similar to the zero-shot generalization problem, and would require a meta-learning approach.

Original: More reasonable would be to expect the agent to generalize to states near the distribution d πs 0 ( s ) .
Revised: More reasonable would be to expect the agent to generalize to states near the distribution d πs 0 ( s ) .

Original: One way this can be achieved is if the observation o (cid:48) ∼ φ test ( ·| s (cid:48) ) of a state that has not been encountered by the agent s (cid:48) ∼ d πs 0 ,test shares some structure with observations o ∼ φ tr ( ·| s ) of seen states s ∼ d πs 0 ,test .
Revised: One way this can be achieved is if the observation o (cid:48) ∼ φ test ( ·| s (cid:48) ) of a state that has not been encountered by the agent s (cid:48) ∼ d πs 0 ,test shares some structure with observations o ∼ φ tr ( ·| s ) of seen states s ∼ d πs 0 ,test .

Original: Then, the function approximator f can generalize to this new state via its observation space generalization.
Revised: Then, the function approximator f can generalize to this new state via its observation space generalization.

Original: It remains to construct an environment with naturally unreachable states, for which the observation structure permits generalization, such as in certain gridworld setups (Witty et al., 2018).
Revised: It remains to construct an environment with naturally unreachable states, for which the observation structure permits generalization, such as in certain gridworld setups (Witty et al., 2018).

Original: 3. Action-space Generalization
Revised: 3. Action-space Generalization

Original: Generalization to out-of-distribution actions is a less explored notion compared to observation or state-space generalization.
Revised: Generalization to out-of-distribution actions is a less explored notion compared to observation or state-space generalization.

Original: To evaluate action-space generalization, the agent sould  be evaluated on quantities related to actions that were not taken or that have low probability under the current policy.
Revised: To evaluate action-space generalization, the agent sould (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) should be evaluated on quantities related to actions that were not taken or that have low probability under the current policy.

Original: In model-free algorithms that learn action-values parameterized by θ for example, the agent can be evaluated on its estimate of the action-value for an action a not ever taken by the agent in state s , and hence not taken in any of it’s observations o ∈
Revised: In model-free algorithms that learn action-values parameterized by θ for example, the agent can be evaluated on its estimate of the action-value for an action a not ever taken by the agent in state s , and hence not taken in any of it’s observations o ∈

Original: O s , g act ( s, o, a, θ ) =
Revised: O s , g act ( s, o, a, θ ) =

Original: ( q ( o, a ; θ ) − q ∗ ( s, a )) 2 .
Revised: ( q ( o, a ; θ ) − q ∗ ( s, a )) 2 .

Original: One way this can be formalized is by setting π (cid:48) to put all of its support on actions outside of the support of π .
Revised: One way this can be formalized is by setting π (cid:48) to put all of its support on actions outside of the support of π .

Original: This setting is particularly important for oﬄine learning, where the π would be the behavior policy and π (cid:48) is the target policy we are trying to learn.
Revised: This setting is particularly important for oﬄine learning, where the π would be the behavior policy and π (cid:48) is the target policy we are trying to learn.

Original: Control is one of RL’s main diﬀerences from supervised learning, wherein the agent collects the data that it uses to learn.
Revised: Control is one of RL’s main diﬀerences from supervised learning, wherein the agent collects the data that it uses to learn.

Original: We argue that action-space generalization is more important than observation or state-space generalization because policies in RL tend to place non-zero probability on all actions for exploration purposes.
Revised: We argue that action-space generalization is more important than observation or state-space generalization because policies in RL tend to place non-zero probability on all actions for exploration purposes.

Original: If the agent chooses to explore when faced in an unknown situation, it must leverage its ability to generalize in action-space.
Revised: If the agent chooses to explore when faced in an unknown situation, it must leverage its ability to generalize in action-space.

Original: It is thus surprising that action-space generalization has seen limited investigation.
Revised: It is thus surprising that action-space generalization has seen limited investigation.

Original: The work of
Revised: The work of

Original: Chandak et al.
Revised: Chandak et al.

Original: ; Dulac-Arnold et al.
Revised: ; Dulac-Arnold et al.

Original: (2015) ﬁnds that learned action representations are able to improve thepolicy’s performance in environments withlarge discrete action spaces.
Revised: (2015) ﬁnds that learned action representations are able to improve the performance in environments with spaces.

Original: While the authors claim that the learned representation improves performance, there is no speciﬁc study of this ﬁnding.
Revised: While the authors claim that the learned representation improves performance, there is no speciﬁc study of this ﬁnding.

Original: Furthermore, no work has explicitly measured an agent’s ability to generalize to an unseen action.
Revised: Furthermore, no work has explicitly measured an agent’s ability to generalize to an unseen action.

Original:  Experiments
Revised:  Experiments

Original: All of our experiments are conducted in a contextual decision process, where the observations are derived from the MNIST dataset (LeCun et al., 2010) and the underlying MDP is either a corridor environment or a four rooms gridworld (Sutton et al., 1999).
Revised: All of our experiments are conducted in a contextual decision process, where the observations are derived from the MNIST dataset (LeCun et al., 2010) and the underlying MDP is either a corridor environment or a four rooms gridworld (Sutton et al., 1999).

Original: See the Appendix for details on each environment, including state, action and observation spaces.
Revised: See the Appendix for details on each environment, including state, action and observation spaces.

Revised: (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experimental (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) protocol, (cid:58)(cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58) well (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original:  While these two environments are relatively simple, they provide a thorough testbed for probing diﬀerent aspects of generalization for an RL agent.
Revised: environments. (cid:58) While these two environments are relatively simple, they provide a thorough testbed for probing diﬀerent aspects of generalization for an RL agent.

Original: From the agent’s perspective, the problem of mapping an observation to a state alone is challenging.
Revised: From the agent’s perspective, the problem of mapping an observation to a state alone is challenging.

Original: Early in training, each observation is unique and so the agent does not know if it is making progress until it reaches the goal.
Revised: Early in training, each observation is unique and so the agent does not know if it is making progress until it reaches the goal.

Original: Even after reaching the goal, it is not immediately clear how many unique states were encountered in the trajectory.
Revised: Even after reaching the goal, it is not immediately clear how many unique states were encountered in the trajectory.

Original: The agent must do this for each state, and then learn that the  action-values in these  diﬀerent observations are the same if the underlying state is the same.
Revised: The agent must do this for each state, and then learn that the (cid:58)(cid:58)(cid:58)(cid:58) learn (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) mapping (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observation (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) action-value (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) fact (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) that action-values in these (cid:58)(cid:58) at diﬀerent observations are the same if the underlying state is the same.

Original: Note that the optimal policy in the corridor environment is the classiﬁcation policy.
Revised: Note that the optimal policy in the corridor environment is the classiﬁcation policy.

Original: However, the classiﬁcation policy must be learned from a sparse reward signal.
Revised: However, the classiﬁcation policy must be learned from a sparse reward signal (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) occurs [Table] only [Table]

Revised: at

Original: Taken together, the agent must generalize across diﬀerent states, observations and actions.
Revised: [Table] the (cid:58)(cid:58)(cid:58) end (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the [Table]

Revised: episode.

Original:  Our goal for each experiment is to rigorously evaluate generalization in diﬀerent RL training regimes.
Revised: Taken together, the agent must generalize across diﬀerent states, observations and actions [Table] [Table] Our goal for each experiment is to rigorously evaluate generaliza

Original:  We can do this because the observations can  be partitioned into training, validation and testing sets.
Revised: [Table] observations (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) actions (cid:58) in diﬀerent RL training regimes (cid:58) , (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diﬀerent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generalization (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) mechanisms. We can do this because the observations can (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) states, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observations (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) actions (cid:58)(cid:58)(cid:58)(cid:58) can (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58) be partitioned into training, validation and testing sets .

Original: In addition to observation generalization, we also investigate action-generalization in the corridor environment and state-generalization in the four rooms environment.
Revised: In addition to observation generalization, we also investigate (cid:58) in (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) setup.

Original:  We focus on oﬀ-policy algorithms, namely DQN (Mnih et al., 2013) and
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) identify (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) three (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) mechanisms (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) have (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) been (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) claimed (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Quantile-Regression DQN (Dabney et al., 2017), because their ability to learn from arbitrary data necessitates generalization.
Revised: [Table] on action-generalization in the corridor environment and (cid:58)(cid:58) on (cid:58) state-generalization in the four rooms environment. We focus on oﬀ-policy algorithms, namely DQN (Mnih et al., 2013) and Quantile-Regression DQN ( ? ), because their ability to learn from arbitrary data necessitates generalization (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) introduce (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corrupted (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) variants (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: We evaluate both oﬀ-policy model-free algorithms after either online and oﬄine training.
Revised: and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generalization (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) mechanism (cid:58) after either online and oﬄine training.

Original: In the online setting, the agent will interact with the environment while updating its policy.
Revised: In the online setting, the agent will interact with the environment while updating its policy.

Original: The agent will be periodically evaluated against ground-truth quantities pertaining to the environment and test set observations.
Revised: The agent will be periodically evaluated against ground-truth quantities pertaining to the environment and test set observations.

Original: In the oﬄine setting, the agent will learn from a logged set of experience in a replay buﬀer without any direct interaction with the environment during learning.
Revised: In the oﬄine setting, the agent will learn from a logged set of experience in a replay buﬀer without any direct interaction with the environment during learning.

Original: We also introduce corrupted variants of these environments, where the class labels in the dataset are shuﬄed to test the agent’s ability to memorize it’s training input.
Revised: We also introduce corrupted variants of these environments, where the class labels in the dataset are shuﬄed to test the agent’s ability to memorize it’s training input.

Original: For all experiment in non-corrupted environments, we experiment with L2 regularization and dropout, as they are common regularization techniques in machine learning and some have reported that these techniques improve generalization.
Revised: For all experiment in non-corrupted environments, we experiment with L2 regularization and dropout, as they are common regularization techniques in machine learning and some have reported that these techniques improve generalization.

Original: 4. Observation generalization
Revised: 4. Observation generalization

Original: To measure observation generalization, we train an agent using observations from the training set and evaluate on a held out test-set.
Revised: To measure observation generalization, we train an agent using observations from the training set and evaluate on a held out test-set.

Original: We only show the results for the corridor environment here, but have additional experiments on four rooms in the Appendix.
Revised: We only show the results for the corridor environment here, but have additional experiments on four rooms in the Appendix.

Original: Referring to the oﬄine learning experiments depicted in Figure 10, we see that QR-DQN is able to generalize better than DQN, as shown in its lower accuracy generalization gap.
Revised: Referring to the oﬄine learning experiments depicted in Figure 1, we see that QR-DQN is able to generalize better than DQN, as shown in its lower accuracy generalization gap.

Original: We also ﬁnd that small amounts of regularization indeed lowers the generalization gap, whereas dropout does not improve observation generalization.
Revised: We also ﬁnd that small amounts of regularization indeed lowers the generalization gap, whereas dropout does not improve observation generalization.

Original: In the online learning experiments, shown in Figure 2, QR-DQN and DQN both generalize to new observations similarly.
Revised: In the online learning experiments, shown in Figure 2, QR-DQN and DQN both generalize to new observations similarly.

Original: Interestingly, the generalization gap is overall larger in the online setting.
Revised: Interestingly, the generalization gap is overall larger in the online setting.

Original: We also ﬁnd that setting the regularization strength to 0.1 now inhibits learning.
Revised: We also ﬁnd that setting the regularization strength to 0.1 now inhibits learning.

Original: Small amounts of regularization still improves generalization to new observations however.
Revised: Small amounts of regularization still improves generalization to new observations however.

Original: Lastly, we see that dropout hurts learning in the oﬄine setting.
Revised: Lastly, we see that dropout hurts learning in the oﬄine setting.

Original: Drawing inspiration from the work of Zhang et al.
Revised: Drawing inspiration from the work of Zhang et al.

Original: (2017), we also run an experiment where the labels are randomized to see if DQN and QR-DQN have similar capacity to memorize the data.
Revised: (2017), we also run an experiment where the labels are randomized to see if DQN and QR-DQN have similar capacity to memorize the data.

Original: Referring to 14, we ﬁnd that DQN and QR-DQN (with a various number of quantile heads) can easily memorize all the training experience in corridor.
Revised: Referring to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Figure (cid:58) 3, we ﬁnd that DQN and QR-DQN (with a various number of quantile heads) can easily memorize all the training experience in corridor.

Original: In the online setting, larger QR-DQN agents begin to struggle to ﬁt the training data.
Revised: In the online setting, larger QR-DQN agents begin to struggle to ﬁt the training data.

Original: This diﬃculty also extends to both the oﬄine and online setting in four rooms, where the observations are higher dimensional.
Revised: This diﬃculty also extends to both the oﬄine and online setting in four rooms, where the observations are higher dimensional.

Original: 4. Action generalization
Revised: 4. Action generalization

Original: For action generalization, we focus on the corridor environment with oﬄine learning from a behaviour policy that systematically avoids a certain action in each state.
Revised: For action generalization, we focus on the corridor environment with oﬄine learning from a behaviour policy that systematically avoids a certain action in each state.

Original: We then measure the action value estimate for the state-action pair that is never seen in the data and compare it to the true action-value.
Revised: We then measure the action value estimate for the state-action pair that is never seen in the data and compare it to the true action-value.

Original: Referring to Figure 4, we ﬁnd QR-DQN is better able to generalize to unseen actions.
Revised: Referring to Figure 4, we ﬁnd QR-DQN is better able to generalize to unseen actions.

Original: We also ﬁnd that, contrasting to observation generalization, dropout now improves action generalization whereas regularization does not.
Revised: We also ﬁnd that, contrasting to observation generalization, dropout now improves action generalization whereas regularization does not.

Original: 4. State generalization
Revised: 4. State generalization

Original: We use the four rooms environment to evaluate an agent’s ability to generalize to new states.
Revised: We use the four rooms environment to evaluate an agent’s ability to generalize to new states.

Original: The interior walls of the four rooms environment share either an x or y coordinate with states that the agent has encountered, and hence the student can leverage generalization between observations to these new states.
Revised: The interior walls of the four rooms environment share either an x or y coordinate with states that the agent has encountered, and hence the student can leverage generalization between observations to these new states.

Original: We evaluate the accuracy of the policy in these interior wall states.
Revised: We evaluate the accuracy of the policy in these interior wall states.

Original: Referring to Figure 5 and Figure 6 for oﬄine and online training respectively, we see similar trends.
Revised: Referring to Figure 5 and Figure 6 for oﬄine and online training respectively, we see similar trends.

Original: In both cases, DQN is able to generalize to the new states better than QR-DQN.
Revised: In both cases, DQN is able to generalize to the new states better than QR-DQN.

Original: We also ﬁnd that, despite a lower generalization gap in some setting, both regularization and dropout hurt training accuracy.
Revised: We also ﬁnd that, despite a lower generalization gap in some setting, both regularization and dropout hurt training accuracy.

Original: This means that, although the training performance is more predictive of its testing performance, it’s overall testing performance is lower. 
Revised: This means that, although the training performance is more predictive of its testing performance, it’s overall testing performance is lower. [Table] Discussion [Table]

Original: Conclusion
Revised: Our [Table] have [Table]

Revised: previous (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁndings, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) namely (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: QR-DQN’s (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) superiority (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: [Table]

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) regime (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (

Revised: [Table]

Revised: ) (cid:58) ,

Revised: [Table]

Revised: et al., 2019) (cid:58) .

Revised: (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) discover (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) nuanced (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) beneﬁt (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dropout, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provides (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) regularization (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) outputs (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: [Table]

Revised: [Table]

Revised: to

Revised: [Table]

Revised: do

Revised: [Table]

Revised: conventional

Revised: [Table]

Revised: studied

Revised: [Table]

Revised: in

Revised: [Table]

Revised: this

Revised: [Table]

Revised: are

Revised: [Table]

Revised: [Table]

Revised: such

Revised: [Table]

Revised: as

Revised: [Table]

Revised: and

Revised: [Table]

Revised: believe (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) new (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁndings (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) would (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generalize (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) conventional (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments (cid:58)(cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58) well.

Original:  Generalization in RL is a multi-faceted issue that deserves rigorous empirical investigation.
Revised: (cid:58)  Conclusion Generalization in RL is a multi-faceted issue that deserves rigorous empirical investigation.

Original: To that end, we have presented a problem formulation using contextual decision processes and supervised learning datasets that provides ground-truth labels for optimal policies and values.
Revised: To that end, we have presented a problem formulation using contextual decision processes and supervised learning datasets that provides ground-truth labels for optimal policies and values.

Revised: Using (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) disentangles (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generalization (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) across (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) states, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observations (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) actions (cid:58)(cid:58)(cid:58) in (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) single (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment.

Original: Using various gridworld environments and the MNIST dataset, we investigate generalization across states, observations and actions.
Revised: (cid:58)(cid:58)(cid:58) By (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) combining (cid:58) various gridworld environments and the MNIST dataset, we investigate generalization (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) recover (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] mechanisms

Original:  We validate previous claims regarding the eﬀectiveness of L2 regularization and QR-DQN’s generalization ability in the oﬄine setting, but ﬁnd that QR-DQN cannot generalize to states as well as DQN.
Revised: [Table] across states, observations and actions. We validate previous claims regarding the eﬀectiveness of L regularization and QR-DQN’s generalization ability in the oﬄine setting, but ﬁnd that QRDQN cannot generalize to states as well as DQN.

Original: We also refute claims, such as dropout’s ability to improve observation generalization, but ﬁnd that it improves action generalization.
Revised: We also refute claims, such as dropout’s ability to improve observation generalization [Table] but ﬁnd that it improves action generalization.

Original: These ﬁndings demonstrate the need for careful consideration  of generalization in RL, and we hope that this line of research will continue to shed light on generalization claims in the literature.
Revised: These ﬁndings demonstrate the need for careful consideration (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) evaluation (cid:58) of generalization in RL, and we hope that this line of research will continue to shed light on generalization claims in the literature [Table] [Table] as

Revised: [Table]

Revised: well

Original:  A Appendix A. Experiment Settings
Revised: [Table] as [Table] A Appendix A. Experiment Settings

Original: Code to reproduce experiments is provided in the following an anonymized dropbox link: https://www.dropbox.com/sh/ij7dvd4fr701ppe/AABf-JszLPe3AVlmaMBDsNyta?dl=
Revised: Code to reproduce experiments is provided in the following an anonymized dropbox link: https://www.dropbox.com/sh/ij7dvd4fr701ppe/AABf-JszLPe3AVlmaMBDsNyta?dl=

Original: Online
Revised: Online

Original: Online Corrupted
Revised: Online Corrupted

Original: Oﬄine
Revised: Oﬄine

Original: Oﬄine Corrupted
Revised: Oﬄine Corrupted

Original: Action Gen
Revised: Action Gen

Original: Init Num Episodes
Revised: Init Num Episodes

Original: Init Policy
Revised: Init Policy

Original: Random Random
Revised: Random Random

Original: Random Random
Revised: Random Random

Original: Custom
Revised: Custom

Original: Max Episode Length
Revised: Max Episode Length

Original: Max Num Episodes
Revised: Max Num Episodes

Original: Num Online Episodes  - - -
Revised: Num Online Episodes  - - -

Original: Num Grad Steps 1 (per env step) 1 (per env step)
Revised: Num Grad Steps 1 (per env step) 1 (per env step)

Original: Target Update Freq
Revised: Target Update Freq

Original: Activation
Revised: Activation

Original: Relu Relu Relu Relu
Revised: Relu (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ReLU

Original: Relu 
Revised: Relu (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ReLU (cid:58) Relu (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ReLU (cid:58) Relu (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ReLU (cid:58) Relu (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ReLU

Original: Hidden Size
Revised: Hidden Size

Original: Num Layers
Revised: Num Layers

Original: Optimizer
Revised: Optimizer

Original: ADAM ADAM ADAM
Revised: ADAM (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Adam (cid:58)

Original: ADAM 
Revised: ADAM (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: ADAM
Revised: Adam

Original: Batch Size Gamma 0. Corruption Rate 1.
Revised: ADAM (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original:  A. Environment Specification
Revised: [Table] A. Environment Specification

Original: The corridor environment has 10 states and 10 actions corresponding to the possible classiﬁcation decisions the policy can make in each state.
Revised: The corridor environment has 10 states and 10 actions corresponding to the possible classiﬁcation decisions the policy can make in each state.

Original: The agent begins at the right-most state of the corridor, with s = 1 , and observes a handwritten 0 digit from the MNIST dataset.
Revised: The agent begins at the right-most state of the corridor, with s = 1 , and observes a handwritten 0 digit from the MNIST dataset.

Original: The agent must correctly classify the digit to move up to the next state.
Revised: The agent must correctly classify the digit to move up to the next state.

Original: For all states, the action a = s sends the agent up a state s (cid:48)
Revised: For all states, the action a = s sends the agent up a state s (cid:48)

Original: =
Revised: =

Original: s +1 .
Revised: s +1 .

Original: For a state S > 1 , the action a = 10 − s +1 sends the agent down a state s (cid:48)
Revised: For a state S > 1 , the action a = 10 − s +1 sends the agent down a state s (cid:48)

Original: =
Revised: =

Original: s − 1 .
Revised: s − 1 .

Original: All other actions keep the agent in the same state, but with a new observation.
Revised: All other actions keep the agent in the same state, but with a new observation.

Original: The reward for all transitions are − 1 and the episode terminates when action a = 10 is taken in s = 10 .
Revised: The reward for all transitions are − 1 and the episode terminates when action a = 10 is taken in s = 10 .

Original: The four rooms environment has four actions for each of the cardinal directions and the observation is a concatenated images corresponding to the class labels for each of the x and y coordinate.
Revised: The four rooms environment has four actions for each of the cardinal directions and the observation is a concatenated images corresponding to the class labels for each of the x and y coordinate.

Original: If the agent tries to move into a wall, it will stay in its current state with a new observation.
Revised: If the agent tries to move into a wall, it will stay in its current state with a new observation.

Original: Lastly, the rewards are initialized randomly (with a ﬁxed seed) to ensure that the optimal action in each state is unique.
Revised: Lastly, the rewards are initialized randomly (with a ﬁxed seed) to ensure that the optimal action in each state is unique.

Original: A. Agent Specification
Revised: T illustration (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MNIST (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Corridor (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labeled (cid:58)(cid:58)(cid:58) by (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) number, (cid:58)(cid:58)(cid:58) its (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) index, (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observations (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MNIST (cid:58)(cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sampled (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) according (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)

Revised: The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) initial (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state (cid:58)(cid:58) 0 , (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) given (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) randomly

Revised: [Table]

Revised: are

Revised: [Table]

Revised: for

Revised: [Table]

Revised: each

Revised: [Table]

Revised: possible

Revised: [Table]

Revised: decision

Revised: [Table]

Revised: that

Revised: [Table]

Revised: must

Revised: [Table]

Revised: be

Revised: [Table]

Revised: through

Revised: [Table]

Revised: learning

Revised: [Table]

Revised: The

Revised: [Table]

Revised: terminal

Revised: [Table]

Original: {+T 28 
+}
Revised: {+T 28 
+}

Revised: [Table]

Revised: from

Revised: [Table]

Revised: for

Revised: [Table]

Revised: each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) number (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) label.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) For (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) example, (cid:58)(cid:58)(cid:58)(cid:58)

Revised: the

Revised: [Table]

Revised: initial (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state (cid:58)(cid:58)(cid:58) is (cid:58)(cid:58)(cid:58) 11 (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) so (cid:58)(cid:58)(cid:58) the

Revised: [Table]

Revised: observation (cid:58)(cid:58) is (cid:58)(cid:58) a

Revised: [Table]

Revised: concatenation (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) two (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) images (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MNIST (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corresponding (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) handwritten (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ones.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: Although (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) label (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) number (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) same (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) initial (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state, (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observations (cid:58)(cid:58)(cid:58)(cid:58) can (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) same (cid:58)(cid:58)(cid:58) or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diﬀerent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) depending (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stochastic (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sample.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) white (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) states (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accessible (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) blue (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) states (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) wall (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) states (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) cannot (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) transitioned (cid:58)(cid:58)(cid:58)(cid:58) into (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) white (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)

Revised: For (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generalization, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) evaluate (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) wall (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: states (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interior (cid:58)(cid:58)(cid:58)(cid:58)

Original:  Both online agents use (cid:15) greedy exploration with (cid:15) = 0 .
Revised: [Table] A. Agent Specification Both online agents use (cid:15) greedy exploration with (cid:15) = 0 .

Original: QR-DQN computes 10 quantiles and is optimized with respect to the Huber loss, with κ =1 . 0 .
Revised: QR-DQN computes 10 quantiles and is optimized with respect to the Huber loss, with κ =

Original:  DQN is optimized with respect to the mean squared error. 
Revised: [Table] DQN is optimized with respect to the mean squared error. [Table]

Original: A. Hyperparameter settings and Experiment Protocol
Revised: A. Hyperparameter settings and Experiment Protocol

Original: The ablation setings  for each experiment are detailed in the legend.
Revised: The ablation setings (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) settings (cid:58) for each experiment are detailed in the legend.

Original: We  also conducted a grid search over learning rates
Revised: We (cid:58)(cid:58)(cid:58) For (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) every (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiment, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58) also conducted a grid search over learning rates

Original: α = { 0 . 005 , 0 .
Revised: α = { 0 . 005 , 0 .

Original: All experiments are averaged over 30 seeds, with a shaded region corresponding to a 95% conﬁdence interval of the mean. 
Revised: All experiments are averaged over 30 seeds, with a shaded region corresponding to a 95% conﬁdence interval of the mean. The best model was selected [Table]

Original: The best model was selected based on validation accuracy, which acts as a proxy for policy performance on the training environment instance. In the corruption experiments, the best model was selected based on training accuracy.
Revised: validation accuracy which acts as a proxy [Equation] This (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provides (cid:58)(cid:58) an (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) optimistic (cid:58)(cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58)(cid:58) fair (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) assessment

Original:  For both the corridor and four rooms environment, we use only 10 samples from each class label to ensure that the agent can ﬁt its training experience, but we do not limit the size of the test and validation sets.
Revised: [Table] For both the corridor and four rooms environment [Table] use only 10 samples from each class label to ensure that the agent can ﬁt its training experience

Revised: [Table]

Revised: but we do not limit the size of the test and validation sets

Revised: [Table]

Revised: analogy (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) supervised (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learning, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) were (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) required (cid:58)(cid:58)(cid:58)

Original: We use dynamic programming to calculate the optimal policy and value function for the states, and choose to report accuracy instead of Monte-Carlo rollouts.
Revised: to [Table] lower [Table]

Revised: the

Revised: [Table]

Revised: so (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)

Revised: the

Revised: [Table]

Revised: agent

Revised: [Table]

Revised: would

Revised: [Table]

Revised: is

Revised: [Table]

Revised: also

Revised: [Table]

Revised: the

Revised: [Table]

Revised: of

Revised: [Table]

Revised: for

Revised: [Table]

Revised: the

Revised: [Table]

Revised: theoretical

Revised: [Table]

Revised: study

Revised: [Table]

Original:  Accuracy provides more acuity in the diﬀerences between policies, allowing us to evaluate the accuracy at each state and under any distribution.
Revised: of [Table] We use dynamic programming to calculate the optimal policy and value function for the states, and choose to report accuracy instead of Monte-Carlo rollouts. Accuracy provides more acuity in the diﬀerences between policies, allowing us to evaluate the accuracy at each state and under any distribution.

Original: All performance criteria (training, validation and testing) are averaged over the same state distribution as the replay buﬀer.
Revised: All performance criteria (training, validation and testing) are averaged over the same state distribution as the replay buﬀer.

Original: A. Observation Generalization in Four Rooms
Revised: A. Observation Generalization in Four Rooms

Revised: A. Monte-Carlo Rollouts in Training and Testing Environments