Original: Hyperbolically Discounted Advantage Estimation for Generalization in Reinforcement Learning
Revised: Hyperbolically Discounted Advantage Estimation for Generalization in Reinforcement Learning

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: In reinforcement learning (RL), agents typically discount future rewards using an exponential scheme.
Revised: In reinforcement learning (RL), agents typically discount future rewards using an exponential scheme.

Original: However, studies have shown that humans and animals instead exhibit hyperbolic timepreferences and thus discount future rewards hyperbolically.
Revised: However, studies have shown that humans and animals instead exhibit hyperbolic timepreferences and thus discount future rewards hyperbolically.

Original: In the quest for RL agents that generalize well to previously unseen scenarios, we study the effects of hyperbolic discounting on generalization tasks and present Hyperbolic
Revised: In the quest for RL agents that generalize well to previously unseen scenarios, we study the effects of hyperbolic discounting on generalization tasks and present Hyperbolic

Original: Discounting for Generalization in Reinforcement
Revised: Discounting for Generalization in Reinforcement

Original: Learning (HDGenRL).
Revised: Learning (HDGenRL).

Original: We propose a hyperbolic discounting-based advantage estimation method that makes the agent aware of and robust to the underlying uncertainty of survival and episode duration.
Revised: We propose a hyperbolic discounting-based advantage estimation method that makes the agent aware of and robust to the underlying uncertainty of survival and episode duration.

Original: On the challenging RL generalization benchmark Procgen, our proposed approach achieves up to 200% performance improvement over the PPO baseline that uses classical exponential discounting.
Revised: On the challenging RL generalization benchmark Procgen, our proposed approach achieves up to 200% performance improvement over the PPO baseline that uses classical exponential discounting.

Original: We also incorporate hyperbolic discounting into another generalization-specific approach (APDAC), and results indicate further improvement in APDAC’s generalization ability.
Revised: We also incorporate hyperbolic discounting into another generalization-specific approach (APDAC), and results indicate further improvement in APDAC’s generalization ability.

Original: This denotes the effectiveness of our approach as a plug-in to any existing methods in aiding generalization.
Revised: This denotes the effectiveness of our approach as a plug-in to any existing methods to aid generalization.

Original: 1. Introduction
Revised: 1. Introduction

Original: Generalization refers to the capability of an agent to perform in similar but unseen environments and is currently an active research challenge.
Revised: Generalization refers to the capability of an agent to perform in similar but unseen environments and is currently an active research challenge.

Original: Training deep RL algorithms is a data-intensive task and given a sufficiently large set of samples, they can learn a specific skill (Mnih et al., 2015; 2016; Haarnoja et al., 2018).
Revised: Training deep RL algorithms is a data-intensive task and given a sufficiently large set of samples, they can learn a specific skill (Mnih et al., 2015; 2016; Haarnoja et al., 2018).

Original: However, they tend to overfit even with large training samples (Cobbe et al., 2020; 2021;
Revised: However, they tend to overfit even with large training samples (Cobbe et al., 2020; 2021;

Original: Grigsby & Qi, 2020; Justesen et al., 2018).
Revised: Grigsby & Qi, 2020; Justesen et al., 2018).

Original: To facilitate research on this issue, newer benchmarks like Procgen (Cobbe 1 Anonymous Institution, Anonymous City, Anonymous Region, Anonymous Country. Correspondence to: Anonymous Author < anon.email@domain.com > .
Revised: To facilitate re- s).

Original: Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute. et al., 2020) have been introduced that present a set of procedurally generated environments to evaluate generalization through held-out levels used only in testing.
Revised: search on this issue, newer benchmarks like Procgen (Cobbe et al., 2020) have been introduced that present a set of procedurally generated environments to evaluate generalization through held-out levels used only in testing.

Original: In this work, we consider the problem of generalization in the context of such procedurally generated environments.
Revised: In this work, we consider the problem of generalization in the context of such procedurally generated environments.

Original: Standard RL algorithms specify a discount factor, 0 ≤ γ ≤ 1 , that discounts the future reward r t at step t as γ t r t (Sutton et al., 1998).
Revised: Standard RL algorithms specify a discount factor, 0 ≤ γ < 1 , that discounts the future reward r t at step t as γ t r t (Sutton et al., 1998).

Original: Such discounting guarantees the theoretical convergence of the value function and stabilize the optimization.
Revised: Such discounting guarantees the theoretical convergence of the value function and stabilizes the optimization.

Original: The value of γ sets a fixed effective horizon for an agent such that all rewards beyond that pointin time will be ignored.
Revised: The value of γ sets a fixed effective horizon for an agent such that all rewards beyond that point will be insignificant (Fedus et al., 2019).

Original: At the same time, the notion of  effective horizon induces a prior belief that there exists a known, fixed risk or hazard rate in the environment (Fedus et al., 2019).
Revised: At the same time, the notion of an effective horizon induces a prior belief that there exists a known, fixed risk or hazard rate in the environment (Fedus et al., 2019).

Original: We argue that this assumption does not comply with the set of diverse, ever-changing levels in a procedurally generated environment, wherein an environment’s dynamics andthe attributes change across levels, hence introducing significant uncertainty and stochasticity compared to the generic RL environments.
Revised: We argue that this assumption does not comply with the set of diverse, ever-changing levels in a procedurally generated environment, wherein an environment’s dynamics and attributes change across levels, hence introducing significant uncertainty and stochasticity compared to the generic RL environments.

Original: Figure 1 shows the distribution of episode length measured over 1000 episodes sampled randomly from the test levels for 8 Procgen environments.
Revised: Figure 1 shows the distribution of episode length measured over 1000 episodes sampled randomly from the test levels for 8 Procgen environments.

Original: The episodes were generated using a learned policy trained on 25M time steps.
Revised: The episodes were generated using a learned policy trained on 25M time steps.

Original: Even with such a mature policy, we see that the episode length varies significantly for each enviornment.
Revised: Even with such a mature policy, we see that the episode length varies significantly for each environment.

Original: This indicates the uncertainty over the hazard rate (survival rate) and why a fixed effective horizon may fail to better assess expected future rewards.
Revised: This indicates the uncertainty over the hazard rate (survival rate) and why a fixed effective horizon may fail to better assess expected future rewards.

Original: Existing literature indicates that when an agent holds uncertainty over the environment’s hazard rate, a non-exponential (such as hyperbolic) discounting factor is more suitable (Sozou, 1998; Fedus et al., 2019).
Revised: Existing literature indicates that when an agent holds uncertainty over the environment’s hazard rate, a non-exponential (such as hyperbolic) discounting factor is more suitable (Sozou, 1998; Fedus et al., 2019).

Original: Hyperbolic discounting of rewards also aligns with the time-preference of animals and humans (Mazur, 1997).
Revised: Hyperbolic discounting of rewards also aligns with the time-preference of animals and humans (Mazur, 1997).

Original: To address this uncertainty regarding an environment’s hazard rate and to mitigate the drawbacks of the fixed effective horizon, we propose to discount future rewards hyperbolically for generalization tasks and present our method
Revised: To address this uncertainty regarding an environment’s hazard rate and to mitigate the drawbacks of the fixed effective horizon, we propose to discount future rewards hyperbolically for generalization tasks and present our method

Original: HDGenRL (Hyperbolic Discounting for Generalization in
Revised: HDGenRL (Hyperbolic Discounting for Generalization in

Original: Reinforcement Learning).
Revised: Reinforcement Learning).

Original: We present a simple extension to calculate hyperbolically-discounted advantage estimates that can be used with policy gradient methods.
Revised: We present a simple extension to calculate hyperbolically-discounted advantage estimates that can be used with policy gradient methods.

Original: We evaluate our approach on all sixteen environments from the Procgen benchmark and our approach significantly outperforms the
Revised: We evaluate our approach on all sixteen environments from the Procgen benchmark and our approach significantly outperforms the

Original: Proximal Policy Optimzation (PPO) baseline that uses exponential discounting (Schulman et al., 2017).
Revised: Proximal Policy Optimzation (PPO) baseline that uses exponential discounting (Schulman et al., 2017).

Original: We further integrate our discounting scheme with Attention-based Partially Decoupled Actor-Critic (APDAC)
Revised: We further integrate our discounting scheme with Attention-based Partially Decoupled Actor-Critic (APDAC)

Original: (Nafi et al., 2021), a recent generalization specific method, and  results indicate the potential of our approach to be used with any existing method that employs classical exponential discounting.
Revised: (Nafi et al., 2021), a recent generalization specific method, and the results indicate the potential of our approach to be used with any existing method that employs classical exponential discounting.

Original: 2. Related Work
Revised: 2. Related Work

Original: Generalization in Deep RL.
Revised: Generalization in Deep RL.

Original: There has been a lot of emphasis lately on building intelligent agents that avoid overfitting and can generalize well to unseen data (Rajeswaran et al., 2017; Justesen et al., 2018; Grigsby & Qi, 2020; Cobbe et al., 2019).
Revised: There has been a lot of emphases lately on building intelligent agents that avoid overfitting and can generalize well to unseen data (Rajeswaran et al., 2017; Justesen et al., 2018; Grigsby & Qi, 2020; Cobbe et al., 2019).

Original: Methods that have been used to some success include regularization techniques like dropout, batch normalization,  data augmentation (Igl et al., 2019;
Revised: Methods that have been used with some success include regularization techniques like dropout, batch normalization, and data augmentation (Igl et al., 2019;

Original: Hu et al., 2021).
Revised: Hu et al., 2021).

Original: Raileanu & Fergus (2021) leverages fully separated policy and valuefunction networks to achieve generalization, while Cobbe et al.
Revised: Raileanu & Fergus (2021) leverages fully separated policy and value networks to achieve generalization, while Cobbe et al.

Original: (2021) introduces a phase-wise training.
Revised: (2021) introduce a phase-wise training.

Original: Hyperbolic Discounting.
Revised: Hyperbolic Discounting.

Original: Hyperbolic discounting has been studied in the fields of behavior psychology, economics, neuroscience and lately, to a limited extent, in reinforcement learning.
Revised: Hyperbolic discounting has been studied in the fields of behavioral psychology, economics, neuroscience, and lately, to a limited extent, in reinforcement learning.

Original: Sozou (1998) proposed a per-time-step death via the hazard rate, whereas Dasgupta & Maskin (2005) proposed that uncertainty over the timing of rewards lead to preference reversals as exhibited in hyperbolic discounting.
Revised: Sozou (1998) proposed a per-time-step death via the hazard rate, whereas Dasgupta & Maskin (2005) proposed that uncertainty over the timing of rewards leads to preference reversals as exhibited in hyperbolic discounting.

Original: Alexander & Brown (2010) proposed a temporal difference (TD) based hyperbolically discounting solution.
Revised: Alexander & Brown (2010) proposed a temporal difference (TD) based hyperbolically discounting solution.

Original: Although
Revised: Although

Original: TD learning relies on exponential discounting in its calculation, naive modifications to it to discount hyperbolically have been shown to be inconsistent.
Revised: TD learning relies on exponential discounting in its calculation, naive modifications to it to discount hyperbolically have been shown to be inconsistent.

Original: Kurth-Nelson & Redish (2009) proposed the modeling of hyperbolic discounting via distributed exponential discounting.
Revised: Kurth-Nelson & Redish (2009) proposed the modeling of hyperbolic discounting via distributed exponential discounting.

Original: Fedus et al.
Revised: Fedus et al.

Original: (2019) were the first to extend this formulation to deep reinforcement learning by approximating hyperbolic discounting from exponential discounting and evaluated their approach using a value-based method, Rainbow (Hessel et al., 2018), on the ALE benchmark.
Revised: (2019) were the first to extend this formulation to deep reinforcement learning by approximating hyperbolic discounting from exponential discounting and evaluated their approach using a value-based method, Rainbow (Hessel et al., 2018), on the ALE benchmark.

Original: 3. Background 3.1. Discount Factor and Effective Horizon
Revised: 3. Background 3.1. Discount Factor and Effective Horizon

Original: In standard RL algorithms, the agent’s objective is to maximize the sum of the discounted rewards over the future.
Revised: In standard RL algorithms, the agent’s objective is to maximize the sum of the discounted rewards over the future.

Original: Formally, the expected discounted return G t is:
Revised: Formally, the expected discounted return G t is:

Original: [Equation]
Revised: [Equation]

Original: where 0 ≤ γ ≤ 1 is called the discount factor (Sutton et al., 1998).
Revised: where 0 ≤ γ < 1 is called the discount factor (Sutton et al., 1998).

Original: The discount factor γ determines the current value of the future rewards.
Revised: The discount factor γ determines the current value of the future rewards.

Original: The horizon refers to how many steps into the future the agent takes into account and values the reward it receives.
Revised: The horizon refers to how many steps into the future the agent takes into account and values the reward it receives.

Original: If γ = 0 , then the agent becomes myopic and only maximizes immediate rewards.
Revised: If γ = 0 , then the agent becomes myopic and only maximizes immediate rewards.

Original: Conversely when γ −→ 1 , the agent values future rewards as well, making the agent far-sighted.
Revised: Conversely when γ −→ 1 , the agent values future rewards as well, making the agent far-sighted.

Original: Thus a particular value of γ sets a fixed effective horizon till which point the rewards are considered and beyond that the rewards are discarded.
Revised: A particular value of γ sets a fixed effective horizon till which point the rewards are considered.

Original: 3.2. Uncertain Hazard and Hyperbolic Discounting
Revised: 3.2. Uncertain Hazard and Hyperbolic Discounting

Original: An alternative perspective connects the discount factor to the notion of an agent not surviving to collect the reward due to encountering a risk or hazard (Sozou, 1998; Fedus et al., 2019).
Revised: An alternative perspective connects the discount factor to the notion of an agent not surviving to collect the reward due to encountering a risk or hazard (Sozou, 1998; Fedus et al., 2019).

Original: If s ( t ) is the probability that the agent survives until time t , then the present value of a future reward r t shall be discounted by the probability that the agent will survive till time t to collect it.
Revised: If s ( t ) is the probability that the agent survives until time t , then the present value of a future reward r t shall be discounted by the probability that the agent will survive till time t to collect it.

Original: Hence v ( r t ) =
Revised: Hence v ( r t ) =

Original: s ( t ) r t .
Revised: s ( t ) r t .

Original: The negative rate of change of the log-survival at time t is called the hazard rate h ( t ) :
Revised: The negative rate of change of the log-survival at time t is called the hazard rate h ( t ) :

Original: [Equation]
Revised: [Equation]

Original: By solving eq.
Revised: By solving eq.

Original: 2 as s ( t ) =
Revised: 2 as s ( t ) =

Original: e − λt , and setting s ( t )
Revised: e − λt , and setting s ( t )

Original: =
Revised: =

Original: γ t (exponential discounting) as in v ( r t ) =
Revised: γ t (exponential discounting) as in v ( r t ) =

Original: s ( t ) r t , the relation between hazard rate λ ∈ [0 , ∞ ] and γ ∈ [0 , 1] can be expressed as γ =
Revised: s ( t ) r t , the relation between hazard rate λ ∈ [0 , ∞ ] and γ ∈ [0 , 1] can be expressed as γ =

Original: e − λ .
Revised: e − λ .

Original: A single value of γ thus represents a known constant hazard rate λ = − ln( γ ) .
Revised: A single value of γ thus represents a known constant hazard rate λ = − ln( γ ) .

Original: However, to reflect the uncertainty in the hazard rate, we can relax this assumption of knowing the exact true hazard rate and replace it with a hazard prior p ( λ ) such that:s [Equation] t ) =
Revised: However, to reflect the uncertainty in the hazard rate, we can relax this assumption of knowing the exact true hazard rate and replace it with a hazard prior p ( λ ) such that:

Original: [Equation]
Revised: [Equation]

Original: [-∞ λ =
 p ( λ ) e − λt dλ.-] It is further shown by (Sozou, 1998) that for an exponential hazard prior p ( λ ) = 1 k e ( − λ/k ) , the survival rate of the agent becomes hyperbolic.
Revised: [-∞ λ =
 p ( λ ) e − λt dλ.-] It is further shown by (Sozou, 1998) that for an exponential hazard prior p ( λ ) = 1 k e ( − λ/k ) , the survival rate of the agent becomes hyperbolic.

Original: [Equation]
Revised: [Equation]

Original: where k is the hyperbolic exponent with value k > 0 .
Revised: where k is the hyperbolic exponent with value k > 0 .

Original: 4. POMDP Formulation
Revised: 4. POMDP Formulation

Original: The problem of generalization recognizes a distribution of
Revised: The problem of generalization consider a distribution of

Original: POMDP, p ( m ) where m ∈ M , and each instance m is defined by ( S m , O m , A , T m , R m , Ω m , γ m ) , where S m is the set of states, O m is the set of observations, A is the set of actions, T m ( s ′ | s, a ) are the transition probabilities, Ω m ( o | s ′ , a ) are the conditional observation probabilities, R m ( s, a ) is the reward function, and γ m is the discount factor sampled from the hazard distribution instead of a constant one.
Revised: POMDP, p ( m ) where m ∈ M , and each instance m is defined by ( S m , O m , A , T m , R m , Ω m , γ m ) , where S m is the set of states, O m is the set of observations, A is the set of actions, T m ( s ′ | s, a ) are the transition probabilities, Ω m ( o | s ′ , a ) are the conditional observation probabilities, R m ( s, a ) is the reward function, and γ m is the discount factor sampled from the hazard distribution instead of a constant one.

Original: Limited number of POMDPs are exposed during training.
Revised: A limited number of POMDPs are exposed during training, M train = { m 1 , m 2 , ..., m k } , where

Original: Let’s say we have M train = { m 1 , m 2 , ..., m n } ,where m i ∼ p and i ∈ { 1 , 2 , ..., n } , the goal is to optimize the policy π θ using the objective function J ( π θ ) =
Revised: M train ⊆ M , m i ∼ p and i ∈ { 1 , 2 , ..., k } . The goal is to optimize the policy π θ using the objective function J ( π θ ) =

Original: E p,π, T m (cid:2) (cid:80) Tt =

Revised: E p,π, T m (cid:2) (cid:80) Tt =

Original: γ t R m ( s t , a t ) (cid:3) over the full distribution of POMDPs.
Revised: γ t R m ( s t , a t ) (cid:3) over the full distribution of POMDPs.

Original: Each environment corresponds to a distribution of POMDPs, p ( m ) and each level of the game is analogous to a sampled POMDP instance.
Revised: Each environment corresponds to a distribution of POMDPs, p ( m ) and each level of the game is analogous to a sampled POMDP instance.

Original: The model is trained onlevels and tested on the full distribution of levels which is significantly larger than the training set.
Revised: The model is trained on 200 levels and tested on the full distribution of levels which is significantly larger than the training set.

Original: This enables the evaluation of the model’s generalization capability beyond those 200 training levels.
Revised: This enables the evaluation of the model’s generalization capability beyond those 200 training levels.

Original: 5. Methodology
Revised: 5. Methodology

Original: We aim to learn a value function and consequently a good advantage estimate to guide the policy optimization, in order to achieve a generalizable policy.
Revised: We aim to learn a value function and consequently a good advantage estimate to guide the policy optimization, in order to achieve a generalizable policy.

Original: We identify that hyperbolic discounting is more suitable than exponential discounting on the generalization task and propose to optimize the policy using hyperbolically discounted advantage estimate.
Revised: We identify that hyperbolic discounting is more suitable than exponential discounting on the generalization task and propose to optimize the policy using hyperbolically discounted advantage estimate.

Original: 5.1. Avoiding the Curse of Fixed Effective Horizon
Revised: 5.1. Avoiding the Curse of Fixed Effective Horizon

Original: As discussed in Section 3.1, a fixed discount factor in the case of exponential discounting can impose a single effective horizon for the agent.
Revised: As discussed in Section 3.1, a fixed discount factor in the case of exponential discounting can impose a single effective horizon for the agent.

Original: Thus the agent’s value function estimate relies on a prior belief about the length of the episode.
Revised: Thus the agent’s value function estimate relies on a prior belief about the length of the episode.

Original: Raileanu & Fergus (2021) demonstrate that even with the same starting state, the length of the episode can differ significantly.
Revised: Raileanu & Fergus (2021) demonstrate that even with the same starting state, the length of the episode can differ significantly.

Original: The episode length can significantly change the value estimate of the earlier states in a trajectory based on the later reward.
Revised: The episode length can significantly change the value estimate of the earlier states in a trajectory based on the later reward.

Original: For example, the final reward of an episode will be highly discounted and perceived as small if the episode length is too big.
Revised: For example, the final reward of an episode will be highly discounted and perceived as small if the episode length is too big.

Original: However, if the episode length is small, then the same final reward will contribute much more to the value estimate.
Revised: However, if the episode length is small, then the same final reward will contribute much more to the value estimate.

Original: Thus, due to the fixed effective horizon, an agent may fail to correctly anticipate the worth of future rewards in case of highly-varied episode length (see Figure 1).
Revised: Thus, due to the fixed effective horizon, an agent may fail to correctly anticipate the worth of future rewards in case of highly-varied episode length (see Figure 1).

Original: As we can not restrict the length of an episode, we propose to relax this fixed effective horizon by considering multiple horizons simultaneously.
Revised: As we can not restrict the length of an episode, we propose to relax this fixed effective horizon by considering multiple horizons simultaneously.

Original: Thus, we need a value estimate that considers multiple discount factors γ s while calculating the value estimate.
Revised: Thus, we need a value estimate that considers multiple discount factors γ s while calculating the value estimate.

Original: 5.2. Modelling the Unknown Hazard
Revised: 5.2. Modeling the Unknown Hazard

Original: Unseen levels in a highly diverse environment imply unknown hazard.
Revised: Unseen levels in a highly diverse environment imply unknown hazards.

Original: Since the agent is unsure about the environment’s hazard rate across unseen levels, we model this uncertainty by injecting a hazard distribution in the POMDP through the variable discount factor, γ m (see Section 4).
Revised: Since the agent is unsure about the environment’s hazard rate across unseen levels, we model this uncertainty by injecting a hazard distribution in the POMDP through the variable discount factor, γ m (see Section 4).

Original: Thus an episode from a new level comes with a hazard rate λ m , where γ m =
Revised: Thus an episode from a new level comes with a hazard rate λ m , where γ m =

Original: e − λ m .
Revised: e − λ m .

Original: Since an agent cannot accurately estimate the value of γ m for each new level in a model-free setup, and hyperbolic discounting is better able to capture this uncertain hazard, we use hyperbolically discounted advantage estimate to learn a robust policy for unseen levels.
Revised: Since an agent cannot accurately estimate the value of γ m for each new level in a model-free setup, and hyperbolic discounting is better able to capture this uncertain hazard, we use hyperbolically discounted advantage estimate to learn a robust policy for unseen levels.

Original: 5.3. Hyperbolically-discounted Advantage Estimation
Revised: 5.3. Hyperbolically-discounted Advantage Estimation

Original: We extend the basic idea of hyperbolic discounting-based value function to estimate hyperbolically-discounted advantage, so that it can be integrated with policy gradient methods.
Revised: We extend the basic idea of hyperbolic discounting-based value function to estimate hyperbolically-discounted advantage so that it can be integrated with policy gradient methods.

Original: Fedus et al.
Revised: Fedus et al.

Original: (2019) presents an approach that leverage multiple discount factors from  exponential discounting scheme to approximate the hyperbolic discounting function.
Revised: (2019) present an approach that leverages multiple discount factors from an exponential discounting scheme to approximate the hyperbolic discounting function.

Original: We adopt this approximation approach but instead use it for advantage estimation with respect to multiple effective horizons.
Revised: We adopt this approximation approach but instead use it for advantage estimation with respect to multiple effective horizons.

Original: Policy gradient objective for PPO (Schulman et al., 2017):
Revised: Policy gradient objective for PPO (Schulman et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where r t ( [Equation] ) =
Revised: where r t ( θ ) =

Original: [Equation] ( a [Equation] | s 
Revised: π ( θ ) ( a t | s t ) π ( θ ) old ( a t

Original: [Equation]

Original: θ [Equation] ( a t | s t ) , and ˆ A t is the estimation of the advantage function at timestep t .
Revised: | s t ) , and ˆ A t is the estimation of the advantage function at timestep t .

Original: Advantage is defined as
Revised: Advantage is defined as

Original: A ( s t , a t ) =

Original: Q
Revised: A

Original: [Equation]
Revised: [Equation]

Original: s t , a t )  − V ( s t ) .
Revised: s t , a t ) = Q ( s t , a t ) − V ( s t ) .

Original: Leveraging the hyperbolic function evaluation Γ k ( t ) = 1 1+ kt = (cid:82) 1 0 γ kt dγ , we propose to estimate hyperbolically-discounted advantage as follows:
Revised: Leveraging the hyperbolic function evaluation Γ k ( t ) = [Equation] +

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Revised: 1 0 γ kt dγ , we propose to estimate hyperbolically-discounted advantage as follows:

Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: ( s t , a t ) can be decomposed into r t +  V ( γ k ) t π ( s t +1 ) .
Revised: ( s t , a t ) can be decomposed into r t + γ k V ( γ k ) t π ( s t +1 ) .

Original: Thus based on the value function calculated over all the γ k where 0 ≤ γ ≤ 1 , we can estimate the hyperbolicallydiscounted advantage.
Revised: Thus based on the value function calculated over all the γ k where 0 ≤ γ < 1 , we can estimate the hyperbolicallydiscounted advantage.

Original: Note that the effective discount factor is γ k , not the original γ .
Revised: Note that the effective discount factor is γ k , not the original γ .

Original: From a practical perspective, we consider a finite set of γ (consequently γ k ) to approximate the advantage.
Revised: From a practical perspective, we consider a finite set of γ (consequently γ k ) to approximate the advantage.

Original: Using a multihead architecture, where each head corresponds to the value function for each γ k , we minimize the average of the losses calculated for these multiple γ k .
Revised: Using a multi-head architecture, where each head corresponds to the value function for each γ k , we minimize the average of the losses calculated for these multiple γ k .

Original: Each loss function corresponding to a γ k is defined as:
Revised: Each loss function corresponding to a γ k is defined as:

Original: [Equation]
Revised: [Equation]

Original: 6. Experiments and Results
Revised: 6. Experiments and Results

Original: Following previous works on Procgen, we use the IMPALACNN architecture as the actor-critic model for the PPO baseline in our experiments (Cobbe et al., 2020).
Revised: Following previous works on Procgen, we use the IMPALACNN architecture as the actor-critic model for the PPO baseline and implement generalized advantage estimate. (Cobbe et al., 2020).

Original: This CNN architecture has three similar blocks each with 5 convolutional layers.
Revised: This CNN architecture has three similar blocks each with 5 convolutional layers.

Original: To implement our proposed approach HDGenRL, we augment the same architecture with five value heads corresponding to five different γ .
Revised: To implement our proposed approach HDGenRL, we augment the same architecture with five value heads corresponding to five different γ .

Original: Then we calculate the advantage value for each of the value predictions.
Revised: Then we calculate the advantage value for each of the value predictions.

Original: Finally, we integrate these advantage values to obtain the approximation for the hyperbolic advantage.
Revised: Finally, we integrate these advantage values to obtain the approximation for the hyperbolic advantage.

Original: We train the model for 25M time steps.
Revised: We train the model for 25M time steps.

Original: Figure 2 shows the experimental results on the test distribution of levels for 8 out of the 16 environments from Procgen, and presents rolling mean test scores and standard deviations calculated over five trials.
Revised: Figure 2 shows the experimental results on the test distribution of levels for 8 out of the 16 environments from Procgen, and presents rolling mean test scores and standard deviations calculated over five trials.

Original: The results indicate that the proposed HDGenRL significantly outperforms the PPO baseline on the test levels.
Revised: The results indicate that the proposed HDGenRL significantly outperforms the PPO baseline on the test levels.

Original: Since PPO was not specifically designed for generalization, we further compare our approach against
Revised: Since PPO was not specifically designed for generalization, we further compare our approach against

Original: APDAC, a recent generalization-specific approach, to get a robust performance comparison on the generalization task.
Revised: APDAC, a recent generalization-specific approach, to get a robust performance comparison on the generalization task.

Original: Figure 3 shows that our proposed hyperbolic discounting based counterpart performs much better than APDAC on the test distribution of four selected games.
Revised: Figure 3 shows that our proposed hyperbolic discountingbased counterpart performs much better than APDAC on the test distribution of four selected games.

Original: 7. Discussion
Revised: 7. Discussion

Original: This work presents a hyperbolic discounting based method of estimating advantages and applies it on the generalization task.
Revised: This work presents a hyperbolic discounting-based method of estimating advantages and applies it on the generalization task.

Original: We argue that since the underlying hazard rate in a procedurally generated environment is more uncertain, having an agent that discount future rewards hyperbolically would perform better on unseen levels.
Revised: We argue that since the underlying hazard rate in a procedurally generated environment is more uncertain, having an agent that discounts future rewards hyperbolically would perform better on unseen levels.

Original: Through out the training, the agent learns the advantage estimate simultaneously over multiple horizons through the exponential discount factors γ 0 , γ 1 , ..., γ n , which has been shown to be an approximation of hyperbolic discounting.
Revised: Throughout the training, the agent learns the advantage estimate simultaneously over multiple horizons through the exponential discount factors γ 0 , γ 1 , ..., γ n , which has been shown to be an approximation of hyperbolic discounting.

Original: We evaluate our proposed method of hyperbolically discounted advantage estimation on PPO and APDAC, and  results show that the modified agent performs well on more than half of the tasks.
Revised: We evaluate our proposed method of hyperbolically discounted advantage estimation on PPO and APDAC, and the results show that the modified agent performs well on more than half of the tasks.

Original: We plan on extending this work by testing the proposed method against more recent state-of-the-art methods, varying number of discount factors, and other ablation studies.
Revised: We plan on extending this work by testing the proposed method against more recent state-of-the-art methods, varying number of discount factors, and other ablation studies.