Original: A DAPTING TO R EWARD P ROGRESSIVITY VIA S PECTRAL R EINFORCEMENT L EARNING
Revised: A DAPTING TO R EWARD P ROGRESSIVITY VIA S PECTRAL R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper we consider reinforcement learning tasks with progressive rewards ; that is, tasks where the rewards tend to increase in magnitude over time.
Revised: In this paper we consider reinforcement learning tasks with progressive rewards ; that is, tasks where the rewards tend to increase in magnitude over time.

Original: We hypothesise that this property may be problematic for value-based deep reinforcement learning agents, particularly if the agent must ﬁrst succeed in relatively unrewarding regions of the task in order to reach more rewarding regions.
Revised: We hypothesise that this property may be problematic for value-based deep reinforcement learning agents, particularly if the agent must ﬁrst succeed in relatively unrewarding regions of the task in order to reach more rewarding regions.

Original: To address this issue, we propose Spectral DQN , which decomposes the reward into frequencies such that the high frequencies only activate when large rewards are found.
Revised: To address this issue, we propose Spectral DQN , which decomposes the reward into frequencies such that the high frequencies only activate when large rewards are found.

Original: This allowsus to balance the training loss, so that it gives more even weighting to regions of the task with very different reward scales.
Revised: This allows the training loss to be balanced so that it gives more even weighting across small and large reward regions.

Original: In two domains with extreme reward progressivity, where standard value-based methods struggle signiﬁcantly, 
Revised: In two domains with extreme reward progressivity, where standard value-based methods struggle signiﬁcantly, Spectral

Original: Spectral DQN is able to make much farther progress.
Revised: DQN is able to make much farther progress.

Original: Moreover, when evaluated on a set of six standard Atari games that do not overtly favour the approach, Spectral DQN remains more than competitive: While it underperforms one of the benchmarks in a single game, it comfortably surpasses the benchmarks in three games.
Revised: Moreover, when evaluated on a set of six standard Atari games that do not overtly favour the approach, Spectral DQN remains more than competitive: While it underperforms one of the benchmarks in a single game, it comfortably surpasses the benchmarks in three games.

Original: This demonstrates that the approach is not overﬁt to its target problem, and suggests that reward progressivity has relevance beyond extreme domains.
Revised: These results demonstrate that the approach is not overﬁt to its target problem, and suggest that Spectral DQN may have advantages beyond addressing reward progressivity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In decision making tasks involving compounding returns, such as stock market investing, it is common for the rewards received by the agent to increase in magnitude over time.
Revised: In decision making tasks involving compounding returns, such as stock market investing, it is common for the rewards received by the agent to increase in magnitude over time.

Original: An investor that starts out with little capital will achieve relatively small proﬁts or losses early on, but if they are successful, their capital – and hence their potential proﬁts and losses – will gradually increase.
Revised: An investor that starts out with little capital will achieve relatively small proﬁts or losses early on, but if they are successful, their capital – and hence their potential proﬁts and losses – will gradually increase.

Original: This property also arises in many games settings.
Revised: This property also arises in many games settings.

Original: For example, in the Atari game Video Pinball , the player can increase a “bonus multiplier” that increases the score paid per bumper hit.
Revised: For example, in the Atari game Video Pinball , the player can increase a “bonus multiplier” that increases the score paid per bumper hit.

Original: Since the bonus multiplier does not reset until the player dies, the rewards typically increase with time.
Revised: Since the bonus multiplier does not reset until the player dies, the rewards typically increase with time.

Original: In this paper, we refer to any task exhibiting this phenomenon as a progressive reward task .
Revised: In this paper, we refer to any task exhibiting this phenomenon as a progressive reward task .

Original: We hypothesise that reward progressivity may be problematic for value-based deep reinforcement learning agents.
Revised: We hypothesise that reward progressivity may be problematic for value-based deep reinforcement learning agents.

Original: Our rationale is that the temporal difference errors that arise under such algorithms typically scale with the magnitude of the training targets.
Revised: Our rationale is that the temporal difference errors that arise under such algorithms typically scale with the magnitude of the training targets.

Original: Accordingly, experience collected from states with large expected returns may dominate training, causing the agent to unlearn information about other states.
Revised: Accordingly, experience collected from states with large expected returns may dominate training, causing the agent to unlearn information about other states.

Original: While this is rational in one sense – generally, it is more important to make accurate decisions in high-stakes situations – it is potentially harmful on progressive reward tasks.
Revised: While this is rational in one sense – generally, it is more important to make accurate decisions in high-stakes situations – it is potentially harmful on progressive reward tasks.

Original: This is because, on progressive reward tasks, it may be necessary for the agent to perform well in relatively unrewarding regions before it can reach more rewarding regions .
Revised: This is because, on progressive reward tasks, it may be necessary for the agent to perform well in relatively unrewarding regions before it can reach more rewarding regions .

Original: For example, in stock market investing, an investor can only reach states where they have large capital if they ﬁrst perform well with small capital.
Revised: For example, in stock market investing, an investor can only reach states where they have large capital if they ﬁrst perform well with small capital.

Original: Note too that the rewards need not be strictly progressive for this problem to arise; all that is needed is for the rewards to be progressive over some extended period.
Revised: Note too that the rewards need not be strictly progressive for this problem to arise; all that is needed is for the rewards to be progressive over some extended period.

Original: Algorithms that clip the reward, such as DQN (Mnih et al., 2015), are less susceptible to this problem, because they do not perceive increases in reward magnitude beyond the clipping point.
Revised: Algorithms that clip the reward, such as DQN (Mnih et al., 2015), are less susceptible to this problem, because they do not perceive increases in reward magnitude beyond the clipping point.

Original: However, it is straightforward to construct examples where reward clipping masks the optimal solution.
Revised: However, it is straightforward to construct examples where reward clipping masks the optimal solution.

Original: For example, in Bowling , clipping the rewards to [-1, 1] makes bowling a strike appear no better than hitting a single pin (Pohlen et al., 2018).
Revised: For example, in Bowling , clipping the rewards to [-1, 1] makes bowling a strike appear no better than hitting a single pin (Pohlen et al., 2018).

Original: While subsequent approaches make learning feasible without reward clipping (van Hasselt et al., 2016; Pohlen et al., 2018), we show in Section 3 that these methods are negatively impacted by strong reward progressivity.
Revised: While subsequent approaches make learning feasible without reward clipping (van Hasselt et al., 2016; Pohlen et al., 2018), we show in Section 3 that these methods are negatively impacted by strong reward progressivity.

Original: Motivated by this, we seek an algorithm that is capable of learning from unclipped rewards, while mitigating the negative impact that large rewards have on the agent’s ability to learn less rewarding regions of the task.
Revised: Motivated by this, we seek an algorithm that is capable of learning from unclipped rewards, while mitigating the negative impact that large rewards have on the agent’s ability to learn less rewarding regions of the task.

Original: To meet this aim, we propose Spectral DQN .
Revised: To meet this aim, we propose Spectral DQN .

Original: Under this approach, rewards are decomposed into a multidimensional spectral reward , where each component is bound to [-1, 1].
Revised: Under this approach, rewards are decomposed into a multidimensional spectral reward , where each component is bound to [-1, 1].

Original: The upper frequencies of the spectrum activate only when large magnitude rewards are received.
Revised: The upper frequencies of the spectrum activate only when large magnitude rewards are received.

Original: For example, in Video
Revised: For example, in Video

Original: Pinball , the lower frequencies activate on any score, while the upper frequencies only activate when the player has accumulated a large bonus multiplier.
Revised: Pinball , the lower frequencies activate on any score, while the upper frequencies only activate when the player has accumulated a large bonus multiplier.

Original: The bounded nature of the spectral rewards allows the agent to learn expected spectral returns without instability arising from large training targets.
Revised: The bounded nature of the spectral rewards allows the agent to learn expected spectral returns without instability arising from large training targets.

Original: Meanwhile, the full, unclipped expected return can be recovered by summing across the return spectrum.
Revised: Meanwhile, the full, unclipped expected return can be recovered by summing across the return spectrum.

Original: (The full, technical deﬁnitions of these terms are provided in Section 4).
Revised: (The full, technical deﬁnitions of these terms are provided in Section 4).

Original: In terms of addressing reward progressivity, a key advantage of this approach is that it allows us to balance the training loss and prevent the upper frequencies from receiving undue weight.
Revised: In terms of addressing reward progressivity, a key advantage of this approach is that it allows us to balance the training loss and prevent the upper frequencies from receiving undue weight.

Original: To test whether this approach helps mitigate the impact of reward progressivity, we perform two set of experiments.
Revised: To test whether this approach helps mitigate the impact of reward progressivity, we perform two set of experiments.

Original: First, we apply Spectral DQN to two domains that we speciﬁcally designed to exhibit strong reward progressivity.
Revised: First, we apply Spectral DQN to two domains that we speciﬁcally designed to exhibit strong reward progressivity.

Original: While previous methods struggle on these tasks, failing to learn almost anything at all in the more extreme domain, Spectral DQN performs markedly better, making considerable progress on both tasks.
Revised: While previous methods struggle on these tasks, failing to learn almost anything at all in the more extreme domain, Spectral DQN performs markedly better, making considerable progress on both tasks.

Original: Next, we apply our approach to a less constructed set of tasks; namely, 6 standard Atari games.
Revised: Next, we apply our approach to a less constructed set of tasks; namely, 6 standard Atari games.

Original: Only some of these games exhibit reward progressivity, and none of them nearly so strongly as the extreme domains from earlier.
Revised: Only some of these games exhibit reward progressivity, and none of them nearly so strongly as the extreme domains from earlier.

Original: While Spectral DQN does not outperform the benchmarks as noticeably in these domains (it is beaten by one of the benchmarks in a single game, but comfortably outperforms both benchmarks in three games), the results validate that Spectral DQN is not overﬁt to constructed domains.
Revised: While Spectral DQN does not outperform the benchmarks as noticeably in these domains (it is beaten by one of the benchmarks in a single game, but comfortably outperforms both benchmarks in three games), the results validate that Spectral DQN is not overﬁt to constructed domains.

Original: Moreover,  its clear edge in three games suggests that reward progressivity may be more than just a theoretical concern.
Revised: Moreover, since its clear edge in some of the games cannot be attributed solely to reward progressivity, it appears that the approach may offer additional advantages, as we explain later in our analysis of the results.

Original:  P REVIOUS A PPROACHES TO H ANDLING R EWARD V ARIABILITY
Revised:  P REVIOUS A PPROACHES TO H ANDLING R EWARD V ARIABILITY

Original: In the Atari domain, where the DQN algorithm was ﬁrst evaluated, the agent receives a reward equal to the score increase at each frame.
Revised: In the Atari domain, where the DQN algorithm was ﬁrst evaluated, the agent receives a reward equal to the score increase at each frame.

Original: However, since score magnitudes vary greatly across games,
Revised: However, since score magnitudes vary greatly across games,

Original: DQN clips all rewards to [ − 1 , +1] .
Revised: DQN clips all rewards to [ − 1 , +1] .

Original: This constrains the size of the training targets so that it is easier to ﬁnd a step size that performs well across the entire suite of games.
Revised: This constrains the size of the training targets so that it is easier to ﬁnd a step size that performs well across the entire suite of games.

Original: While this heuristic turns out to perform well in Atari , a clear downside of the approach is that the agent becomes unable to distinguish large rewards from small rewards.
Revised: While this heuristic turns out to perform well in Atari , a clear downside of the approach is that the agent becomes unable to distinguish large rewards from small rewards.

Original: The Pop-Art algorithm (van Hasselt et al., 2016) offers a more principled solution, by training from the unclipped reward, while normalising the training targets to have zero mean and unit variance.
Revised: The Pop-Art algorithm (van Hasselt et al., 2016) offers a more principled solution, by training from the unclipped reward, while normalising the training targets to have zero mean and unit variance.

Original: To ensure that the network’s predictions are preserved when the normalisation parameters are updated, the output of the ﬁnal layer is scaled and shifted by an offsetting amount.
Revised: To ensure that the network’s predictions are preserved when the normalisation parameters are updated, the output of the ﬁnal layer is scaled and shifted by an offsetting amount.

Original: Pohlen et al.
Revised: Pohlen et al.

Original: (2018) propose an alternative approach to handling unclipped rewards that reduces the variance of the training targets by applying a squashing function, h .
Revised: (2018) propose an alternative approach to handling unclipped rewards that reduces the variance of the training targets by applying a squashing function, h .

Original: The agent learns squashed action-values, ˜ Q ( s, a )
Revised: The agent learns squashed action-values, ˜ Q ( s, a )

Original: =
Revised: =

Original: h ( Q ( s, a )) , via the transformed Bellman backup:
Revised: h ( Q ( s, a )) , via the transformed Bellman backup:

Original: [Equation]
Revised: [Equation]

Original: The authors prove that the transformed Bellman operator remains a contraction provided that both h and h − 1 are Lipschitz continuous and γ < 1 /L
Revised: The authors prove that the transformed Bellman operator remains a contraction provided that both h and h − 1 are Lipschitz continuous and γ < 1 /L

Original: h L h − 1 (where L h and L h − 1 are the respective
Revised: h L h − 1 (where L h and L h − 1 are the respective

Original: Lipschitz constants).
Revised: Lipschitz constants).

Original: In their experiments, they use the following squashing function:
Revised: In their experiments, they use the following squashing function:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 ensures that h − 1 is Lipschitz continuous with L h − 1 = 1 /(cid:15) .
Revised: where (cid:15) > 0 ensures that h − 1 is Lipschitz continuous with L h − 1 = 1 /(cid:15) .

Original: For the remainder of this paper, we refer to this approach as target compression .
Revised: For the remainder of this paper, we refer to this approach as target compression .

Original: In the Atari domain,
Revised: In the Atari domain,

Original: Pohlen et al. found target compression to perform signiﬁcantly better than Pop-Art.
Revised: Pohlen et al. found target compression to perform signiﬁcantly better than Pop-Art.

Original: It has since become the dominant method for handling unclipped rewards in Atari , with several recent largescale agents (Kapturowski et al., 2019; Badia et al., 2020a;b), including the state-of-the-art Agent57 , favouring target compression over Pop-Art.
Revised: It has since become the dominant method for handling unclipped rewards in Atari , with several recent largescale agents (Kapturowski et al., 2019; Badia et al., 2020a;b), including the state-of-the-art Agent57 , favouring target compression over Pop-Art.

Original:  D EMONSTRATING THE P ROBLEM OF R EWARD P ROGRESSIVITY
Revised:  D EMONSTRATING THE P ROBLEM OF R EWARD P ROGRESSIVITY

Original: While both Pop-Art and target compression address reward variance across tasks, making it easier to devise a single learning conﬁguration that performs well across a task suite, they were not designed to address intra-task reward variability.
Revised: While both Pop-Art and target compression address reward variance across tasks, making it easier to devise a single learning conﬁguration that performs well across a task suite, they were not designed to address intra-task reward variability.

Original: As such, we posit that they are ill-suited to addressing reward progressivity, since it is an intra-task phenomenon.
Revised: As such, we posit that they are ill-suited to addressing reward progressivity, since it is an intra-task phenomenon.

Original: To test this claim, we introduce two domains that are designed to exhibit extreme reward progressivity, and thus exacerbate the issue:
Revised: To test this claim, we introduce two domains that are designed to exhibit extreme reward progressivity, and thus exacerbate the issue:

Original: Ponglantis and Exponential Pong .
Revised: Ponglantis and Exponential Pong .

Original: Ponglantis.
Revised: Ponglantis.

Original: In Ponglantis , the game starts out exactly the same as the Atari game Pong , where valued-based deep reinforcement learning methods are known to perform well (Mnih et al., 2015).
Revised: In Ponglantis , the game starts out exactly the same as the Atari game Pong , where valued-based deep reinforcement learning methods are known to perform well (Mnih et al., 2015).

Original: The reward scale in Pong is small relative to most Atari games; the player receives a +1 reward for hitting the ball past the computer paddle, and a -1 reward if they concede a point.
Revised: The reward scale in Pong is small relative to most Atari games; the player receives a +1 reward for hitting the ball past the computer paddle, and a -1 reward if they concede a point.

Original: In Ponglantis , however, if the player manages to score 10 points, the task immediately switches to Atlantis , where the reward scale is roughly 1000 times larger.
Revised: In Ponglantis , however, if the player manages to score 10 points, the task immediately switches to Atlantis , where the reward scale is roughly 1000 times larger.

Original: The Atlantis phase ends upon the lose of a single life.
Revised: The Atlantis phase ends upon the lose of a single life.

Original: Exponential Pong.
Revised: Exponential Pong.

Original: The game Exponential Pong is identical to regular Pong , except that the rewards are multiplied by 2 n , where n is the player’s current score.
Revised: The game Exponential Pong is identical to regular Pong , except that the rewards are multiplied by 2 n , where n is the player’s current score.

Original: We applied two agents to these games: Pop-Art and DQN+TC, where the latter is a version of DQN that we modiﬁed to use target compression and learn from the unclipped reward.
Revised: We applied two agents to these games: Pop-Art and DQN+TC, where the latter is a version of DQN that we modiﬁed to use target compression and learn from the unclipped reward.

Original: The results of this experiment, averaged over 5 seeds, are shown in Figure 1.
Revised: The results of this experiment, averaged over 5 seeds, are shown in Figure 1.

Original: (For full experimental details, see the
Revised: (For full experimental details, see the

Original: Appendix.)
Revised: Appendix.)

Original: To simplify analysis of the Exponential Pong results, we have plotted the unexponentiated scores, i.e. the scores that the agents would have achieved under standard scoring in Pong .
Revised: To simplify analysis of the Exponential Pong results, we have plotted the unexponentiated scores, i.e. the scores that the agents would have achieved under standard scoring in Pong .

Original: For context, the best possible score on standard Pong is +21, and the worst possible score is -21.
Revised: For context, the best possible score on standard Pong is +21, and the worst possible score is -21.

Original: In Exponential Pong , which exhibits the more extreme reward progressivity (since the reward multiplier can potentially reach 2 20 ≈ 1 million), both agents essentially fail, providing strong support for our hypothesis that reward progressivity can be damaging, at least in extreme cases.
Revised: In Exponential Pong , which exhibits the more extreme reward progressivity (since the reward multiplier can potentially reach 2 20 ≈ 1 million), both agents essentially fail, providing strong support for our hypothesis that reward progressivity can be damaging, at least in extreme cases.

Original: In the more moderate case of Ponglantis , Pop-Art fails to make meaningful progress beyond the low-scoring Pong phase of the task.
Revised: In the more moderate case of Ponglantis , Pop-Art fails to make meaningful progress beyond the low-scoring Pong phase of the task.

Original: The likely reason for this is that as soon as the agent starts reaching Atlantis and receiving large rewards, the target scaling mechanism employed by Pop-Art starts dividing the training targets by a large factor, heavily dampening the much smaller learning signals arising from the Pong phase of the task.
Revised: The likely reason for this is that as soon as the agent starts reaching Atlantis and receiving large rewards, the target scaling mechanism employed by Pop-Art starts dividing the training targets by a large factor, heavily dampening the much smaller learning signals arising from the Pong phase of the task.

Original: On the other hand, DQN+TC is eventually able to make some progress into the second, Atlantis phase.
Revised: On the other hand, DQN+TC is eventually able to make some progress into the second, Atlantis phase.

Original: The fact that DQN+TC outperforms PopArt is consistent with our hypothesis; while Pop-Art sees the reward scale jump by a factor of ≈ 1000 upon reaching Atlantis , the training targets of DQN+TC jump by a much smaller factor, owing to the square root squashing function (Equation 2).
Revised: The fact that DQN+TC outperforms PopArt is consistent with our hypothesis; while Pop-Art sees the reward scale jump by a factor of ≈ 1000 upon reaching Atlantis , the training targets of DQN+TC jump by a much smaller factor, owing to the square root squashing function (Equation 2).

Original: In Appendix B.2, we perform some additional experiments in Ponglantis to ﬁnd where the “cutoff point” is, i.e. how big the jump in reward magnitude has to be for Pop-Art and DQN+TC to deteriorate.
Revised: In Appendix B.2, we perform some additional experiments in Ponglantis to ﬁnd where the “cutoff point” is, i.e. how big the jump in reward magnitude has to be for Pop-Art and DQN+TC to deteriorate.

Original: Given the above discussion, a natural question to ponder is whether DQN+TC might achieve much better performance in Exponential Pong by employing a logarithmic squashing function instead of a square root.
Revised: Given the above discussion, a natural question to ponder is whether DQN+TC might achieve much better performance in Exponential Pong by employing a logarithmic squashing function instead of a square root.

Original: Unfortunately though, as Pohlen et al.
Revised: Unfortunately though, as Pohlen et al.

Original: (2018) point out to a reviewer of their work (see https://openreview.net/forum?id=BkfPnoActQ ), a logarithmic transform does not have a Lipschitz continuous inverse, and thus the transformed Bellman in this case is no longer guaranteed to be a contraction.
Revised: (2018) point out to a reviewer of their work (see https://openreview.net/forum?id=BkfPnoActQ ), a logarithmic transform does not have a Lipschitz continuous inverse, and thus the transformed Bellman in this case is no longer guaranteed to be a contraction.

Original: [Figure]
Revised: [Figure]

Original:  S PECTRAL R EINFORCEMENT L EARNING
Revised:  S PECTRAL R EINFORCEMENT L EARNING

Original: Based on the ﬁndings of the previous section, it seems that what we require is an algorithm where: (1) The training targets are constrained in magnitude, no matter how large the reward scale grows.
Revised: Based on the ﬁndings of the previous section, it seems that what we require is an algorithm where: (1) The training targets are constrained in magnitude, no matter how large the reward scale grows.

Original: (2) Regions of the task with small rewards yield meaningfully large temporal difference errors, even once much larger rewards have been discovered.
Revised: (2) Regions of the task with small rewards yield meaningfully large temporal difference errors, even once much larger rewards have been discovered.

Original: Note that Pop-Art fails to meet the second criterion, while target compression fails to meet both, albeit it takes larger shifts in reward magnitude for the problems associated with reward progressivity to become apparent.
Revised: Note that Pop-Art fails to meet the second criterion, while target compression fails to meet both, albeit it takes larger shifts in reward magnitude for the problems associated with reward progressivity to become apparent.

Original: In this section, we introduce a reinforcement learning approach that meets both of the above criteria by leveraging the idea of spectral decomposition.
Revised: In this section, we introduce a reinforcement learning approach that meets both of the above criteria by leveraging the idea of spectral decomposition.

Original: This algorithm does not in and of itself address reward progressivity; in fact, we show that it behaves identically to the standard Q-learning algorithm (Watkins, 1989), given certain caveats.
Revised: This algorithm does not in and of itself address reward progressivity; in fact, we show that it behaves identically to the standard Q-learning algorithm (Watkins, 1989), given certain caveats.

Original: However, the properties of this algorithm facilitate adaption to progressive rewards, and in Section 5 we present a deep learning version of the approach that is capable of learning in domains with extreme reward progressivity, such as Exponential Pong .
Revised: However, the properties of this algorithm facilitate adaption to progressive rewards, and in Section 5 we present a deep learning version of the approach that is capable of learning in domains with extreme reward progressivity, such as Exponential Pong .

Original: 4. S PECTRAL DECOMPOSITION
Revised: 4. S PECTRAL DECOMPOSITION

Original: Spectral decomposition is a broad term that arises in many different ﬁelds, but in this paper it refers to a speciﬁc method for decomposing real numbers into a sum of exponentially weighted components.
Revised: Spectral decomposition is a broad term that arises in many different ﬁelds, but in this paper it refers to a speciﬁc method for decomposing real numbers into a sum of exponentially weighted components.

Original: The intuition behind our method is illustrated in Figure 2.
Revised: The intuition behind our method is illustrated in Figure 2.

Original: In this example, the number being spectrally decomposed is 6.5.
Revised: In this example, the number being spectrally decomposed is 6.5.

Original: The method begins by dividing the number line into buckets of width b i , where b > 0 is the base of the decomposition and
Revised: The method begins by dividing the number line into buckets of width b i , where b > 0 is the base of the decomposition and

Original: i ∈
Revised: i ∈

Original: Z ≥ 0 .
Revised: Z ≥ 0 .

Original: Since the base here is 2, bucket 0 has width 1 and spans from 0 to 1, bucket 1 has width 2 and spans from 1 to 3, bucket 2 has width 4 and spans from 3 to 7, etc.
Revised: Since the base here is 2, bucket 0 has width 1 and spans from 0 to 1, bucket 1 has width 2 and spans from 1 to 3, bucket 2 has width 4 and spans from 3 to 7, etc.

Original: The idea is that each bucket corresponds to a different frequency in the spectrum.
Revised: The idea is that each bucket corresponds to a different frequency in the spectrum.

Original: The spectral decomposition is found by calculating the proportion of each bucket that is ﬁlled.
Revised: The spectral decomposition is found by calculating the proportion of each bucket that is ﬁlled.

Original: Since the number 6.5 completely ﬁlls the ﬁrst two buckets, then 87.5% of the next bucket, then none of the remaining buckets, the spectral decomposition of 6.5 to base 2 is (1, 1, 0.875, 0, 0, ...).
Revised: Since the number 6.5 completely ﬁlls the ﬁrst two buckets, then 87.5% of the next bucket, then none of the remaining buckets, the spectral decomposition of 6.5 to base 2 is (1, 1, 0.875, 0, 0, ...).

Original: More generally, let r be a non-negative real number, and let r i
Revised: More generally, let r be a non-negative real number, and let r i

Original: + b denote the i th element of r ’s spectral decomposition to base b .
Revised: + b denote the i th element of r ’s spectral decomposition to base b .

Original: The formula for calculating each element is:
Revised: The formula for calculating each element is:

Original: [Equation]
Revised: [Equation]

Original: (See the Appendix for the derivation of this formula.)
Revised: (See the Appendix for the derivation of this formula.)

Original: The value r can be recovered from the spectral decomposition by multiplying the bucket widths by the bucket ﬁll proportions and summing:
Revised: The value r can be recovered from the spectral decomposition by multiplying the bucket widths by the bucket ﬁll proportions and summing:

Original: [Equation]
Revised: [Equation]

Original: This method can be extended to negative numbers by negating the sign on each element of the representation.
Revised: This method can be extended to negative numbers by negating the sign on each element of the representation.

Original: Given any real number r , the formula for calculating each element then becomes:
Revised: Given any real number r , the formula for calculating each element then becomes:

Original: [Equation]
Revised: [Equation]

Original: 4. D ECOMPOSING THE R ETURN
Revised: 4. D ECOMPOSING THE R ETURN

Original: The core idea behind our reinforcement learning approach is to apply spectral decomposition to the reward stream, and thus learn a spectral decomposition of the expected return.
Revised: The core idea behind our reinforcement learning approach is to apply spectral decomposition to the reward stream, and thus learn a spectral decomposition of the expected return.

Original: Before delving into the technical details, see Table 1 for a worked example that highlights the intuition.
Revised: Before delving into the technical details, see Table 1 for a worked example that highlights the intuition.

Original: The discount
Revised: The discount

Original: Bucket number, i
Revised: Bucket number, i

Original: Filled amount  3.
Revised: Filled amount  3.

Original: Filled proportion  0.
Revised: Filled proportion  0.

Original: ⇒ Spectral representation of 6.5 = (1, 1, 0.875, 0, 0, 0, …)
Revised: ⇒ Spectral representation of 6.5 = (1, 1, 0.875, 0, 0, 0, …)

Original: Bucket width (=2 i )
Revised: Bucket width (=2 i )

Original: Spectral reward components
Revised: Spectral reward components

Original: [Table]
Revised: [Table]

Original: factor is γ = 0 . 99 , and the return over the ﬁrst ﬁve time steps is 2.254.
Revised: factor is γ = 0 . 99 , and the return over the ﬁrst ﬁve time steps is 2.254.

Original: The usual way of calculating the return is, of course, to multiply the rewards by γ t then sum.
Revised: The usual way of calculating the return is, of course, to multiply the rewards by γ t then sum.

Original: The alternative, spectral approach is to ﬁrst compute the spectral decomposition of the rewards according to Equation 5, then calculate the returns arising from each reward frequency.
Revised: The alternative, spectral approach is to ﬁrst compute the spectral decomposition of the rewards according to Equation 5, then calculate the returns arising from each reward frequency.

Original: We refer to the decomposed rewards as spectral rewards , and to the decomposed returns as spectral returns .
Revised: We refer to the decomposed rewards as spectral rewards , and to the decomposed returns as spectral returns .

Original: The overall return of 2.254 can be found by multiplying the spectral returns by their corresponding weights of b i then summing.
Revised: The overall return of 2.254 can be found by multiplying the spectral returns by their corresponding weights of b i then summing.

Original: To appreciate why we go to this extra effort in calculating the return, note that the spectral rewards are all bound to [-1, 1].
Revised: To appreciate why we go to this extra effort in calculating the return, note that the spectral rewards are all bound to [-1, 1].

Original: This in turn constrains the magnitude of the spectral returns to be no more than 1 / (1 - γ ), which, again, is useful from a function approximation standpoint.
Revised: This in turn constrains the magnitude of the spectral returns to be no more than 1 / (1 - γ ), which, again, is useful from a function approximation standpoint.

Original: Moreover, note that the relative sparsity of large rewards is made more apparent by this approach: As we progress from r 02 to r 32 , the magnitudes of the spectral reward components decrease, with only the two largest magnitude rewards triggering a response in r 32 .
Revised: Moreover, note that the relative sparsity of large rewards is made more apparent by this approach: As we progress from r 02 to r 32 , the magnitudes of the spectral reward components decrease, with only the two largest magnitude rewards triggering a response in r 32 .

Original: 4. S PECTRAL Q- LEARNING
Revised: 4. S PECTRAL Q- LEARNING

Original: To translate the above intuition into a concrete algorithm, we begin by deﬁning the spectral actionvalue function as:
Revised: To translate the above intuition into a concrete algorithm, we begin by deﬁning the spectral actionvalue function as:

Original: [Equation]
Revised: [Equation]

Original: where r ib,t denotes the i th element of the spectral reward at time t .
Revised: where r ib,t denotes the i th element of the spectral reward at time t .

Original: As shown in the Appendix, it is straightforward to prove via the linearity of expectation that the standard action-value function can be recovered as the weighted sum:
Revised: As shown in the Appendix, it is straightforward to prove via the linearity of expectation that the standard action-value function can be recovered as the weighted sum:

Original: [Equation]
Revised: [Equation]

Original: This leads to a natural extension of the Q-learning algorithm (Watkins, 1989), whereby the greedy action is found by maximising this expression with respect to the available actions, and Q π ( s, a, i ) is trained via the standard Bellman backup, except that the reward is replaced with the i th element of the spectral reward.
Revised: This leads to a natural extension of the Q-learning algorithm (Watkins, 1989), whereby the greedy action is found by maximising this expression with respect to the available actions, and Q π ( s, a, i ) is trained via the standard Bellman backup, except that the reward is replaced with the i th element of the spectral reward.

Original: For clarity, the full pseudocode for this algorithm, which we term Spectral
Revised: For clarity, the full pseudocode for this algorithm, which we term Spectral

Original: Q-learning , is provided in the Appendix (see Algorithm 1).
Revised: Q-learning , is provided in the Appendix (see Algorithm 1).

Original: An important practical limitation of this algorithm is that it is impossible to train over full, inﬁnite spectrum; that is, one must choose a ﬁnite upper bound for i .
Revised: An important practical limitation of this algorithm is that it is impossible to train over full, inﬁnite spectrum; that is, one must choose a ﬁnite upper bound for i .

Original: (This would not be an issue if we knew the maximum reward magnitude in advance, but for generality we do not wish to assume this.)
Revised: (This would not be an issue if we knew the maximum reward magnitude in advance, but for generality we do not wish to assume this.)

Original: However, for base b , observe that the ﬁrst N + 1 frequencies fully capture all rewards up to magnitude ( b N +1 − 1) / ( b − 1) , due to the fact that the bucket widths form a geometric series.
Revised: However, for base b , observe that the ﬁrst N + 1 frequencies fully capture all rewards up to magnitude ( b N +1 − 1) / ( b − 1) , due to the fact that the bucket widths form a geometric series.

Original: Given the exponential nature of this term, it is possible to capture large magnitude rewards with a relatively small value of N .
Revised: Given the exponential nature of this term, it is possible to capture large magnitude rewards with a relatively small value of N .

Original: For example, with b = 2 and N = 20 , the algorithm fully captures all rewards of magnitude up to 1 million, which is more than sufﬁcient for Atari .
Revised: For example, with b = 2 and N = 20 , the algorithm fully captures all rewards of magnitude up to 1 million, which is more than sufﬁcient for Atari .

Original: Taking this idea one step further, we arrive at the following proposition, which is proved in the Appendix:
Revised: Taking this idea one step further, we arrive at the following proposition, which is proved in the Appendix:

Original: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .
Revised: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .

Original: Now that we have presented the full algorithm, it is worth reﬂecting on why we use a spectral representation for the rewards instead of, say, a binary encoding.
Revised: Now that we have presented the full algorithm, it is worth reﬂecting on why we use a spectral representation for the rewards instead of, say, a binary encoding.

Original: Put simply, very small differences in reward magnitude should not matter much to a rational agent.
Revised: Put simply, very small differences in reward magnitude should not matter much to a rational agent.

Original: For example, it should not matter much whether a particular reward’s magnitude is 63 or 64.
Revised: For example, it should not matter much whether a particular reward’s magnitude is 63 or 64.

Original: However, under a binary encoding,is represented as 0111111, while 64 is represented as 1000000.
Revised: However, under a binary encoding,is represented as 0111111, while 64 is represented as 1000000.

Original: As such, a reward of 63 would trigger a larger overall TD error at the start of training, compared to a reward of 64, under Spectral
Revised: As such, a reward of 63 would trigger a larger overall TD error at the start of training, compared to a reward of 64, under Spectral

Original: Q-learning.
Revised: Q-learning.

Original: The thermometer-style encoding introduced earlier addresses this problem.
Revised: The thermometer-style encoding introduced earlier addresses this problem.

Original:  S PECTRAL DQN
Revised:  S PECTRAL DQN

Original: In order to create a deep learning version of the Spectral Q-learning algorithm, we modify the
Revised: In order to create a deep learning version of the Spectral Q-learning algorithm, we modify the

Original: DQN algorithm in an analogous way to which we modiﬁed the Q-learning algorithm in the previous section.
Revised: DQN algorithm in an analogous way to which we modiﬁed the Q-learning algorithm in the previous section.

Original: We name the resultant algorithm Spectral DQN .
Revised: We name the resultant algorithm Spectral DQN .

Original: To parameterise all N frequencies of the spectral action-value function via a single network, we multiply the number of outputs on the standard DQN network by N .
Revised: To parameterise all N frequencies of the spectral action-value function via a single network, we multiply the number of outputs on the standard DQN network by N .

Original: One crucial detail here is that we initialise the weights and biases of the ﬁnal layer to zero.
Revised: One crucial detail here is that we initialise the weights and biases of the ﬁnal layer to zero.

Original: This allows us to set N to a conservatively large value, since it ensures that superﬂuous heads have no effect on training.
Revised: This allows us to set N to a conservatively large value, since it ensures that superﬂuous heads have no effect on training.

Original: Moreover, it ensures that the action-value calculations do not blow out due to the exponential weights in Equation 7 being applied to random noise.
Revised: Moreover, it ensures that the action-value calculations do not blow out due to the exponential weights in Equation 7 being applied to random noise.

Original: For the training loss, we use a weighted sum of the temporal difference errors for each frequency:
Revised: For the training loss, we use a weighted sum of the temporal difference errors for each frequency:

Original: [Equation]
Revised: [Equation]

Original: (8) where w i is the error weight given to the i th frequency, s is a state sampled uniformly from the replay memory, a is the sampled action, s (cid:48) is the next state, r is the reward (with spectral components r ib for i ∈ { 0 , . . .
Revised: (8) where w i is the error weight given to the i th frequency, s is a state sampled uniformly from the replay memory, a is the sampled action, s (cid:48) is the next state, r is the reward (with spectral components r ib for i ∈ { 0 , . . .

Original: , N } ), θ are the parameters of the network being trained, and θ − are the parameters of the periodically refreshed target network.
Revised: , N } ), θ are the parameters of the network being trained, and θ − are the parameters of the periodically refreshed target network.

Original: An important detail here is the approach we take to weighting the loss for each frequency.
Revised: An important detail here is the approach we take to weighting the loss for each frequency.

Original: Given that the ith frequency is weighted by b i when calculating the overall action-values, it is tempting to set w i
Revised: Given that the ith frequency is weighted by b i when calculating the overall action-values, it is tempting to set w i

Original: = b i .
Revised: = b i .

Original: However, this goes completely against one of the aims of our approach, which is to prevent the large reward frequencies from dominating the loss.
Revised: However, this goes completely against one of the aims of our approach, which is to prevent the large reward frequencies from dominating the loss.

Original: In fact, we show in the next section that weighting by b i has a disastrous impact in the two extreme domains introduced earlier, Exponential
Revised: In fact, we show in the next section that weighting by b i has a disastrous impact in the two extreme domains introduced earlier, Exponential

Original: Pong and Ponglantis .
Revised: Pong and Ponglantis .

Original: In light of the above, rather than weighting by b
Revised: In light of the above, rather than weighting by b

Original: i , we strive to give all frequencies equal weighting.
Revised: i , we strive to give all frequencies equal weighting.

Original: However, we do not simply set w i = 1 either, because doing so would effectively give the small reward frequencies too much importance.
Revised: However, we do not simply set w i = 1 either, because doing so would effectively give the small reward frequencies too much importance.

Original: As pointed out by van Hasselt et al.
Revised: As pointed out by van Hasselt et al.

Original: (see Section 2.3 of their work), the gradients propagated to the Q-network’s hidden layers depend quadratically on the scale of the targets.
Revised: (see Section 2.3 of their work), the gradients propagated to the Q-network’s hidden layers depend quadratically on the scale of the targets.

Original: Since the lower frequencies have denser rewards, and hence larger training targets, they will dominate the gradients at the hidden layers if this effect is not accounted for.
Revised: Since the lower frequencies have denser rewards, and hence larger training targets, they will dominate the gradients at the hidden layers if this effect is not accounted for.

Original: To address this, we set w i = 

Revised: To address this, we set w i =

Original: /σ 

Revised: /σ

Original: i , where σ i is the standard deviation of the training targets for frequency
Revised: i , where σ i is the standard deviation of the training targets for frequency

Original: i , calculated as a running average as in the Pop-Art algorithm.
Revised: i , calculated as a running average as in the Pop-Art algorithm.

Original: Finally, to ensure that this correction only affects the hidden layers, we unscale the gradients at the ﬁnal layer to train as if w i = 1 , as in standard DQN.
Revised: Finally, to ensure that this correction only affects the hidden layers, we unscale the gradients at the ﬁnal layer to train as if w i = 1 , as in standard DQN.

Original: (This is not especially difﬁcult; the precise details can be found in the provided source code.)
Revised: (This is not especially difﬁcult; the precise details can be found in the provided source code.)

Original: In Appendix B.1, we provide an ablation where we do not correct the hidden layer gradients, and ﬁnd that there is a signiﬁcant negative impact on performance.
Revised: In Appendix B.1, we provide an ablation where we do not correct the hidden layer gradients, and ﬁnd that there is a signiﬁcant negative impact on performance.

Original:  R ETURNING TO THE E XTREME D OMAINS
Revised:  R ETURNING TO THE E XTREME D OMAINS

Original: To test how well Spectral DQN meets our aims, we ﬁrst applied it to the two extreme domains from Section 3 – Ponglantis and Exponential Pong – where reward progressivity is clearly the key issue.
Revised: To test how well Spectral DQN meets our aims, we ﬁrst applied it to the two extreme domains from Section 3 – Ponglantis and Exponential Pong – where reward progressivity is clearly the key issue.

Original: In addition, to probe the importance of the frequency loss weighting, we included a variant of Spectral DQN with exponential loss weighting, i.e. w i
Revised: In addition, to probe the importance of the frequency loss weighting, we included a variant of Spectral DQN with exponential loss weighting, i.e. w i

Original: =
Revised: =

Original: b i , which we argued against in the previous section.
Revised: b i , which we argued against in the previous section.

Original: The results of this experiment, including the baseline results from Section 3, are shown in Figure 3, with each curve averaged over 5 seeds.
Revised: The results of this experiment, including the baseline results from Section 3, are shown in Figure 3, with each curve averaged over 5 seeds.

Original: There are two clear ﬁndings here: (1) Spectral DQN is much better at handling extreme reward progressivity than the baselines, scoring much higher in Ponglantis , and making far more meaningful
Revised: There are two clear ﬁndings here: (1) Spectral DQN is much better at handling extreme reward progressivity than the baselines, scoring much higher in Ponglantis , and making far more meaningful

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: progress in Exponential Pong , eventually reaching a higher average score than the computer player.
Revised: progress in Exponential Pong , eventually reaching a higher average score than the computer player.

Original: (2) The loss weighting term is of crucial importance, with the agent using exponential weights performing worse than even DQN+TC.
Revised: (2) The loss weighting term is of crucial importance [Figure] with the agent using exponential weights performing worse than even DQN+TC.

Original: The latter ﬁnding matches our expectations; as we noted in Section 3, target compression does at least somewhat mitigate the overemphasis given to large reward regions of the task, while Spectral DQN with w i =
Revised: The latter ﬁnding matches our expectations; as we noted in Section 3, target compression does at least somewhat mitigate the overemphasis given to large reward regions of the task, while Spectral DQN with w i =

Original: b i gives heavy emphasis to such regions
Revised: b i gives heavy emphasis to such regions.

Original: [Figure]

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: we examine how the agents’ training errors evolve as they play Exponential Pong .
Revised: we examine how the agents’ training errors evolve as they play Exponential Pong .

Original: The average training errors are broken down by the player’s current score ∈ [0 , 20] , with bluer lines corresponding to states with a low player score and redder lines corresponding to states with high player scores.
Revised: The average training errors are broken down by the player’s current score ∈ [0 , 20] , with bluer lines corresponding to states with a low player score and redder lines corresponding to states with high player scores.

Original: To make the graphs more comparable, we plot percentage TD errors, i.e.: mean
Revised: To make the graphs more comparable, we plot percentage TD errors, i.e.: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the means are running averages, calculated from states sampled from the replay buffer.
Revised: where the means are running averages, calculated from states sampled from the replay buffer.

Original: In all cases we report the true TD error, i.e. for DQN+TC we report the error on the uncompressed action-values, and for Spectral DQN we report the error on the full, summed action-values.
Revised: In all cases we report the true TD error, i.e. for DQN+TC we report the error on the uncompressed action-values, and for Spectral DQN we report the error on the full, summed action-values.

Original: The instability of Pop-Art is quite apparent here: After a very brief period of stable learning, the TD errors start spiking as soon as the player reaches even middling scores.
Revised: The instability of Pop-Art is quite apparent here: After a very brief period of stable learning, the TD errors start spiking as soon as the player reaches even middling scores.

Original: By comparison, DQN+TC appears a lot more stable.
Revised: By comparison, DQN+TC appears a lot more stable.

Original: For states with low player scores, the TD percentage errors hover at around 5%, although the agent never really learns accurate action-values for states where the player score exceeds ≈ 7 points.
Revised: For states with low player scores, the TD percentage errors hover at around 5%, although the agent never really learns accurate action-values for states where the player score exceeds ≈ 7 points.

Original: At ﬁrst this might seem strange – since the absolute TD errors increase with the player’s score, states with a high player score ought to be effectively prioritised by the loss function.
Revised: At ﬁrst this might seem strange – since the absolute TD errors increase with the player’s score, states with a high player score ought to be effectively prioritised by the loss function.

Original: However, there is an additional effect to consider here: As soon the agent’s predictions start to deteriorate for low-score states, its policy for these states likewise deteriorates and it can no longer reach the higher-score states.
Revised: However, there is an additional effect to consider here: As soon the agent’s predictions start to deteriorate for low-score states, its policy for these states likewise deteriorates and it can no longer reach the higher-score states.

Original: As such, the agent does not encounter the high-score states often enough to learn accurate values there, nor for the estimates in low-score states to truly deteriorate.
Revised: As such, the agent does not encounter the high-score states often enough to learn accurate values there, nor for the estimates in low-score states to truly deteriorate.

Original: It appears that a TD percentage error of around 5% is a crucial cutoff line, where the agent is right on the cusp of being able to play well, but playing better, conversely, leads to deterioration.
Revised: It appears that a TD percentage error of around 5% is a crucial cutoff line, where the agent is right on the cusp of being able to play well, but playing better, conversely, leads to deterioration.

Original: In contrast to both Pop-Art and DQN+TC, when Spectral DQN reaches a new high score, it quickly learns accurate action-value estimates for the associated states, as indicated by the sharply dropping red lines in Figure 4.
Revised: In contrast to both Pop-Art and DQN+TC, when Spectral DQN reaches a new high score, it quickly learns accurate action-value estimates for the associated states, as indicated by the sharply dropping red lines in Figure 4.

Original: Moreover, the TD errors for low-score states remain stable as the agent improves, almost certainly because Spectral DQN’s loss does not give so much weight to large magnitude rewards.
Revised: Moreover, the TD errors for low-score states remain stable as the agent improves, almost certainly because Spectral DQN’s loss does not give so much weight to large magnitude rewards.

Original: Finally, note that the agent’s sudden rise in performance seems to coincide with the errors reducing beyond the 5% line, in accordance with the analysis above.
Revised: Finally, note that the agent’s sudden rise in performance seems to coincide with the errors reducing beyond the 5% line, in accordance with the analysis above.

Original:  E XPERIMENTS IN S TANDARD A TARI G AMES
Revised:  E XPERIMENTS IN S TANDARD A TARI G AMES

Original: To test whether Spectral DQN’s success in extreme domains comes at the expense of performance in less constructed tasks, we also applied it to 6 standard Atari games.
Revised: To test whether Spectral DQN’s success in extreme domains comes at the expense of performance in less constructed tasks, we also applied it to 6 standard Atari games.

Original: Moreover, to validate that its earlier success was not attributable to it performing some implicit form of reward clipping, we included 3 games where reward clipping is known to be detrimental, based on the fact that Pop-Art strongly outperforms DQN (van Hasselt et al., 2016): Bowling , Centipede and Ms. Pacman .
Revised: Moreover, to validate that its earlier success was not attributable to it performing some implicit form of reward clipping, we included 3 games where reward clipping is known to be detrimental, based on the fact that Pop-Art strongly outperforms DQN (van Hasselt et al., 2016): Bowling , Centipede and Ms. Pacman .

Original: The results of these experiments are shown in Figure 5, with all curves averaged over 5 seeds.
Revised: The results of these experiments are shown in Figure 5, with all curves averaged over 5 seeds.

Original: While Spectral DQN’s performance relative to the baselines is not as exceptional here as in the extreme domains, it remains the strongest agent overall.
Revised: While Spectral DQN’s performance relative to the baselines is not as exceptional here as in the extreme domains, it remains the strongest agent overall.

Original: Though it is beaten by DQN+TC in Centipede , it noticeably outperforms both baselines in Bowling , Chopper Command and Video Pinball , and is at least on par with the benchmarks in the other games.
Revised: Though it is beaten by DQN+TC in Centipede , it noticeably outperforms both baselines in Bowling , Chopper Command and Video Pinball , and is at least on par with the benchmarks in the other games.

Original: Moreover, the fact that its performance compares favourably to Pop-Art in Bowling , Centipede and Ms. Pacman shows that it is truly responding to the unclipped reward.
Revised: Moreover, the fact that its performance compares favourably to Pop-Art in Bowling , Centipede and Ms. Pacman shows that it is truly responding to the unclipped reward.

Original: Since Video Pinball exhibits reward progressivity, as explained in the introduction, Spectral DQN’s strong performance there ﬁts well with our hypothesis.
Revised: Since Video Pinball exhibits reward progressivity, as explained in the introduction, Spectral DQN’s strong performance there ﬁts well with our hypothesis.

Original: On the other hand, while Ms. Pacman also has progressive rewards – the player receives an increasing bonus for eating multiple ghosts with a single power pellet – these rewards come so close together that they are within the agent’s temporal horizon at the start of an episode.
Revised: On the other hand, while Ms. Pacman also has progressive rewards – the player receives an increasing bonus for eating multiple ghosts with a single power pellet – these rewards come so close together that they are within the agent’s temporal horizon at the start of an episode.

Original: In other words, there is not a small-expected-return phase preceding a large-expected-return phase, which is a central component of the issue.
Revised: In other words, there is not a small-expected-return phase preceding a large-expected-return phase, which is a central component of the issue.

Original: (We discuss this further in the next section.)
Revised: (We discuss this further in the next section.)

Original: Besides addressing reward progressivity, another possible advantage of Spectral DQN in high scoring games is simply that its training targets are more constrained.
Revised: Besides addressing reward progressivity, another possible advantage of Spectral DQN in high scoring games is simply that its training targets are more constrained.

Original: In games such as Video Pinball and Chopper Command , the expected returns can reach into the thousands, so even with target compression this may be problematic for DQN+TC.
Revised: In games such as Video Pinball and Chopper Command , the expected returns can reach into the thousands, so even with target compression this may be problematic for DQN+TC.

Original: On the other hand, the only game where DQN+TC clearly outperforms Spectral DQN is Centipede , which is also high scoring.
Revised: On the other hand, the only game where DQN+TC clearly outperforms Spectral DQN is Centipede , which is also high scoring.

Original: One possible explanation for this is that the reward structure in Centipede “confuses” Spectral DQN: Since large rewards also trigger low frequencies, it is likely that Spectral DQN learns a positive association between the various components of the return.
Revised: One possible explanation for this is that the reward structure in Centipede “confuses” Spectral DQN: Since large rewards also trigger low frequencies, it is likely that Spectral DQN learns a positive association between the various components of the return.

Original: However, in Centipede , losing a life triggers a stream of consecutive small rewards.
Revised: However, in Centipede , losing a life triggers a stream of consecutive small rewards.

Original: Therefore, associating small and large rewards may cause Spectral DQN to predict an inﬂated return from dying.
Revised: Therefore, associating small and large rewards may cause Spectral DQN to predict an inﬂated return from dying.

Original: There has been some conjecture that one advantage of combining distributional RL with function approximation is that it forces the agent to learn a richer representation (Bellemare et al., 2017;
Revised: There has been some conjecture that one advantage of combining distributional RL with function approximation is that it forces the agent to learn a richer representation (Bellemare et al., 2017;

Original: Lyle et al., 2019).
Revised: Lyle et al., 2019).

Original: The same argument may also apply to our approach.
Revised: The same argument may also apply to our approach.

Original: In particular, in some tasks, small rewards are of more informational value than their face value represents.
Revised: In particular, in some tasks, small rewards are of more informational value than their face value represents.

Original: For example, in Video
Revised: For example, in Video

Original: [Figure]
Revised: [Figure]

Original: Pinball , where rewards range into the thousands, the play ﬁeld includes “spinners” that pay only one point per hit.
Revised: Pinball , where rewards range into the thousands, the play ﬁeld includes “spinners” that pay only one point per hit.

Original: Despite their negligible face value, hitting the spinners can dramatically change the velocity of the ball, so giving the lower frequencies more weight in the training loss may promote a stronger representation.
Revised: Despite their negligible face value, hitting the spinners can dramatically change the velocity of the ball, so giving the lower frequencies more weight in the training loss may promote a stronger representation.

Original: Similarly, in Bowling , certain frequencies only activate after bowling strikes and spares.
Revised: Similarly, in Bowling , certain frequencies only activate after bowling strikes and spares.

Original: Since this game does not have a strongly progressive reward structure, improved representation learning is a more plausible explanation of Spectral DQN’s outperformance.
Revised: Since this game does not have a strongly progressive reward structure, improved representation learning is a more plausible explanation of Spectral DQN’s outperformance.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this paper, we brought attention to the idea of reward progressivity, a task property that poses a potential issue for value-based reinforcement learning agents.
Revised: In this paper, we brought attention to the idea of reward progressivity, a task property that poses a potential issue for value-based reinforcement learning agents.

Original: In particular, it can cause an agent’s performance to degrade on relatively unrewarding regions of a task once it discovers large rewards.
Revised: In particular, it can cause an agent’s performance to degrade on relatively unrewarding regions of a task once it discovers large rewards.

Original: To address this, we proposed Spectral DQN, which decomposes the expected return magnitudinally, allowing the losses arising from regions of the task with vastly different reward scales to be balanced.
Revised: To address this, we proposed Spectral DQN, which decomposes the expected return magnitudinally, allowing the losses arising from regions of the task with vastly different reward scales to be balanced.

Original: Our experiments showed this approach to be particularly effective in domains exhibiting extreme reward progressivity, while still yielding more than competitive performance elsewhere.
Revised: Our experiments showed this approach to be particularly effective in domains exhibiting extreme reward progressivity, while still yielding more than competitive performance elsewhere.

Original: Our discussion of the Ms. Pacman results in the previous section raised an interesting point: There is clearly a link between reward progressivity and discounting.
Revised: Our discussion of the Ms. Pacman results in the previous section raised an interesting point: There is clearly a link between reward progressivity and discounting.

Original: If it were somehow possible to train reliably with γ = 1, a Ponglantis agent would be able to see that reaching the distant, large rewards in the Atlantis phase is contingent on ﬁrst performing well in Pong .
Revised: If it were somehow possible to train reliably with γ = 1, a Ponglantis agent would be able to see that reaching the distant, large rewards in the Atlantis phase is contingent on ﬁrst performing well in Pong .

Original: However, despite some recent progress, DQN-style agents typically destabilise with discounts very close to 1.
Revised: However, despite some recent progress, DQN-style agents typically destabilise with discounts very close to 1.

Original: Agent (Badia et al., 2020a) trains multiple value functions across a range of discounts, ranging as high as γ = 0 . 9999 .
Revised: Agent (Badia et al., 2020a) trains multiple value functions across a range of discounts, ranging as high as γ = 0 . 9999 .

Original: However, the authors note that γ = 0 .
Revised: However, the authors note that γ = 0 .

Original: 9999 generally yields very unstable learning, and they rely on a meta-controller to select smaller discounts in games where learning destabilises.
Revised: 9999 generally yields very unstable learning, and they rely on a meta-controller to select smaller discounts in games where learning destabilises.

Original: It is also worth noting a link to the issue of reward sparsity here: In a progressive reward task, if the agent is cognizant of large, distant rewards, then the smaller, more immediate rewards may appear so negligible that the task effectively becomes a sparse reward problem.
Revised: It is also worth noting a link to the issue of reward sparsity here: In a progressive reward task, if the agent is cognizant of large, distant rewards, then the smaller, more immediate rewards may appear so negligible that the task effectively becomes a sparse reward problem.

Original: We noted in Section 3 that another idea for addressing reward progressivity is to apply target compression, but with logarithmic squashing instead of a square root.
Revised: We noted in Section 3 that another idea for addressing reward progressivity is to apply target compression, but with logarithmic squashing instead of a square root.

Original: However, as Pohlen et al.
Revised: However, as Pohlen et al.

Original: (2018) note, the inverse of a logarithmic mapping is not Lipschitz continuous, and thus the transformed
Revised: (2018) note, the inverse of a logarithmic mapping is not Lipschitz continuous, and thus the transformed

Original: Bellman is no longer guaranteed to be a contraction.
Revised: Bellman is no longer guaranteed to be a contraction.

Original: On the other hand, one way to get around poor convergence in practice is to use a shaper discount factor.
Revised: On the other hand, one way to get around poor convergence in practice is to use a shaper discount factor.

Original: In fact, Van Seijen et al.
Revised: In fact, Van Seijen et al.

Original: (2019) show that it is possible to achieve performance comparable to DQN via a logarithmic squash, provided the discount is small ( γ = 0.96).
Revised: (2019) show that it is possible to achieve performance comparable to DQN via a logarithmic squash, provided the discount is small ( γ = 0.96).

Original: However, they clip the rewards, which constrains the gradient of the exponential inverse mapping and thus limits the degree to which their approach transgresses the theory.
Revised: However, they clip the rewards, which constrains the gradient of the exponential inverse mapping and thus limits the degree to which their approach transgresses the theory.

Original: Moreover, the performance of their method degrades signiﬁcantly with γ = 0.99, consistent with convergence issues (see Appendix C.2 in their work).
Revised: Moreover, the performance of their method degrades signiﬁcantly with γ = 0.99, consistent with convergence issues (see Appendix C.2 in their work).

Original: Unfortunately, it is simple to provide examples where γ = 0.96 will lead to poor performance.
Revised: Unfortunately, it is simple to provide examples where γ = 0.96 will lead to poor performance.

Original: For example, in the game Bowling, there is a delay of about 100 frames between the ball being released and the subsequent reward.
Revised: For example, in the game Bowling, there is a delay of about 100 frames between the ball being released and the subsequent reward.

Original: However, if the player bowls a spare, the reward is delayed by a further 100 frames.
Revised: However, if the player bowls a spare, the reward is delayed by a further 100 frames.

Original: Thus, under a discount of 0.96, it is better to consistently bowl 9s than to bowl spares.
Revised: Thus, under a discount of 0.96, it is better to consistently bowl 9s than to bowl spares.

Original: Our work ﬁts into a broader line of research whereby reinforcement learning agents seek to predict quantities other than just the mean expected return (Jaderberg et al., 2016; Bellemare et al., 2017;
Revised: Our work ﬁts into a broader line of research whereby reinforcement learning agents seek to predict quantities other than just the mean expected return (Jaderberg et al., 2016; Bellemare et al., 2017;

Original: Fedus et al., 2019).
Revised: Fedus et al., 2019).

Original: Of this related work, one of the most similar approaches to ours is that of Romoff et al.
Revised: Of this related work, one of the most similar approaches to ours is that of Romoff et al.

Original: (2019), who learn the value function in a modular way by decomposing based on the discount factor.
Revised: (2019), who learn the value function in a modular way by decomposing based on the discount factor.

Original: Another closely related line of work is distributional reinforcement learning (Bellemare et al., 2017; Dabney et al., 2018b;a).
Revised: Another closely related line of work is distributional reinforcement learning (Bellemare et al., 2017; Dabney et al., 2018b;a).

Original: Finally, another, older line of work in this vein is decomposed RL (Russell & Zimdars, 2003; Sprague & Ballard, 2003; Van Seijen et al., 2017), whereby an agent is responsible for predicting only a portion of the action-value function.
Revised: Finally, another, older line of work in this vein is decomposed RL (Russell & Zimdars, 2003; Sprague & Ballard, 2003; Van Seijen et al., 2017), whereby an agent is responsible for predicting only a portion of the action-value function.

Original: While some of these works bear many surface-level similarities to our own — the distributional C51 agent (Bellemare et al., 2017) in particular bears quite a similar network architecture to ours — on a deeper level, they are not actually addressing the same problem.
Revised: While some of these works bear many surface-level similarities to our own — the distributional C51 agent (Bellemare et al., 2017) in particular bears quite a similar network architecture to ours — on a deeper level, they are not actually addressing the same problem.

Original: Leveraging a probabilistic loss, as in C51, could foreseeably help in addressing the large training errors that arise in tasks like Exponential Pong .
Revised: Leveraging a probabilistic loss, as in C51, could foreseeably help in addressing the large training errors that arise in tasks like Exponential Pong .

Original: However, the method assumes that returns are constrained to [-10, 10], and it is unclear how this could be extended to the unclipped reward setting without requiring an intractable number of atoms in the distribution, or greatly reducing the precision of the atoms.
Revised: However, the method assumes that returns are constrained to [-10, 10], and it is unclear how this could be extended to the unclipped reward setting without requiring an intractable number of atoms in the distribution, or greatly reducing the precision of the atoms.

Original: A E XPERIMENTAL S ETUP
Revised: A E XPERIMENTAL S ETUP

Original: We injected stochasticity into the Atari environment via the no-ops regime used in the Nature DQN paper (Mnih et al., 2015).
Revised: We injected stochasticity into the Atari environment via the no-ops regime used in the Nature DQN paper (Mnih et al., 2015).

Original: Rather than conducting periodic evaluations of each agent with a small exploration constant, we simply report training performance in all graphs, since this arguably provides a less variable measure of progress.
Revised: Rather than conducting periodic evaluations of each agent with a small exploration constant, we simply report training performance in all graphs, since this arguably provides a less variable measure of progress.

Original: All curves in the paper were calculated by averaging over 5 seeds.
Revised: All curves in the paper were calculated by averaging over 5 seeds.

Original: Shaded regions represent bootstrapped single standard deviation conﬁdence intervals.
Revised: Shaded regions represent bootstrapped single standard deviation conﬁdence intervals.

Original: Hyperparameters speciﬁc to Spectral DQN are listed in Table 2, while the hyperparameters common to all agents are listed in Table 3.
Revised: Hyperparameters speciﬁc to Spectral DQN are listed in Table 2, while the hyperparameters common to all agents are listed in Table 3.

Original: Other than multiplying the number of outputs by the frequency count, N , Spectral DQN used an identical network architecture to Nature DQN Mnih et al. (2015).
Revised: Other than multiplying the number of outputs by the frequency count, N , Spectral DQN used an identical network architecture to Nature DQN Mnih et al. (2015).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B A DDITIONAL E XPERIMENTS
Revised: B A DDITIONAL E XPERIMENTS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide the results of some additional experiments that did not ﬁt within the main text.
Revised: we provide the results of some additional experiments that did not ﬁt within the main text.

Original: B. A BLATION : N O E RROR W EIGHTING
Revised: B. A BLATION : N O E RROR W EIGHTING

Original: Recall from Section 5 that instead than weighting the frequencies’ temporal difference errors equally in the loss, we set w i = 

Revised: Recall from Section 5 that instead than weighting the frequencies’ temporal difference errors equally in the loss, we set w i =

Original: /σ 

Revised: /σ

Original: i , where σ i is the standard deviation of the training targets for frequency i .
Revised: i , where σ i is the standard deviation of the training targets for frequency i .

Original: In Figure
Revised: In Figure

Original: [Table]
Revised: [Table]

Original: we show the effect ablating this correction by simply setting w i = 1 .
Revised: we show the effect ablating this correction by simply setting w i = 1 .

Original: The results reveal that gradient correction is absolutely crucial, with the ablation proving disastrous in 4 of the 6 games.
Revised: The results reveal that gradient correction is absolutely crucial, with the ablation proving disastrous in 4 of the 6 games.

Original: Moreover, it unsurprising that the Fishing results are unaffected, since only one reward frequency is active in that game.
Revised: Moreover, it unsurprising that the Fishing results are unaffected, since only one reward frequency is active in that game.

Original: B. E ASIER P ONGLANTIS
Revised: B. E ASIER P ONGLANTIS

Original: In Section 3, we applied the agents Pop-Art and DQN+TC to the game Ponglantis .
Revised: In Section 3, we applied the agents Pop-Art and DQN+TC to the game Ponglantis .

Original: The results suggested that DQN+TC is somewhat more robust to reward progressivity.
Revised: The results suggested that DQN+TC is somewhat more robust to reward progressivity.

Original: Here, we provide some further experiments to determine just how strong the reward progression has to be for these methods to break down.
Revised: Here, we provide some further experiments to determine just how strong the reward progression has to be for these methods to break down.

Original: To ﬁnd the agents’ breaking points, we created two modiﬁed versions of the game: Ponglantis Easier and Ponglantis Even Easier .
Revised: To ﬁnd the agents’ breaking points, we created two modiﬁed versions of the game: Ponglantis Easier and Ponglantis Even Easier .

Original: In Ponglantis Easier , the agent’s perceived rewards in the Atlantis phase of the task are scaled down by a factor 10.
Revised: In Ponglantis Easier , the agent’s perceived rewards in the Atlantis phase of the task are scaled down by a factor 10.

Original: In Ponglantis Even Easier , they are scaled down by a factor of 100.
Revised: In Ponglantis Even Easier , they are scaled down by a factor of 100.

Original: The agents’ results in these games are shown in Figure 7.
Revised: The agents’ results in these games are shown in Figure 7.

Original: For comparability, the plots
Revised: For comparability, the plots

Original: [Figure]
Revised: [Figure]

Original: show the true episodic return, rather than the perceived episodic return.
Revised: show the true episodic return, rather than the perceived episodic return.

Original: We also include results for
Revised: We also include results for

Original: Spectral DQN as a reference point.
Revised: Spectral DQN as a reference point.

Original: Relative to its performance in the original Ponglantis task, DQN+TC performs much better in both of these tasks, reaching scores of around 300,000–400,000 points by the end of training.
Revised: Relative to its performance in the original Ponglantis task, DQN+TC performs much better in both of these tasks, reaching scores of around 300,000–400,000 points by the end of training.

Original: This suggests that it can tolerate a jump in reward magnitude of around 100x.
Revised: This suggests that it can tolerate a jump in reward magnitude of around 100x.

Original: Further, this implies that its underperformance relative to Spectral DQN in the main Atari experiments is likely attributable to more than just reward progressivity, pointing to our alternate hypotheses about representation learning and the relative ease with which Spectral DQN can represent large expected returns.
Revised: Further, this implies that its underperformance relative to Spectral DQN in the main Atari experiments is likely attributable to more than just reward progressivity, pointing to our alternate hypotheses about representation learning and the relative ease with which Spectral DQN can represent large expected returns.

Original: Pop-Art, on the other hand, still performs poorly when the perceived rewards in Atlantis are scaled down by a factor of 10, only learning reliably when they are scaled down 100 times.
Revised: Pop-Art, on the other hand, still performs poorly when the perceived rewards in Atlantis are scaled down by a factor of 10, only learning reliably when they are scaled down 100 times.

Original: In this instance, it is quite plausible that the algorithm struggles with reward progressivity in standard Atari games.
Revised: In this instance, it is quite plausible that the algorithm struggles with reward progressivity in standard Atari games.

Original: B. R EVERSE P ONGLANTIS
Revised: B. R EVERSE P ONGLANTIS

Original: To test whether reward progressivity is the key issue, as opposed to just reward variability , we created a further domain – Reverse Ponglantis – where the Pong and Atlantis phases are reversed.
Revised: To test whether reward progressivity is the key issue, as opposed to just reward variability , we created a further domain – Reverse Ponglantis – where the Pong and Atlantis phases are reversed.

Original: The player progresses to the Pong phase upon losing a life in Atlantis .
Revised: The player progresses to the Pong phase upon losing a life in Atlantis .

Original: The agents’ results in this game are shown in Figure 8.
Revised: The agents’ results in this game are shown in Figure 8.

Original: While all agents improve steadily in this domain, which largely conﬁrms our hypothesis, it is interesting that Spectral DQN still outperforms the other methods.
Revised: While all agents improve steadily in this domain, which largely conﬁrms our hypothesis, it is interesting that Spectral DQN still outperforms the other methods.

Original: In the previous experiment, Spectral
Revised: In the previous experiment, Spectral

Original: DQN did not outperform DQN+TC in Ponglantis Even Easier (see Figure 7, right).
Revised: DQN did not outperform DQN+TC in Ponglantis Even Easier (see Figure 7, right).

Original: Combined, these results suggest that Spectral DQN is simply better at handling the large rewards in Atlantis .
Revised: Combined, these results suggest that Spectral DQN is simply better at handling the large rewards in Atlantis .

Original: [Figure]
Revised: [Figure]

Original: In fact, this is unsurprising; since the training targets are more constrained under Spectral DQN, it ought to be easier to ﬁnd a step size that performs well across games with greatly different return magnitudes.
Revised: In fact, this is unsurprising; since the training targets are more constrained under Spectral DQN, it ought to be easier to ﬁnd a step size that performs well across games with greatly different return magnitudes.

Original: [Figure]
Revised: [Figure]

Original: C S PECTRAL Q-L EARNING A LGORITHM
Revised: C S PECTRAL Q-L EARNING A LGORITHM

Original: Pseudocode for the Spectral Q-Learning algorithm is provided in Algorithm 1.
Revised: Pseudocode for the Spectral Q-Learning algorithm is provided in Algorithm 1.

Original: 1: var: the base of the spectral decomposition
Revised: 1: var: the base of the spectral decomposition

Original: 2: var: the spectral
Revised: 2: var: the spectral

Original: 4: 5: Initialise
Revised: 4: 5: Initialise

Original: 6: for do 7:
Revised: 6: for do 7:

Original: Sample do 9:
Revised: Sample do 9:

Original: Choose a according to (cid:15) -greedy policy derived
Revised: Choose a according to (cid:15) -greedy policy derived

Original: Q s a
Revised: Q s a

Original: Ni i
Revised: Ni i

Original: Take action a , observe r , s
Revised: Take action a , observe r , s

Original: 11: a (cid:48)∗ = arg max a (cid:48) (cid:80) Ni =

Revised: 11: a (cid:48)∗ = arg max a (cid:48) (cid:80) Ni =

Original: b i Q ( s (cid:48) , a (cid:48) , i ) 12: for i ∈ { 0 , . . .
Revised: b i Q ( s (cid:48) , a (cid:48) , i ) 12: for i ∈ { 0 , . . .

Original: , N } do 13:
Revised: , N } do 13:

Original: Q ( s, a, i ) ← Q ( s, a, i ) + α [ r ib + γQ ( s (cid:48) , a (cid:48)∗ , i ) − Q ( s, a, i )] 14: s ← s (cid:48)
Revised: Q ( s, a, i ) ← Q ( s, a, i ) + α [ r ib + γQ ( s (cid:48) , a (cid:48)∗ , i ) − Q ( s, a, i )] 14: s ← s (cid:48)

Original: D D ERIVATION OF THE S PECTRAL D ECOMPOSITION F ORMULA
Revised: D D ERIVATION OF THE S PECTRAL D ECOMPOSITION F ORMULA

Original: Recall from Section 4.1 that the spectral representation of a number is found by laying buckets of exponentially increasing width on the number line, then calculating the proportion of each bucket that is ﬁlled.
Revised: Recall from Section 4.1 that the spectral representation of a number is found by laying buckets of exponentially increasing width on the number line, then calculating the proportion of each bucket that is ﬁlled.

Original: To calculate the proportion of the i th bucket that is ﬁlled by a non-negative number r , note that the sum of all the preceding buckets’ widths is equal to:
Revised: To calculate the proportion of the i th bucket that is ﬁlled by a non-negative number r , note that the sum of all the preceding buckets’ widths is equal to:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the amount leftover to ﬁll the i th bucket is:
Revised: Therefore, the amount leftover to ﬁll the i th bucket is:

Original: [Equation]
Revised: [Equation]

Original: To calculate the proportion of the i th bucket that is ﬁlled, we must divide this quantity by the width of the i th bucket (which is equal to b i ) then clamp the result to the allowable range of values:
Revised: To calculate the proportion of the i th bucket that is ﬁlled, we must divide this quantity by the width of the i th bucket (which is equal to b i ) then clamp the result to the allowable range of values:

Original: [Equation]
Revised: [Equation]

Original: Finally, to convert this into an odd function so that the representation of − r is equal to the negated representation of r , we replace r with | r | and multiply the result by sign ( r ) .
Revised: Finally, to convert this into an odd function so that the representation of − r is equal to the negated representation of r , we replace r with | r | and multiply the result by sign ( r ) .

Original: This yields the formula for the i th element of the spectral representation:
Revised: This yields the formula for the i th element of the spectral representation:

Original: [Equation]
Revised: [Equation]

Original: E P ROOFS E. P ROOF OF E QUATION
Revised: E P ROOFS E. P ROOF OF E QUATION

Original: Starting with the standard deﬁnition of the action-value function, then replacing the reward at each time step with its spectral decomposition, we have:
Revised: Starting with the standard deﬁnition of the action-value function, then replacing the reward at each time step with its spectral decomposition, we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since there is no interaction between the summation indices, we can rearrange (15) into:
Revised: Since there is no interaction between the summation indices, we can rearrange (15) into:

Original: [Equation]
Revised: [Equation]

Original: Finally, using the linearity of expectation, we can take the expectation operator inside and write:
Revised: Finally, using the linearity of expectation, we can take the expectation operator inside and write:

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF P ROPOSITION
Revised: E. P ROOF OF P ROPOSITION

Original: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .
Revised: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .

Original: We prove this statement via induction.
Revised: We prove this statement via induction.

Original: The ﬁrst point to observe is that, provided the action-value estimates of the two algorithms agree at a given point in time, their (cid:15) -greedy policies at that time will also be equivalent.
Revised: The ﬁrst point to observe is that, provided the action-value estimates of the two algorithms agree at a given point in time, their (cid:15) -greedy policies at that time will also be equivalent.

Original: Therefore, assuming that the action-value functions agree at t = 

Revised: Therefore, assuming that the action-value functions agree at t =

Original: (e.g. they are both initialised to zero), all we need to prove is the inductive step, i.e. that agreement in action-values at time t implies agreement in action-values at time t + 1 .
Revised: (e.g. they are both initialised to zero), all we need to prove is the inductive step, i.e. that agreement in action-values at time t implies agreement in action-values at time t + 1 .

Original: To distinguish between the action-value estimates of the two algorithms at different time steps, we use the following notation:
Revised: To distinguish between the action-value estimates of the two algorithms at different time steps, we use the following notation:

Original: • Let ˆ Q ( s, a ; t ) denote the action-value estimates under standard Q-learning after t updates.
Revised: • Let ˆ Q ( s, a ; t ) denote the action-value estimates under standard Q-learning after t updates.

Original: • Let Q ( s, a ; t ) =
Revised: • Let Q ( s, a ; t ) =

Original: (cid:80) Ni =
Revised: (cid:80) Ni =

Original: b i Q ( s, a, i ; t ) denote the action-values estimates under Spectral Q-learning after t updates.
Revised: b i Q ( s, a, i ; t ) denote the action-values estimates under Spectral Q-learning after t updates.

Original: For the induction step, we need to show that Q ( s, a ; t + 1)
Revised: For the induction step, we need to show that Q ( s, a ; t + 1)

Original: =
Revised: =

Original: ˆ Q ( s, a ; t + 1) ∀ s ∈ S , a ∈ A given that Q ( s, a ; t ) =
Revised: ˆ Q ( s, a ; t + 1) ∀ s ∈ S , a ∈ A given that Q ( s, a ; t ) =

Original: ˆ Q ( s, a ; t ) ∀ s ∈ S , a ∈ A .
Revised: ˆ Q ( s, a ; t ) ∀ s ∈ S , a ∈ A .

Original: To begin, note that the standard Q-learning backup expressed under the above notation is:
Revised: To begin, note that the standard Q-learning backup expressed under the above notation is:

Original: [Equation]
Revised: [Equation]

Original: (19) where a (cid:48)∗ = arg max a (cid:48)
Revised: (19) where a (cid:48)∗ = arg max a (cid:48)

Original: ˆ Q ( s (cid:48) , a (cid:48)∗ ; t ) is the greedy action for the next state.
Revised: ˆ Q ( s (cid:48) , a (cid:48)∗ ; t ) is the greedy action for the next state.

Original: Since we assume equal action-values at time t , Spectral Q-learning will calculate the same greedy action for s (cid:48) .
Revised: Since we assume equal action-values at time t , Spectral Q-learning will calculate the same greedy action for s (cid:48) .

Original: Hence, the backup for each frequency of the spectral action-value function is:
Revised: Hence, the backup for each frequency of the spectral action-value function is:

Original: [Equation]
Revised: [Equation]

Original: After performing this update on each frequency
Revised: After performing this update on each frequency

Original: [Equation]
Revised: [Equation]

Original: the function at time t + 1 becomes:
Revised: the function at time t + 1 becomes:

Original: [Equation]
Revised: [Equation]

Original: Since the induction base case assumes that ˆ Q ( s
Revised: Since the induction base case assumes that ˆ Q ( s

Original: [Equation]
Revised: [Equation]

Original: a ; t ) =
Revised: a ; t ) =

Original: Q ( s
Revised: Q ( s

Original: [Equation]
Revised: [Equation]

Original: a ; t ) ∀ s ∈ S
Revised: a ; t ) ∀ s ∈ S

Original: [Equation]
Revised: [Equation]

Original: a ∈ A
Revised: a ∈ A

Original: [Equation]
Revised: [Equation]

Original: we can rewrite this as:
Revised: we can rewrite this as:

Original: [Equation]
Revised: [Equation]

Original: Now, all that remains to establish equality with the standard Q-learning update in (19) is to show that (cid:80) Ni =

Revised: Now, all that remains to establish equality with the standard Q-learning update in (19) is to show that (cid:80) Ni =

Original: b i r ib = r .
Revised: b i r ib = r .

Original: That is, we need to show that the partial sum over indices 0 to N fully captures the reward, or, equivalently, that r ib = 0 for all
Revised: That is, we need to show that the partial sum over indices 0 to N fully captures the reward, or, equivalently, that r ib = 0 for all

Original: i > = N + 1 .
Revised: i > = N + 1 .

Original: From the spectral decomposition formula (13) and the proposition’s statement that the rewards are bound to no more than ( b N +1 − 1) / ( b − 1) , we have:
Revised: From the spectral decomposition formula (13) and the proposition’s statement that the rewards are bound to no more than ( b N +1 − 1) / ( b − 1) , we have:

Original: [Equation]
Revised: [Equation]

Original: Now, if i > = N + 1 , it implies that ( b N +1 − i − 1) ≤ 0 , so the whole expression in (30) becomes 0.
Revised: Now, if i > = N + 1 , it implies that ( b N +1 − i − 1) ≤ 0 , so the whole expression in (30) becomes 0.

Original: Thus (cid:80) Ni =

Revised: Thus (cid:80) Ni =

Original: b i r ib
Revised: b i r ib

Original: =
Revised: =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: i =

Revised: i =

Original: b i r ib =
Revised: b i r ib =

Original: r and the proof is complete.
Revised: r and the proof is complete.

Original: (cid:4)
Revised: (cid:4)