Original: A DAPTING TO R EWARD P ROGRESSIVITY VIA S PECTRAL R EINFORCEMENT L EARNING
Revised: A DAPTING TO R EWARD P ROGRESSIVITY VIA S PECTRAL R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper we consider reinforcement learning tasks with progressive rewards ; that is, tasks where the rewards tend to increase in magnitude over time.
Revised: In this paper we consider reinforcement learning tasks with progressive rewards ; that is, tasks where the rewards tend to increase in magnitude over time.

Original: We hypothesise that this property may be problematic for value-based deep reinforcement learning agents, particularly if the agent must ﬁrst succeed in relatively unrewarding regions of the task in order to reach more rewarding regions.
Revised: We hypothesise that this property may be problematic for value-based deep reinforcement learning agents, particularly if the agent must ﬁrst succeed in relatively unrewarding regions of the task in order to reach more rewarding regions.

Original: To address this issue, we propose Spectral DQN , which decomposes the reward into frequencies such that the high frequencies only activate when large rewards are found.
Revised: To address this issue, we propose Spectral DQN , which decomposes the reward into frequencies such that the high frequencies only activate when large rewards are found.

Original: This allows the training loss to be balanced so that it gives more even weighting across small and large reward regions.
Revised: This allows the training loss to be balanced so that it gives more even weighting across small and large reward regions.

Original: In two domains with extreme reward progressivity, where standard value-based methods struggle signiﬁcantly, Spectral
Revised: In two domains with extreme reward progressivity, where standard value-based methods struggle signiﬁcantly, Spectral

Original: DQN is able to make much farther progress.
Revised: DQN is able to make much farther progress.

Original: Moreover, when evaluated on a set of six standard Atari games that do not overtly favour the approach, Spectral DQN remains more than competitive: While it underperforms one of the benchmarks in a single game, it comfortably surpasses the benchmarks in three games.
Revised: Moreover, when evaluated on a set of six standard Atari games that do not overtly favour the approach, Spectral DQN remains more than competitive: While it underperforms one of the benchmarks in a single game, it comfortably surpasses the benchmarks in three games.

Original: These results demonstrate that the approach is not overﬁt to its target problem, and suggest that Spectral DQN may have advantages beyond addressing reward progressivity.
Revised: These results demonstrate that the approach is not overﬁt to its target problem, and suggest that Spectral DQN may have advantages beyond addressing reward progressivity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In decision making tasks involving compounding returns, such as stock market investing, it is common for the rewards received by the agent to increase in magnitude over time.
Revised: In decision making tasks involving compounding returns, such as stock market investing, it is common for the rewards received by the agent to increase in magnitude over time.

Original: An investor that starts out with little capital will achieve relatively small proﬁts or losses early on, but if they are successful, their capital – and hence their potential proﬁts and losses – will gradually increase.
Revised: An investor that starts out with little capital will achieve relatively small proﬁts or losses early on but if they are successful their capital – and hence their potential proﬁts and losses – will gradually increase.

Original: This property also arises in many games settings.
Revised: This property also arises in many games settings.

Original: For example, in the Atari game Video Pinball , the player can increase a “bonus multiplier” that increases the score paid per bumper hit.
Revised: For example, in the Atari game Video Pinball , the player can increase a “bonus multiplier” that increases the score paid per bumper hit.

Original: Since the bonus multiplier does not reset until the player dies, the rewards typically increase with time.
Revised: Since the bonus multiplier does not reset until the player dies, the rewards typically increase with time.

Original: In this paper, we refer to any task exhibiting this phenomenon as a progressive reward task .
Revised: In this paper, we refer to any task exhibiting this phenomenon as a progressive reward task .

Original: We hypothesise that reward progressivity may be problematic for value-based deep reinforcement learning agents.
Revised: We hypothesise that reward progressivity may be problematic for value-based deep reinforcement learning agents.

Original: Our rationale is that the temporal difference errors that arise under such algorithms typically scale with the magnitude of the training targets.
Revised: Our rationale is that the temporal difference errors that arise under such algorithms typically scale with the magnitude of the training targets.

Original: Accordingly, experience collected from states with large expected returns may dominate training, causing the agent’s performance to degrade in other states.
Revised: Accordingly, experience collected from states with large expected returns may dominate training, causing the agent’s performance to degrade in other states.

Original: While this is rational in one sense – generally, it is more important to make accurate decisions in high-stakes situations – it is potentially harmful on progressive reward tasks.
Revised: While this is rational in one sense – generally, it is more important to make accurate decisions in high-stakes situations – it is potentially harmful on progressive reward tasks.

Original: This is because, on progressive reward tasks, it may be necessary for the agent to perform well in relatively unrewarding regions before it can reach more rewarding regions .
Revised: This is because, on progressive reward tasks, it may be necessary for the agent to perform well in relatively unrewarding regions before it can reach more rewarding regions .

Original: For example, in stock market investing, an investor can only reach states where they have large capital if they ﬁrst perform well with small capital.
Revised: For example, in stock market investing, an investor can only reach states where they have large capital if they ﬁrst perform well with small capital.

Original: Note too that the rewards need not be strictly progressive for this problem to arise; all that is needed is for the rewards to be progressive over some extended period.
Revised: Note too that the rewards need not be strictly progressive for this problem to arise; all that is needed is for the rewards to be progressive over some extended period.

Original: Algorithms that clip the reward, such as DQN (Mnih et al., 2015), are less susceptible to this problem, because they do not perceive increases in reward magnitude beyond the clipping point.
Revised: Algorithms that clip the reward, such as DQN (Mnih et al., 2015), are less susceptible to this problem, because they do not perceive increases in reward magnitude beyond the clipping point.

Original: However, it is straightforward to construct examples where reward clipping masks the optimal solution.
Revised: However, it is straightforward to construct examples where reward clipping masks the optimal solution.

Original: For example, in Bowling , clipping the rewards to [-1, 1] makes bowling a strike appear no better than hitting a single pin (Pohlen et al., 2018).
Revised: For example, in Bowling , clipping the rewards to [-1, 1] makes bowling a strike appear no better than hitting a single pin (Pohlen et al., 2018).

Original: While subsequent approaches make learning feasible without reward clipping (van Hasselt et al., 2016; Pohlen et al., 2018), we show in Section 3 that these methods are negatively impacted by strong reward progressivity.
Revised: While subsequent approaches make learning feasible without reward clipping (van Hasselt et al., 2016; Pohlen et al., 2018), we show in Section 3 that these methods are negatively impacted by strong reward progressivity.

Original: Motivated by this, we seek an algorithm that is capable of learning from unclipped rewards, while mitigating the negative impact that large rewards have on the agent’s ability to learn in less rewarding regions of the task.
Revised: Motivated by this, we seek an algorithm that is capable of learning from unclipped rewards, while mitigating the negative impact that large rewards have on the agent’s ability to learn in less rewarding regions of the task.

Original: To meet this aim, we propose Spectral DQN .
Revised: To meet this aim, we propose Spectral DQN .

Original: Under this approach, rewards are decomposed into a multidimensional spectral reward , where each component is bound to [-1, 1].
Revised: Under this approach, rewards are decomposed into a multidimensional spectral reward , where each component is bound to [-1, 1].

Original: The upper frequencies of the spectrum activate only when large magnitude rewards are received.
Revised: The upper frequencies of the spectrum activate only when large magnitude rewards are received.

Original: For example, in Video
Revised: For example, in Video

Original: Pinball , the lowest frequency activates on any score, while the upper frequencies only activate when the player has accumulated a large bonus multiplier.
Revised: Pinball , the lowest frequency activates on any score, while the upper frequencies only activate when the player has accumulated a large bonus multiplier.

Original: The bounded nature of the spectral rewards allows the agent to learn expected spectral returns without instability arising from large training targets.
Revised: The bounded nature of the spectral rewards allows the agent to learn expected spectral returns without instability arising from large training targets.

Original: Meanwhile, the full, unclipped expected return can be recovered by summing across the return spectrum.
Revised: Meanwhile, the full, unclipped expected return can be recovered by summing across the return spectrum.

Original: (The full, technical deﬁnitions of these terms are provided in Section 4).
Revised: (The full, technical deﬁnitions of these terms are provided in Section 4).

Original: In terms of addressing reward progressivity, a key advantage of this approach is that it allows us to balance the training loss and prevent the upper frequencies from receiving undue weight.
Revised: In terms of addressing reward progressivity, a key advantage of this approach is that it allows us to balance the training loss and prevent the upper frequencies from receiving undue weight.

Original: To test whether this approach helps mitigate the impact of reward progressivity, we perform two set of experiments: First, we apply Spectral DQN to two domains that we speciﬁcally designed to exhibit strong reward progressivity.
Revised: To test whether this approach helps mitigate the impact of reward progressivity, we perform two set of experiments: First, we apply Spectral DQN to two domains that we speciﬁcally designed to exhibit strong reward progressivity.

Original: While previous methods struggle on these tasks, failing to learn almost anything at all in the more extreme domain, Spectral DQN performs markedly better, making considerable progress on both tasks.
Revised: While previous methods struggle on these tasks, failing to learn almost anything at all in the more extreme domain, Spectral DQN performs markedly better, making considerable progress on both tasks.

Original: Next, we apply our approach to a less constructed set of tasks; namely, 6 standard Atari games.
Revised: Next, we apply our approach to a less constructed set of tasks; namely, 6 standard Atari games.

Original: Only some of these games exhibit reward progressivity, and none of them nearly so strongly as the extreme domains from earlier.
Revised: Only some of these games exhibit reward progressivity, and none of them nearly so strongly as the extreme domains from earlier.

Original: While Spectral DQN does not outperform the benchmarks as noticeably in these domains (it is beaten by one of the benchmarks in a single game, but comfortably outperforms both benchmarks in three games), the results clearly show that Spectral DQN is not overﬁt to constructed domains.
Revised: While Spectral DQN does not outperform the benchmarks as noticeably in these domains (it is beaten by one of the benchmarks in a single game, but comfortably outperforms both benchmarks in three games), the results clearly show that Spectral DQN is not overﬁt to constructed domains.

Original: Moreover, since its outperformance in some of the games cannot be attributed solely to reward progressivity, it appears that the approach may offer additional advantages, as explained later in our analysis of the results.
Revised: Moreover, since its outperformance in some of the games cannot be attributed solely to reward progressivity, it appears that the approach may offer additional advantages, as explained later in our analysis of the results.

Original:  P REVIOUS A PPROACHES TO H ANDLING R EWARD V ARIABILITY
Revised:  P REVIOUS A PPROACHES TO H ANDLING R EWARD V ARIABILITY

Original: In the Atari domain, where the DQN algorithm was ﬁrst evaluated, the agent receives a reward equal to the score increase at each frame.
Revised: In the Atari domain, where the DQN algorithm was ﬁrst evaluated, the agent receives a reward equal to the score increase at each frame.

Original: However, since score magnitudes vary greatly across games,
Revised: However, since score magnitudes vary greatly across games,

Original: DQN clips all rewards to [ − 1 , +1] .
Revised: DQN clips all rewards to [ − 1 , +1] .

Original: This constrains the size of the training targets so that it is easier to ﬁnd a step size that performs well across the entire suite of games.
Revised: This constrains the size of the training targets so that it is easier to ﬁnd a step size that performs well across the entire suite of games.

Original: While this heuristic turns out to perform well in Atari , a clear downside of the approach is that the agent becomes unable to distinguish large rewards from small rewards.
Revised: While this heuristic turns out to perform well in Atari , a clear downside of the approach is that the agent becomes unable to distinguish large rewards from small rewards.

Original: The Pop-Art algorithm (van Hasselt et al., 2016) offers a more principled solution.
Revised: The Pop-Art algorithm (van Hasselt et al., 2016) offers a more principled solution.

Original: It trains from the unclipped reward, while normalising the training targets to have zero mean and unit variance.
Revised: It trains from the unclipped reward, while normalising the training targets to have zero mean and unit variance.

Original: To ensure that the network’s predictions are preserved when the normalisation parameters are updated, the output of the ﬁnal layer is scaled and shifted by an offsetting amount.
Revised: To ensure that the network’s predictions are preserved when the normalisation parameters are updated, the output of the ﬁnal layer is scaled and shifted by an offsetting amount.

Original: Pohlen et al.
Revised: Pohlen et al.

Original: (2018) propose an alternative approach to handling unclipped rewards that reduces the variance of the training targets by applying a squashing function, h .
Revised: (2018) propose an alternative approach to handling unclipped rewards that reduces the variance of the training targets by applying a squashing function, h .

Original: The agent learns squashed action-values, ˜ Q ( s, a )
Revised: The agent learns squashed action-values, ˜ Q ( s, a )

Original: =
Revised: =

Original: h ( Q ( s, a )) , via the transformed Bellman backup:
Revised: h ( Q ( s, a )) , via the transformed Bellman backup:

Original: [Equation]
Revised: [Equation]

Original: The authors prove that the transformed Bellman operator remains a contraction provided that both h and h − 1 are Lipschitz continuous and γ < 1 /L
Revised: The authors prove that the transformed Bellman operator remains a contraction provided that both h and h − 1 are Lipschitz continuous and γ < 1 /L

Original: h L h − 1 (where L h and L h − 1 are the respective
Revised: h L h − 1 (where L h and L h − 1 are the respective

Original: Lipschitz constants).
Revised: Lipschitz constants).

Original: In their experiments, they use the following squashing function:
Revised: In their experiments, they use the following squashing function:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 ensures that h − 1 is Lipschitz continuous.
Revised: where (cid:15) > 0 ensures that h − 1 is Lipschitz continuous.

Original: For the remainder of this paper, we refer to this approach as target compression .
Revised: For the remainder of this paper, we refer to this approach as target compression .

Original: In the Atari domain,
Revised: In the Atari domain,

Original: Pohlen et al. found target compression to perform signiﬁcantly better than Pop-Art.
Revised: Pohlen et al. found target compression to perform signiﬁcantly better than Pop-Art.

Original: It has since become the dominant method for handling unclipped rewards in Atari , with several recent largescale agents (Kapturowski et al., 2019; Badia et al., 2020a;b), including the state-of-the-art Agent57 , favouring target compression over Pop-Art.
Revised: It has since become the dominant method for handling unclipped rewards in Atari , with several recent largescale agents (Kapturowski et al., 2019; Badia et al., 2020a;b), including the state-of-the-art Agent57 , favouring target compression over Pop-Art.

Original:  D EMONSTRATING THE P ROBLEM OF R EWARD P ROGRESSIVITY
Revised:  D EMONSTRATING THE P ROBLEM OF R EWARD P ROGRESSIVITY

Original: While both Pop-Art and target compression address reward variance across tasks, making it easier to devise a single learning conﬁguration that performs well across a task suite, they were not designed to address intra-task reward variability.
Revised: While both Pop-Art and target compression address reward variance across tasks, making it easier to devise a single learning conﬁguration that performs well across a task suite, they were not designed to address intra-task reward variability.

Original: As such, we posit that they are ill-suited to addressing reward progressivity, since it is an intra-task property.
Revised: As such, we posit that they are ill-suited to addressing reward progressivity, since it is an intra-task property.

Original: To test this claim, we introduce two domains that are designed to exhibit extreme reward progressivity, and thus exacerbate the issue: Ponglantis and Exponential Pong .
Revised: To test this claim, we introduce two domains that are designed to exhibit extreme reward progressivity, and thus exacerbate the issue: Ponglantis and Exponential Pong .

Original: Ponglantis.
Revised: Ponglantis.

Original: In Ponglantis , the game starts out exactly the same as the Atari game Pong , where valued-based deep reinforcement learning methods are known to perform well (Mnih et al., 2015).
Revised: In Ponglantis , the game starts out exactly the same as the Atari game Pong , where valued-based deep reinforcement learning methods are known to perform well (Mnih et al., 2015).

Original: The reward scale in Pong is small relative to most Atari games; the player receives a +1 reward for hitting the ball past the computer paddle, and a -1 reward if they concede a point.
Revised: The reward scale in Pong is small relative to most Atari games; the player receives a +1 reward for hitting the ball past the computer paddle, and a -1 reward if they concede a point.

Original: In Ponglantis , however, if the player manages to score 10 points, the task immediately switches to Atlantis , where the reward scale is roughly 1000 times larger.
Revised: In Ponglantis , however, if the player manages to score 10 points, the task immediately switches to Atlantis , where the reward scale is roughly 1000 times larger.

Original: The Atlantis phase ends upon the loss of a single life.
Revised: The Atlantis phase ends upon the loss of a single life.

Original: Exponential Pong.
Revised: Exponential Pong.

Original: The game Exponential Pong is identical to regular Pong , except that the rewards are multiplied by 2 n , where n is the player’s current score.
Revised: The game Exponential Pong is identical to regular Pong , except that the rewards are multiplied by 2 n , where n is the player’s current score.

Original: We applied two agents to these games: Pop-Art and DQN+TC, where the latter is a version of DQN that we modiﬁed to use target compression and learn from the unclipped reward.
Revised: We applied two agents to these games: Pop-Art and DQN+TC, where the latter is a version of DQN that we modiﬁed to use target compression and learn from the unclipped reward.

Original: The results of this experiment, averaged over 5 seeds, are shown in Figure 1.
Revised: The results of this experiment, averaged over 5 seeds, are shown in Figure 1.

Original: (For full experimental details, see the
Revised: (For full experimental details, see the

Original: Appendix.)
Revised: Appendix.)

Original: To simplify analysis of the Exponential Pong results, we have plotted the unexponentiated scores, i.e. the scores that the agents would have achieved under standard scoring in Pong .
Revised: To simplify analysis of the Exponential Pong results, we have plotted the unexponentiated scores, i.e. the scores that the agents would have achieved under standard scoring in Pong .

Original: For context, the best possible score on standard Pong is +21, and the worst possible score is -21.
Revised: For context, the best possible score on standard Pong is +21, and the worst possible score is -21.

Original: In Exponential Pong , which exhibits the more extreme reward progressivity (since the reward multiplier can potentially reach 2 20 ≈ 1 million), both agents essentially fail, providing strong support for our hypothesis that reward progressivity can be damaging, at least in extreme cases.
Revised: In Exponential Pong , which exhibits the more extreme reward progressivity (since the reward multiplier can potentially reach 2 20 ≈ 1 million), both agents essentially fail, providing strong support for our hypothesis that reward progressivity can be damaging, at least in extreme cases.

Original: In the more moderate case of Ponglantis , Pop-Art fails to make meaningful progress beyond the low-scoring Pong phase of the task.
Revised: In the more moderate case of Ponglantis , Pop-Art fails to make meaningful progress beyond the low-scoring Pong phase of the task.

Original: The likely reason for this is that as soon as the agent starts reaching Atlantis and receiving large rewards, the target scaling mechanism employed by Pop-Art starts dividing the training targets by a large factor, heavily dampening the much smaller learning signals arising from the Pong phase of the task.
Revised: The likely reason for this is that as soon as the agent starts reaching Atlantis and receiving large rewards, the target scaling mechanism employed by Pop-Art starts dividing the training targets by a large factor, heavily dampening the much smaller learning signals arising from the Pong phase of the task.

Original: On the other hand, DQN+TC is eventually able to make some progress into the Atlantis phase.
Revised: On the other hand, DQN+TC is eventually able to make some progress into the Atlantis phase.

Original: The fact that DQN+TC outperforms Pop-Art is consistent with our hypothesis; while Pop-Art sees the reward scale jump by a factor of ≈ 1000 upon reaching Atlantis , the training targets of DQN+TC jump by a smaller factor, owing to the square root squashing function (Equation 2).
Revised: The fact that DQN+TC outperforms Pop-Art is consistent with our hypothesis; while Pop-Art sees the reward scale jump by a factor of ≈ 1000 upon reaching Atlantis , the training targets of DQN+TC jump by a smaller factor, owing to the square root squashing function (Equation 2).

Original: In Appendix B.2, we perform some additional experiments in
Revised: In Appendix B.2, we perform some additional experiments in

Original: Ponglantis to ﬁnd where the “cutoff points” for Pop-Art and DQN+TC lie, i.e. how big the jump in reward magnitude has to be for the agents’ performances to start deteriorating.
Revised: Ponglantis to ﬁnd where the “cutoff points” for Pop-Art and DQN+TC lie, i.e. how big the jump in reward magnitude has to be for the agents’ performances to start deteriorating.

Original: Given the above discussion, a natural question to ponder is whether DQN+TC might achieve better performance in Exponential Pong by employing a logarithmic squashing function instead of a square root.
Revised: Given the above discussion, a natural question to ponder is whether DQN+TC might achieve better performance in Exponential Pong by employing a logarithmic squashing function instead of a square root.

Original: Unfortunately though, as Pohlen et al.
Revised: Unfortunately though, as Pohlen et al.

Original: (2018) point out to a reviewer of their work (see https://openreview.net/forum?id=BkfPnoActQ ), a logarithmic transform does not have a Lipschitz continuous inverse, and thus the transformed Bellman is no longer guaranteed to be a contraction.
Revised: (2018) point out to a reviewer of their work (see https://openreview.net/forum?id=BkfPnoActQ ), a logarithmic transform does not have a Lipschitz continuous inverse, and thus the transformed Bellman is no longer guaranteed to be a contraction.

Original: [Figure]
Revised: [Figure]

Original:  S PECTRAL R EINFORCEMENT L EARNING
Revised:  S PECTRAL R EINFORCEMENT L EARNING

Original: Based on the ﬁndings of the previous section, it seems that what we require is an algorithm where: (1) The training targets are constrained in magnitude, no matter how large the reward scale grows.
Revised: Based on the ﬁndings of the previous section, it seems that what we require is an algorithm where: (1) The training targets are constrained in magnitude, no matter how large the reward scale grows.

Original: (2) Regions of the task with small rewards yield meaningfully large temporal difference errors, even once much larger rewards have been discovered.
Revised: (2) Regions of the task with small rewards yield meaningfully large temporal difference errors, even once much larger rewards have been discovered.

Original: Note that Pop-Art fails to meet the second criterion, while target compression fails to meet both, albeit it takes larger shifts in reward magnitude for the problems associated with reward progressivity to become apparent.
Revised: Note that Pop-Art fails to meet the second criterion, while target compression fails to meet both, albeit it takes larger shifts in reward magnitude for the problems associated with reward progressivity to become apparent.

Original: In this section, we introduce a reinforcement learning approach that meets both of the above criteria by leveraging the idea of spectral decomposition.
Revised: In this section, we introduce a reinforcement learning approach that meets both of the above criteria by leveraging the idea of spectral decomposition.

Original: This algorithm does not in and of itself address reward progressivity; in fact, we show that it behaves identically to the standard Q-learning algorithm (Watkins, 1989), given certain caveats.
Revised: This algorithm does not in and of itself address reward progressivity; in fact, we show that it behaves identically to the standard Q-learning algorithm (Watkins, 1989), given certain caveats.

Original: However, the properties of this algorithm facilitate adaption to progressive rewards, and in Section 5 we present a deep learning version of the approach that is capable of learning in domains with extreme reward progressivity, such as Exponential Pong .
Revised: However, the properties of this algorithm facilitate adaption to progressive rewards, and in Section 5 we present a deep learning version of the approach that is capable of learning in domains with extreme reward progressivity, such as Exponential Pong .

Original: 4. S PECTRAL DECOMPOSITION
Revised: 4. S PECTRAL DECOMPOSITION

Original: Spectral decomposition is a broad term that arises in many different ﬁelds, but in this paper it refers to a speciﬁc method for decomposing real numbers into a sum of exponentially weighted components.
Revised: Spectral decomposition is a broad term that arises in many different ﬁelds, but in this paper it refers to a speciﬁc method for decomposing real numbers into a sum of exponentially weighted components.

Original: The intuition behind our method is illustrated in Figure 2.
Revised: The intuition behind our method is illustrated in Figure 2.

Original: In this example, the number being spectrally decomposed is 6.5.
Revised: In this example, the number being spectrally decomposed is 6.5.

Original: The method begins by dividing the number line into buckets of width b i , where b > 0 is the base of the decomposition and
Revised: The method begins by dividing the number line into buckets of width b i , where b > 0 is the base of the decomposition and

Original: i ∈
Revised: i ∈

Original: Z ≥ 0 .
Revised: Z ≥ 0 .

Original: Since the base here is 2, bucket 0 has width 1 and spans from 0 to 1, bucket 1 has width 2 and spans from 1 to 3, bucket 2 has width 4 and spans from 3 to 7, etc.
Revised: Since the base here is 2, bucket 0 has width 1 and spans from 0 to 1, bucket 1 has width 2 and spans from 1 to 3, bucket 2 has width 4 and spans from 3 to 7, etc.

Original: The idea is that each bucket corresponds to a different frequency in the spectrum.
Revised: The idea is that each bucket corresponds to a different frequency in the spectrum.

Original: The spectral decomposition is found by calculating the proportion of each bucket that is ﬁlled.
Revised: The spectral decomposition is found by calculating the proportion of each bucket that is ﬁlled.

Original: Since the number 6.5 completely ﬁlls the ﬁrst two buckets, then 87.5% of the next bucket, then none of the remaining buckets, the spectral decomposition of 6.5 to base 2 is (1, 1, 0.875, 0, 0, ...).
Revised: Since the number 6.5 completely ﬁlls the ﬁrst two buckets, then 87.5% of the next bucket, then none of the remaining buckets, the spectral decomposition of 6.5 to base 2 is (1, 1, 0.875, 0, 0, ...).

Original: More generally, let r be a non-negative real number, and let r i
Revised: More generally, let r be a non-negative real number, and let r i

Original: + b denote the i th element of r ’s spectral decomposition to base b .
Revised: + b denote the i th element of r ’s spectral decomposition to base b .

Original: The formula for calculating each element is:
Revised: The formula for calculating each element is:

Original: [Equation]
Revised: [Equation]

Original: (See the Appendix for the derivation of this formula.)
Revised: (See the Appendix for the derivation of this formula.)

Original: The value r can be recovered from the spectral decomposition by multiplying the bucket widths by the bucket ﬁll proportions and summing:
Revised: The value r can be recovered from the spectral decomposition by multiplying the bucket widths by the bucket ﬁll proportions and summing:

Original: [Equation]
Revised: [Equation]

Original: This method can be extended to negative numbers by negating the sign on each element of the representation.
Revised: This method can be extended to negative numbers by negating the sign on each element of the representation.

Original: Given any real number r , the formula for calculating each element becomes:
Revised: Given any real number r , the formula for calculating each element becomes:

Original: [Equation]
Revised: [Equation]

Original: 4. D ECOMPOSING THE R ETURN
Revised: 4. D ECOMPOSING THE R ETURN

Original: The core idea behind our reinforcement learning approach is to apply spectral decomposition to the reward stream, and thus learn a spectral decomposition of the expected return.
Revised: The core idea behind our reinforcement learning approach is to apply spectral decomposition to the reward stream, and thus learn a spectral decomposition of the expected return.

Original: Before delving into the technical details, see Table 1 for a worked example that highlights the intuition.
Revised: Before delving into the technical details, see Table 1 for a worked example that highlights the intuition.

Original: The discount
Revised: The discount

Original: [Figure]
Revised: [Figure]

Original: , i

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: ⇒ Spectral representation of 6.5 = (1, 1, 0.875, 0, 0, 0, …)

Original: [Figure]
Revised: [Figure]

Original: 2 i )

Original: Spectral reward components
Revised: Spectral reward components

Original: [Table]
Revised: [Table]

Original: factor is γ = 0 . 99 , and the return over the ﬁrst ﬁve time steps is 2.254.
Revised: factor is γ = 0 . 99 , and the return over the ﬁrst ﬁve time steps is 2.254.

Original: The usual way of calculating the return is, of course, to multiply the rewards by γ t then sum.
Revised: The usual way of calculating the return is, of course, to multiply the rewards by γ t then sum.

Original: The alternative, spectral approach is to ﬁrst compute the spectral decomposition of the rewards according to Equation 5, then calculate the returns arising from each reward frequency.
Revised: The alternative, spectral approach is to ﬁrst compute the spectral decomposition of the rewards according to Equation 5, then calculate the returns arising from each reward frequency.

Original: We refer to the decomposed rewards as spectral rewards , and to the decomposed returns as spectral returns .
Revised: We refer to the decomposed rewards as spectral rewards , and to the decomposed returns as spectral returns .

Original: The overall return of 2.254 can be found by multiplying the spectral returns by their corresponding weights of b i then summing.
Revised: The overall return of 2.254 can be found by multiplying the spectral returns by their corresponding weights of b i then summing.

Original: To appreciate why we go to this extra effort in calculating the return, note that the spectral rewards are all bound to [-1, 1].
Revised: To appreciate why we go to this extra effort in calculating the return, note that the spectral rewards are all bound to [-1, 1].

Original: This in turn constrains the magnitude of the spectral returns to be no more than 1 / (1 - γ ), which, again, is useful from a function approximation standpoint.
Revised: This in turn constrains the magnitude of the spectral returns to be no more than 1 / (1 - γ ), which, again, is useful from a function approximation standpoint.

Original: Moreover, note that the relative sparsity of large rewards is made more apparent by this approach: As we progress from r 02 to r 32 , the magnitudes of the spectral reward components decrease, with only the two largest magnitude rewards triggering a response in r 32 .
Revised: Moreover, note that the relative sparsity of large rewards is made more apparent by this approach: As we progress from r 02 to r 32 , the magnitudes of the spectral reward components decrease, with only the two largest magnitude rewards triggering a response in r 32 .

Original: 4. S PECTRAL Q- LEARNING
Revised: 4. S PECTRAL Q- LEARNING

Original: To translate the above intuition into a concrete algorithm, we begin by deﬁning the spectral actionvalue function as:
Revised: To translate the above intuition into a concrete algorithm, we begin by deﬁning the spectral actionvalue function as:

Original: [Equation]
Revised: [Equation]

Original: where r ib,t denotes the i th element of the spectral reward at time t .
Revised: where r ib,t denotes the i th element of the spectral reward at time t .

Original: As shown in the Appendix, it is straightforward to prove via the linearity of expectation that the standard action-value function can be recovered as the weighted sum:
Revised: As shown in the Appendix, it is straightforward to prove via the linearity of expectation that the standard action-value function can be recovered as the weighted sum:

Original: [Equation]
Revised: [Equation]

Original: This leads to a natural extension of the Q-learning algorithm (Watkins, 1989), whereby the greedy action is found by maximising this expression with respect to the available actions, and Q π ( s, a, i ) is trained via the standard Bellman backup, except that the reward is replaced by the i th element of the spectral reward.
Revised: This leads to a natural extension of the Q-learning algorithm (Watkins, 1989), whereby the greedy action is found by maximising this expression with respect to the available actions, and Q π ( s, a, i ) is trained via the standard Bellman backup, except that the reward is replaced by the i th element of the spectral reward.

Original: For clarity, the full pseudocode for this algorithm, which we term Spectral
Revised: For clarity, the full pseudocode for this algorithm, which we term Spectral

Original: Q-learning , is provided in the Appendix (see Algorithm 1).
Revised: Q-learning , is provided in the Appendix (see Algorithm 1).

Original: An important practical limitation of this algorithm is that it is impossible to train over full, inﬁnite spectrum; that is, one must choose a ﬁnite upper bound for i .
Revised: An important practical limitation of this algorithm is that it is impossible to train over full, inﬁnite spectrum; that is, one must choose a ﬁnite upper bound for i .

Original: (This would not be an issue if we knew the maximum reward magnitude in advance, but for generality we do not wish to assume this.)
Revised: (This would not be an issue if we knew the maximum reward magnitude in advance, but for generality we do not wish to assume this.)

Original: Regardless, for base b , observe that the ﬁrst N + 1 frequencies capture all rewards up to magnitude ( b N +1 − 1) / ( b − 1) , since the bucket widths form a geometric series.
Revised: Regardless, for base b , observe that the ﬁrst N + 1 frequencies capture all rewards up to magnitude ( b N +1 − 1) / ( b − 1) , since the bucket widths form a geometric series.

Original: Given the exponential nature of this term, it is possible to capture large magnitude rewards with a relatively small value of N .
Revised: Given the exponential nature of this term, it is possible to capture large magnitude rewards with a relatively small value of N .

Original: For example, with b = 2 and N = 20 , the algorithm fully captures all rewards of magnitude up tomillion, which is more than sufﬁcient for Atari .
Revised: For example, with b = 2 and N = 20 , the algorithm fully captures all rewards of magnitude up tomillion, which is more than sufﬁcient for Atari .

Original: Taking this logic one step further, we arrive at the following proposition, which is proved in the Appendix:
Revised: Taking this logic one step further, we arrive at the following proposition, which is proved in the Appendix:

Original: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .
Revised: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .

Original: Now that we have presented the full algorithm, it is worth reﬂecting on why we use a spectral representation for the rewards instead of, say, a binary encoding.
Revised: Now that we have presented the full algorithm, it is worth reﬂecting on why we use a spectral representation for the rewards instead of, say, a binary encoding.

Original: Put simply, very small differences in reward should not matter much to a rational agent.
Revised: Put simply, very small differences in reward should not matter much to a rational agent.

Original: For example, it should not matter much whether a particular reward’s magnitude is 63 or 64.
Revised: For example, it should not matter much whether a particular reward’s magnitude is 63 or 64.

Original: However, under a binary encoding, 63 is represented as 0111111, while 64 is represented as 1000000.
Revised: However, under a binary encoding, 63 is represented as 0111111, while 64 is represented as 1000000.

Original: Since a reward of 63 activates more frequencies than a reward of 64, it would thus trigger a larger TD error at the start of training.
Revised: Since a reward of 63 activates more frequencies than a reward of 64, it would thus trigger a larger TD error at the start of training.

Original: The thermometer-style encoding introduced earlier addresses this problem.
Revised: The thermometer-style encoding introduced earlier addresses this problem.

Original:  S PECTRAL DQN
Revised:  S PECTRAL DQN

Original: In order to create a deep learning version of the Spectral Q-learning algorithm, we modify the
Revised: In order to create a deep learning version of the Spectral Q-learning algorithm, we modify the

Original: DQN algorithm in an analogous way to which we modiﬁed the Q-learning algorithm in the previous section.
Revised: DQN algorithm in an analogous way to which we modiﬁed the Q-learning algorithm in the previous section.

Original: We name the resultant algorithm Spectral DQN .
Revised: We name the resultant algorithm Spectral DQN .

Original: To parameterise all N + 1 frequencies of the spectral action-value function via a single network, we multiply the number of outputs on the standard DQN network by the same count.
Revised: To parameterise all N + 1 frequencies of the spectral action-value function via a single network, we multiply the number of outputs on the standard DQN network by the same count.

Original: One crucial detail here is that we initialise the weights and biases of the ﬁnal layer to zero.
Revised: One crucial detail here is that we initialise the weights and biases of the ﬁnal layer to zero.

Original: This allows us to set
Revised: This allows us to set

Original: N to a conservatively large value, since it ensures that superﬂuous heads have no effect on training.
Revised: N to a conservatively large value, since it ensures that superﬂuous heads have no effect on training.

Original: Moreover, it ensures that the action-value calculations do not blow out due to the exponential weights in Equation 7 being applied to random noise.
Revised: Moreover, it ensures that the action-value calculations do not blow out due to the exponential weights in Equation 7 being applied to random noise.

Original: For the training loss, we use a weighted sum of the temporal difference errors for each frequency:
Revised: For the training loss, we use a weighted sum of the temporal difference errors for each frequency:

Original: [Equation]
Revised: [Equation]

Original: (8) where w i is the error weighting given to the i th frequency, s is a state sampled uniformly from the replay memory, a is the sampled action, s (cid:48) is the next state, r is the reward (with spectral components r ib for i ∈ { 0 , . . .
Revised: (8) where w i is the error weighting given to the i th frequency, s is a state sampled uniformly from the replay memory, a is the sampled action, s (cid:48) is the next state, r is the reward (with spectral components r ib for i ∈ { 0 , . . .

Original: , N } ), θ are the parameters of the network being trained, and θ − are the parameters of the periodically refreshed target network.
Revised: , N } ), θ are the parameters of the network being trained, and θ − are the parameters of the periodically refreshed target network.

Original: An important detail here is the approach we take to weighting the loss for each frequency.
Revised: An important detail here is the approach we take to weighting the loss for each frequency.

Original: Given that the ith frequency is weighted by b i when calculating the overall action-values, it is tempting to set w i
Revised: Given that the ith frequency is weighted by b i when calculating the overall action-values, it is tempting to set w i

Original: = b i .
Revised: = b i .

Original: However, this goes completely against one of the aims of our approach, which is to prevent the large reward frequencies from dominating the loss.
Revised: However, this goes completely against one of the aims of our approach, which is to prevent the large reward frequencies from dominating the loss.

Original: In fact, we show in the next section that weighting by b i has a disastrous impact in the two extreme domains introduced earlier.
Revised: In fact, we show in the next section that weighting by b i has a disastrous impact in the two extreme domains introduced earlier.

Original: In light of the above, rather than weighting by b
Revised: In light of the above, rather than weighting by b

Original: i , we strive to give all frequencies equal weighting.
Revised: i , we strive to give all frequencies equal weighting.

Original: However, we do not simply set w i = 1 , because doing so would effectively give too much emphasis to the small reward frequencies.
Revised: However, we do not simply set w i = 1 , because doing so would effectively give too much emphasis to the small reward frequencies.

Original: As pointed out by van Hasselt et al.
Revised: As pointed out by van Hasselt et al.

Original: , the gradients propagated to the Q-network’s hidden layers depend quadratically on the scale of the targets (see Section 2. of their work).
Revised: , the gradients propagated to the Q-network’s hidden layers depend quadratically on the scale of the targets (see Section 2. of their work).

Original: Since the lower frequencies have denser rewards, and hence larger training targets, they will dominate the gradients at the hidden layers if this effect is not accounted for.
Revised: Since the lower frequencies have denser rewards, and hence larger training targets, they will dominate the gradients at the hidden layers if this effect is not accounted for.

Original: To address this, we set w i = 

Revised: To address this, we set w i =

Original: /σ 

Revised: /σ

Original: i , where σ i is the standard deviation of the training targets for frequency
Revised: i , where σ i is the standard deviation of the training targets for frequency

Original: i , calculated as a running average as in the Pop-Art algorithm.
Revised: i , calculated as a running average as in the Pop-Art algorithm.

Original: Finally, to ensure that this correction only affects the hidden layers, we unscale the gradients at the ﬁnal layer to train as if w i = 1 , as in standard DQN.
Revised: Finally, to ensure that this correction only affects the hidden layers, we unscale the gradients at the ﬁnal layer to train as if w i = 1 , as in standard DQN.

Original: (Implementing this is not especially difﬁcult; the precise details can be found in our source code.)
Revised: (Implementing this is not especially difﬁcult; the precise details can be found in our source code.)

Original: In Appendix B.1, we provide an ablation where we do not correct the hidden layer gradients, and we ﬁnd that this has a signiﬁcant adverse effect on performance.
Revised: In Appendix B.1, we provide an ablation where we do not correct the hidden layer gradients, and we ﬁnd that this has a signiﬁcant adverse effect on performance.

Original:  R ETURNING TO THE E XTREME D OMAINS
Revised:  R ETURNING TO THE E XTREME D OMAINS

Original: To test how well Spectral DQN meets our aims, we ﬁrst applied it to the two extreme domains from
Revised: To test how well Spectral DQN meets our aims, we ﬁrst applied it to the two extreme domains from

Original: Section 3 – Ponglantis and Exponential Pong – where reward progressivity is clearly a key issue.
Revised: Section 3 – Ponglantis and Exponential Pong – where reward progressivity is clearly a key issue.

Original: In addition, to probe the importance of the frequency loss weighting, we included a variant of Spectral
Revised: In addition, to probe the importance of the frequency loss weighting, we included a variant of Spectral

Original: DQN with exponential loss weighting, i.e. w i =
Revised: DQN with exponential loss weighting, i.e. w i =

Original: b i , which we argued against in the previous section.
Revised: b i , which we argued against in the previous section.

Original: The results of this experiment, including the baseline results from Section 3, are shown in Figure 3, with each curve averaged over 5 seeds.
Revised: The results of this experiment, including the baseline results from Section 3, are shown in Figure 3, with each curve averaged over 5 seeds.

Original: There are two clear ﬁndings here: (1) Spectral DQN is much better at handling extreme reward progressivity than the baselines.
Revised: There are two clear ﬁndings here: (1) Spectral DQN is much better at handling extreme reward progressivity than the baselines.

Original: It scores much higher in Ponglantis , and makes far more meaningful progress in Exponential Pong , eventually learning to beat the computer player comfortably on
Revised: It scores much higher in Ponglantis , and makes far more meaningful progress in Exponential Pong , eventually learning to beat the computer player comfortably on

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: average.
Revised: average.

Original: The loss weighting is of crucial importance, with the agent using exponential weights performing worse than even DQN+TC.
Revised: The loss weighting is of crucial importance, with the agent using exponential weights performing worse than even DQN+TC.

Original: The latter ﬁnding matches our expectations; as we noted in Section 3, target compression does at least somewhat mitigate the overemphasis given to large reward regions of the task
Revised: The latter ﬁnding matches our expectations; as we noted in Section 3, target compression does at least somewhat mitigate the overemphasis given to large reward regions of the task

Original: [Figure]
Revised: [Figure]

Original: while Spectral DQN with w i =
Revised: while Spectral DQN with w i =

Original: b i gives heavy emphasis to such regions.
Revised: b i gives heavy emphasis to such regions.

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: we examine how the agents’ training errors evolve as they play Exponential Pong .
Revised: we examine how the agents’ training errors evolve as they play Exponential Pong .

Original: The errors are broken down by the player’s current score ∈ [0 , 20] , with bluer lines corresponding to states with a low player score and redder lines corresponding to states with high player scores.
Revised: The errors are broken down by the player’s current score ∈ [0 , 20] , with bluer lines corresponding to states with a low player score and redder lines corresponding to states with high player scores.

Original: To make the graphs comparable, we plot percentage TD errors, i [Figure] e.:
Revised: To make the graphs comparable, we plot percentage TD errors, i.e.:

Original: [Equation]
Revised: [Equation]

Original: where the means are running averages, calculated from states sampled from the replay buffer.
Revised: where the means are running averages, calculated from states sampled from the replay buffer.

Original: In all cases we report the true TD error, i.e. for DQN+TC we report the error on the uncompressed action-values, and for Spectral DQN we report the error on the full, summed action-values.
Revised: In all cases we report the true TD error, i.e. for DQN+TC we report the error on the uncompressed action-values, and for Spectral DQN we report the error on the full, summed action-values.

Original: The instability of Pop-Art is quite apparent here: After a very brief period of stable learning, the TD errors start spiking as soon as the agent reaches even middling scores.
Revised: The instability of Pop-Art is quite apparent here: After a very brief period of stable learning, the TD errors start spiking as soon as the agent reaches even middling scores.

Original: Consistent with our earlier analysis, DQN+TC appears a lot more stable.
Revised: Consistent with our earlier analysis, DQN+TC appears a lot more stable.

Original: For states with low player scores, the TD percentage errors hover at around 5%, although the agent never really learns accurate action-values for states where the player score exceeds ≈ 7 points.
Revised: For states with low player scores, the TD percentage errors hover at around 5%, although the agent never really learns accurate action-values for states where the player score exceeds ≈ 7 points.

Original: At ﬁrst this might seem strange – since the reward scale increases exponentially with the player’s score, states with a high player score ought to be effectively prioritised by the loss function.
Revised: At ﬁrst this might seem strange – since the reward scale increases exponentially with the player’s score, states with a high player score ought to be effectively prioritised by the loss function.

Original: However, there is an additional effect to consider here: As soon the agent’s predictions start to deteriorate for the lower-score states, its policy for these states likewise deteriorates and it starts struggling to reach the higher-score states.
Revised: However, there is an additional effect to consider here: As soon the agent’s predictions start to deteriorate for the lower-score states, its policy for these states likewise deteriorates and it starts struggling to reach the higher-score states.

Original: As such, the agent seems to get stuck in a middle ground, where it does not encounter the higher-score states often enough to learn accurate action-values there, nor for the estimates in the lower-score states to truly deteriorate.
Revised: As such, the agent seems to get stuck in a middle ground, where it does not encounter the higher-score states often enough to learn accurate action-values there, nor for the estimates in the lower-score states to truly deteriorate.

Original: It appears that a TD percentage error of around 5% is a crucial cutoff line, where the agent is right on the cusp of being able to play well, but playing better, conversely, leads to deterioration.
Revised: It appears that a TD percentage error of around 5% is a crucial cutoff line, where the agent is right on the cusp of being able to play well, but playing better, conversely, leads to deterioration.

Original: In contrast to both Pop-Art and DQN+TC, when Spectral DQN reaches a new high score, it quickly learns accurate action-value estimates for the associated states, as indicated by the sharply dropping red lines in Figure 4.
Revised: In contrast to both Pop-Art and DQN+TC, when Spectral DQN reaches a new high score, it quickly learns accurate action-value estimates for the associated states, as indicated by the sharply dropping red lines in Figure 4.

Original: Moreover, the TD errors for the lower-score states remain stable as the agent improves, almost certainly because Spectral DQN’s loss does not give so much weight to large magnitude rewards.
Revised: Moreover, the TD errors for the lower-score states remain stable as the agent improves, almost certainly because Spectral DQN’s loss does not give so much weight to large magnitude rewards.

Original: Finally, note that the agent’s sudden rise in performance at around 2 .
Revised: Finally, note that the agent’s sudden rise in performance at around 2 .

Original: 5 × 10frames in Figure 3 seems to coincide with the errors reducing below the 5% line, adding further credence to the analysis above.
Revised: 5 × 10frames in Figure 3 seems to coincide with the errors reducing below the 5% line, adding further credence to the analysis above.

Original:  E XPERIMENTS IN S TANDARD A TARI G AMES
Revised:  E XPERIMENTS IN S TANDARD A TARI G AMES

Original: To validate that Spectral DQN’s success in extreme domains does not come at the expense of performance in less constructed tasks, we also applied it to 6 standard Atari games.
Revised: To validate that Spectral DQN’s success in extreme domains does not come at the expense of performance in less constructed tasks, we also applied it to 6 standard Atari games.

Original: Moreover, we included 3 games where reward clipping is known to be detrimental, as per van Hasselt et al.
Revised: Moreover, we included 3 games where reward clipping is known to be detrimental, as per van Hasselt et al.

Original: Bowling , Centipede and Ms. Pacman .
Revised: Bowling , Centipede and Ms. Pacman .

Original: This was done to validate that the agent’s earlier success was not attributable to it somehow performing an implicit form of reward clipping.
Revised: This was done to validate that the agent’s earlier success was not attributable to it somehow performing an implicit form of reward clipping.

Original: The results of these experiments are shown in Figure 5, with all curves averaged over 5 seeds.
Revised: The results of these experiments are shown in Figure 5, with all curves averaged over 5 seeds.

Original: While Spectral DQN’s performance relative to the baselines is not as exceptional here as in the extreme domains, it remains the strongest overall.
Revised: While Spectral DQN’s performance relative to the baselines is not as exceptional here as in the extreme domains, it remains the strongest overall.

Original: Though the agent underperforms DQN+TC in
Revised: Though the agent underperforms DQN+TC in

Original: Centipede , it noticeably outperforms both baselines in Bowling , Chopper Command and Video Pinball .
Revised: Centipede , it noticeably outperforms both baselines in Bowling , Chopper Command and Video Pinball .

Original: On the other games, it performs at least on par with the benchmarks.
Revised: On the other games, it performs at least on par with the benchmarks.

Original: Moreover, the fact that Spectral DQN’s performance compares favourably to Pop-Art in Bowling , Centipede and Ms.
Revised: Moreover, the fact that Spectral DQN’s performance compares favourably to Pop-Art in Bowling , Centipede and Ms.

Original: Pacman shows that it is truly responding to the unclipped reward.
Revised: Pacman shows that it is truly responding to the unclipped reward.

Original: Since Video Pinball exhibits reward progressivity, as explained in the introduction, Spectral DQN’s strong performance there ﬁts well with our hypothesis.
Revised: Since Video Pinball exhibits reward progressivity, as explained in the introduction, Spectral DQN’s strong performance there ﬁts well with our hypothesis.

Original: On the other hand, while Ms. Pacman also has progressive rewards — the player receives an increasing bonus for eating multiple ghosts with a single power pellet — these rewards come so close together that they are within the agent’s temporal horizon at the start of an episode.
Revised: On the other hand, while Ms. Pacman also has progressive rewards — the player receives an increasing bonus for eating multiple ghosts with a single power pellet — these rewards come so close together that they are within the agent’s temporal horizon at the start of an episode.

Original: That is, there is not a small-expected-return phase preceding a large-expected-return phase, which we hypothesised to be a central component of the issue.
Revised: That is, there is not a small-expected-return phase preceding a large-expected-return phase, which we hypothesised to be a central component of the issue.

Original: (We return to this point and discuss further in the next section.)
Revised: (We return to this point and discuss further in the next section.)

Original: Besides addressing reward progressivity, another possible advantage of Spectral DQN in high scoring games is simply that its training targets are more constrained.
Revised: Besides addressing reward progressivity, another possible advantage of Spectral DQN in high scoring games is simply that its training targets are more constrained.

Original: In games such as Video Pinball and Chopper Command , the expected returns can reach into the thousands, so even with target compression this may be problematic for DQN+TC.
Revised: In games such as Video Pinball and Chopper Command , the expected returns can reach into the thousands, so even with target compression this may be problematic for DQN+TC.

Original: On the other hand, the only game where DQN+TC clearly outperforms Spectral DQN is Centipede , which is also high scoring.
Revised: On the other hand, the only game where DQN+TC clearly outperforms Spectral DQN is Centipede , which is also high scoring.

Original: One possible explanation for this is that the reward structure in Centipede “confuses” Spectral DQN: Since large rewards trigger both low and high frequencies, it is likely that Spectral DQN learns a positive association between the various components of the return.
Revised: One possible explanation for this is that the reward structure in Centipede “confuses” Spectral DQN: Since large rewards trigger both low and high frequencies, it is likely that Spectral DQN learns a positive association between the various components of the return.

Original: However, in Centipede , losing a life triggers a stream of consecutive small rewards.
Revised: However, in Centipede , losing a life triggers a stream of consecutive small rewards.

Original: Therefore, associating small and large rewards may cause Spectral
Revised: Therefore, associating small and large rewards may cause Spectral

Original: DQN to predict an inﬂated return from dying.
Revised: DQN to predict an inﬂated return from dying.

Original: There has been some conjecture that an advantage of combining distributional RL (Bellemare et al., 2017; Lyle et al., 2019) with function approximation is that it forces the agent to learn a richer representation.
Revised: There has been some conjecture that an advantage of combining distributional RL (Bellemare et al., 2017; Lyle et al., 2019) with function approximation is that it forces the agent to learn a richer representation.

Original: The same argument may also apply to our approach.
Revised: The same argument may also apply to our approach.

Original: In particular, in some tasks, small rewards are of more informational value than their face value represents.
Revised: In particular, in some tasks, small rewards are of more informational value than their face value represents.

Original: For example, in Video
Revised: For example, in Video

Original: [Figure]
Revised: [Figure]

Original: Pinball , where rewards range into the thousands, the play ﬁeld includes “spinners” that pay only one point per hit.
Revised: Pinball , where rewards range into the thousands, the play ﬁeld includes “spinners” that pay only one point per hit.

Original: Despite their negligible face value, hitting the spinners can dramatically change the velocity of the ball, so giving the lower frequencies more weight in the training loss may promote a stronger representation.
Revised: Despite their negligible face value, hitting the spinners can dramatically change the velocity of the ball, so giving the lower frequencies more weight in the training loss may promote a stronger representation.

Original: Similarly, in Bowling , certain frequencies only activate after bowling strikes and spares.
Revised: Similarly, in Bowling , certain frequencies only activate after bowling strikes and spares.

Original: Since the game does not have a strongly progressive reward structure, improved representation learning is a more plausible explanation of Spectral DQN’s outperformance.
Revised: Since the game does not have a strongly progressive reward structure, improved representation learning is a more plausible explanation of Spectral DQN’s outperformance.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this paper, we brought attention to the idea of reward progressivity, a task property that poses a potential issue for value-based reinforcement learning agents.
Revised: In this paper, we brought attention to the idea of reward progressivity, a task property that poses a potential issue for value-based reinforcement learning agents.

Original: Speciﬁcally, an agent’s performance in relatively unrewarding regions of a task may degrade once it discovers larger rewards.
Revised: Speciﬁcally, an agent’s performance in relatively unrewarding regions of a task may degrade once it discovers larger rewards.

Original: To address this issue, we proposed Spectral DQN, which decomposes the expected return magnitudinally, allowing the losses arising across different regions of the task to be balanced.
Revised: To address this issue, we proposed Spectral DQN, which decomposes the expected return magnitudinally, allowing the losses arising across different regions of the task to be balanced.

Original: Our experiments showed this approach to be particularly effective in domains exhibiting extreme reward progressivity, while still yielding more than competitive performance in less constructed domains.
Revised: Our experiments showed this approach to be particularly effective in domains exhibiting extreme reward progressivity, while still yielding more than competitive performance in less constructed domains.

Original: Our discussion of the Ms. Pacman results from the previous section raised an interesting point: There is clearly a link between reward progressivity and discounting.
Revised: Our discussion of the Ms. Pacman results from the previous section raised an interesting point: There is clearly a link between reward progressivity and discounting.

Original: If it were somehow possible to train reliably with γ = 1 , a Ponglantis agent would be able to see that reaching the large, distant rewards in the Atlantis phase is contingent on ﬁrst performing well in Pong .
Revised: If it were somehow possible to train reliably with γ = 1 , a Ponglantis agent would be able to see that reaching the large, distant rewards in the Atlantis phase is contingent on ﬁrst performing well in Pong .

Original: However, despite some recent progress, DQN-style agents typically destabilise with discounts very close to 1.
Revised: However, despite some recent progress, DQN-style agents typically destabilise with discounts very close to 1.

Original: Agent57 (Badia et al., 2020a) trains multiple value functions across a range of discounts, ranging as high as γ = 0 . 9999 .
Revised: Agent57 (Badia et al., 2020a) trains multiple value functions across a range of discounts, ranging as high as γ = 0 . 9999 .

Original: However, the authors note that γ = 0 .
Revised: However, the authors note that γ = 0 .

Original: 9999 generally yields very unstable learning, and they rely on a meta-controller to select smaller discounts in games where learning destabilises.
Revised: 9999 generally yields very unstable learning, and they rely on a meta-controller to select smaller discounts in games where learning destabilises.

Original: As such, it does not offer a general solution to the problem.
Revised: As such, it does not offer a general solution to the problem.

Original: The idea of setting γ = 1 also raises the issue of reward sparsity: If the agent is cognizant of large, distant rewards, then smaller, more immediate rewards may appear so negligible that the task effectively becomes a sparse reward problem.
Revised: The idea of setting γ = 1 also raises the issue of reward sparsity: If the agent is cognizant of large, distant rewards, then smaller, more immediate rewards may appear so negligible that the task effectively becomes a sparse reward problem.

Original: We noted in Section 3 that another idea for addressing reward progressivity is to apply target compression, but with logarithmic squashing instead of a square root.
Revised: We noted in Section 3 that another idea for addressing reward progressivity is to apply target compression, but with logarithmic squashing instead of a square root.

Original: However, as Pohlen et al.
Revised: However, as Pohlen et al.

Original: (2018) note, the inverse of a logarithmic mapping is not Lipschitz continuous, and thus the transformed
Revised: (2018) note, the inverse of a logarithmic mapping is not Lipschitz continuous, and thus the transformed

Original: Bellman is no longer guaranteed to be a contraction.
Revised: Bellman is no longer guaranteed to be a contraction.

Original: On the other hand, one way to get around poor convergence in practice is to use a shaper discount factor.
Revised: On the other hand, one way to get around poor convergence in practice is to use a shaper discount factor.

Original: Van Seijen et al.
Revised: Van Seijen et al.

Original: (2019) show that it is possible to achieve performance comparable to DQN via a logarithmic squash, provided the discount is small ( γ = 0 . 96 ).
Revised: (2019) show that it is possible to achieve performance comparable to DQN via a logarithmic squash, provided the discount is small ( γ = 0 . 96 ).

Original: However, they clip the rewards, which greatly constrains the gradient of the inverse mapping h − 1 , and thus limits the degree to which their implementation transgresses the theory.
Revised: However, they clip the rewards, which greatly constrains the gradient of the inverse mapping h − 1 , and thus limits the degree to which their implementation transgresses the theory.

Original: Moreover, the performance of their method degrades signiﬁcantly with γ = 0 . 99 , consistent with convergence issues (see Appendix C.2 in their work).
Revised: Moreover, the performance of their method degrades signiﬁcantly with γ = 0 . 99 , consistent with convergence issues (see Appendix C.2 in their work).

Original: Unfortunately, it is simple to provide examples where γ = 0 .
Revised: Unfortunately, it is simple to provide examples where γ = 0 .

Original: 96 will lead to poor performance.
Revised: 96 will lead to poor performance.

Original: For example, in the game Bowling, there is a delay of about 100 frames between the ball being released and the subsequent reward.
Revised: For example, in the game Bowling, there is a delay of about 100 frames between the ball being released and the subsequent reward.

Original: However, if the player bowls a spare, the reward is delayed by a further 100 frames.
Revised: However, if the player bowls a spare, the reward is delayed by a further 100 frames.

Original: Thus, under a discount of 0.96, it is better to consistently bowl 9s than to bowl spares.
Revised: Thus, under a discount of 0.96, it is better to consistently bowl 9s than to bowl spares.

Original: Our work ﬁts into a broader line of research whereby reinforcement learning agents seek to predict quantities other than just the mean expected return (Jaderberg et al., 2016; Bellemare et al., 2017;
Revised: Our work ﬁts into a broader line of research whereby reinforcement learning agents seek to predict quantities other than just the mean expected return (Jaderberg et al., 2016; Bellemare et al., 2017;

Original: Fedus et al., 2019).
Revised: Fedus et al., 2019).

Original: Of this related work, one of the most similar approaches to ours is that of Romoff et al.
Revised: Of this related work, one of the most similar approaches to ours is that of Romoff et al.

Original: (2019), who learn the value function in a modular way by decomposing based on the discount factor.
Revised: (2019), who learn the value function in a modular way by decomposing based on the discount factor.

Original: Distributional reinforcement learning (Bellemare et al., 2017; Dabney et al., 2018b;a) is another closely related approach.
Revised: Distributional reinforcement learning (Bellemare et al., 2017; Dabney et al., 2018b;a) is another closely related approach.

Original: Finally, an older line of work in this vein is decomposed RL (Russell & Zimdars, 2003; Sprague & Ballard, 2003; Van Seijen et al., 2017), whereby an agent is responsible for predicting only a portion of the action-value function.
Revised: Finally, an older line of work in this vein is decomposed RL (Russell & Zimdars, 2003; Sprague & Ballard, 2003; Van Seijen et al., 2017), whereby an agent is responsible for predicting only a portion of the action-value function.

Original: While some of these works bear many surface-level similarities to our own — the distributional C51 agent (Bellemare et al., 2017) in particular bears quite a similar network architecture to ours — on a deeper level, they are not addressing the same problem.
Revised: While some of these works bear many surface-level similarities to our own — the distributional C51 agent (Bellemare et al., 2017) in particular bears quite a similar network architecture to ours — on a deeper level, they are not addressing the same problem.

Original: Leveraging a probabilistic loss, as in C51, could foreseeably help to address the large training errors that arise in tasks like Exponential Pong .
Revised: Leveraging a probabilistic loss, as in C51, could foreseeably help to address the large training errors that arise in tasks like Exponential Pong .

Original: However, the method assumes that returns are constrained to [-10, 10], and it is unclear how this could be extended to unclipped rewards without requiring an intractable number of atoms in the distribution, or greatly reducing the precision of the atoms.
Revised: However, the method assumes that returns are constrained to [-10, 10], and it is unclear how this could be extended to unclipped rewards without requiring an intractable number of atoms in the distribution, or greatly reducing the precision of the atoms.

Original: A E XPERIMENTAL S ETUP
Revised: A E XPERIMENTAL S ETUP

Original: We injected stochasticity into the Atari environment via the no-ops regime used in the Nature DQN paper (Mnih et al., 2015).
Revised: We injected stochasticity into the Atari environment via the no-ops regime used in the Nature DQN paper (Mnih et al., 2015).

Original: Rather than conducting periodic evaluations of the agents with a small exploration constant, we simply report their training performance, since this arguably provides a more reliable measure of progress.
Revised: Rather than conducting periodic evaluations of the agents with a small exploration constant, we simply report their training performance, since this arguably provides a more reliable measure of progress.

Original: All curves in the paper were averaged over 5 seeds.
Revised: All curves in the paper were averaged over 5 seeds.

Original: Shaded regions represent bootstrapped single standard deviation conﬁdence intervals.
Revised: Shaded regions represent bootstrapped single standard deviation conﬁdence intervals.

Original: Hyperparameters speciﬁc to Spectral DQN are listed in Table 2, while hyperparameters common to all agents are listed in Table
Revised: Hyperparameters speciﬁc to Spectral DQN are listed in Table 2, while hyperparameters common to all agents are listed in Table

Original: [Table]
Revised: [Table]

Original: Spectral DQN used an identical network architecture to Nature DQN Mnih et al.
Revised: Spectral DQN used an identical network architecture to Nature DQN Mnih et al.

Original: (2015), except that we multiplied the number of outputs by N + 1 .
Revised: (2015), except that we multiplied the number of outputs by N + 1 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B A DDITIONAL E XPERIMENTS
Revised: B A DDITIONAL E XPERIMENTS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide the details of some additional experiments that had to be omitted from the main text due to space constraints.
Revised: we provide the details of some additional experiments that had to be omitted from the main text due to space constraints.

Original: B. A BLATION : N O E RROR W EIGHTING
Revised: B. A BLATION : N O E RROR W EIGHTING

Original: Recall from Section 5 that we weight the frequencies’ individual losses by w i = 

Revised: Recall from Section 5 that we weight the frequencies’ individual losses by w i =

Original: /σ 

Revised: /σ

Original: i , where σ i is the standard deviation of the training targets for frequency i .
Revised: i , where σ i is the standard deviation of the training targets for frequency i .

Original: In Figure
Revised: In Figure

Original: [Table]
Revised: [Table]

Original: we show the effect of ablating this adjustment and simply setting w i = 1 .
Revised: we show the effect of ablating this adjustment and simply setting w i = 1 .

Original: The results reveal that the adjustment is a crucial component, with the ablation proving disastrous in 4 of the 6 games.
Revised: The results reveal that the adjustment is a crucial component, with the ablation proving disastrous in 4 of the 6 games.

Original: (Note: It unsurprising that the
Revised: (Note: It unsurprising that the

Original: Fishing results are unaffected, because only one reward frequency is active in that game.)
Revised: Fishing results are unaffected, because only one reward frequency is active in that game.)

Original: B. E ASIER P ONGLANTIS
Revised: B. E ASIER P ONGLANTIS

Original: In Section 3, we applied Pop-Art and DQN+TC to the game Ponglantis .
Revised: In Section [Table] we applied Pop-Art and DQN+TC to the game Ponglantis .

Original: The results suggested that DQN+TC was somewhat more robust to reward progressivity.
Revised: The results suggested that DQN+TC was somewhat more robust to reward progressivity.

Original: Here, we provide some further experiments to determine just how strong the reward progressivity has to be for these agents to start deteriorating.
Revised: Here, we provide some further experiments to determine just how strong the reward progressivity has to be for these agents to start deteriorating.

Original: To hone in on the agents’ breaking points, we created two modiﬁed versions of the game: Ponglantis
Revised: To hone in on the agents’ breaking points, we created two modiﬁed versions of the game: Ponglantis

Original: Easier and Ponglantis Even Easier .
Revised: Easier and Ponglantis Even Easier .

Original: In Ponglantis Easier , the agent’s perceived rewards in the
Revised: In Ponglantis Easier , the agent’s perceived rewards in the

Original: Atlantis phase of the task are scaled down by a factor 10.
Revised: Atlantis phase of the task are scaled down by a factor 10.

Original: In Ponglantis Even Easier , they are scaled down by a factor of 100.
Revised: In Ponglantis Even Easier , they are scaled down by a factor of 100.

Original: The agents’ results in these games are shown in Figure 7.
Revised: The agents’ results in these games are shown in Figure 7.

Original: For comparability,
Revised: For comparability,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the plots show the true episodic return, rather than the perceived episodic return.
Revised: the plots show the true episodic return, rather than the perceived episodic return.

Original: We also include results for Spectral DQN as a reference point.
Revised: We also include results for Spectral DQN as a reference point.

Original: Relative to its performance in the original Ponglantis task, DQN+TC performs markedly better in both easier tasks.
Revised: Relative to its performance in the original Ponglantis task, DQN+TC performs markedly better in both easier tasks.

Original: Since the rewards jump by a factor of around 1000 in the original task, this suggests that DQN+TC can tolerate a jump in reward magnitude of around 100x.
Revised: Since the rewards jump by a factor of around 1000 in the original task, this suggests that DQN+TC can tolerate a jump in reward magnitude of around 100x.

Original: Further, this implies that
Revised: Further, this implies that

Original: DQN+TC’s underperformance relative to Spectral DQN in the main Atari experiments is attributable to more than just reward progressivity, as none of the standard Atari games exhibit this level of jump.
Revised: DQN+TC’s underperformance relative to Spectral DQN in the main Atari experiments is attributable to more than just reward progressivity, as none of the standard Atari games exhibit this level of jump.

Original: Our alternate hypotheses about representation learning and the relative ease with which the spectral architecture can represent large expected returns thus appear more likely.
Revised: Our alternate hypotheses about representation learning and the relative ease with which the spectral architecture can represent large expected returns thus appear more likely.

Original: Pop-Art, on the other hand, still performs poorly when the perceived rewards in Atlantis are scaled down by a factor of 10, only learning reliably when they are scaled down 100 times.
Revised: Pop-Art, on the other hand, still performs poorly when the perceived rewards in Atlantis are scaled down by a factor of 10, only learning reliably when they are scaled down 100 times.

Original: In this case, it is plausible that the algorithm struggles with reward progressivity in standard Atari games.
Revised: In this case, it is plausible that the algorithm struggles with reward progressivity in standard Atari games.

Original: B. R EVERSE P ONGLANTIS
Revised: B. R EVERSE P ONGLANTIS

Original: To test whether it is truly reward progressivity that is the key issue in Ponglantis , as opposed to just reward variability , we created one further task, Reverse Ponglantis , where the Pong and Atlantis phases are reversed.
Revised: To test whether it is truly reward progressivity that is the key issue in Ponglantis , as opposed to just reward variability , we created one further task, Reverse Ponglantis , where the Pong and Atlantis phases are reversed.

Original: The player progresses to the Pong phase upon losing a life in Atlantis .
Revised: The player progresses to the Pong phase upon losing a life in Atlantis .

Original: The agents’ results in this game are plotted in Figure 8.
Revised: The agents’ results in this game are plotted in Figure 8.

Original: While all agents improve steadily, which conﬁrms that progressivity is the main issue for Pop-Art, it is notable DQN+TC’s performance is actually worse here than in Ponglantis Even Easier .
Revised: While all agents improve steadily, which conﬁrms that progressivity is the main issue for Pop-Art, it is notable DQN+TC’s performance is actually worse here than in Ponglantis Even Easier .

Original: This is despite the fact that the agents now start in the high-scoring Atlantis phase, so there is no need to play Pong well.
Revised: This is despite the fact that the agents now start in the high-scoring Atlantis phase, so there is no need to play Pong well.

Original: From this result, it is clear that a key issue for DQN+TC is simply the reward scale in Atlantis .
Revised: From this result, it is clear that a key issue for DQN+TC is simply the reward scale in Atlantis .

Original: This adds further weight to our hypothesis from Section 7 that one of the main advantages of Spectral DQN in standard Atari games is the constrained nature of its training targets.
Revised: This adds further weight to our hypothesis from Section 7 that one of the main advantages of Spectral DQN in standard Atari games is the constrained nature of its training targets.

Original: [Figure]
Revised: [Figure]

Original: C S PECTRAL Q-L EARNING A LGORITHM
Revised: C S PECTRAL Q-L EARNING A LGORITHM

Original: Pseudocode for the Spectral Q-Learning algorithm is provided in Algorithm 1.
Revised: Pseudocode for the Spectral Q-Learning algorithm is provided in Algorithm 1.

Original:  do 9: 
Revised: 1: var: the base of the spectral decomposition, b 2: var: the number of spectral components trained, N 3: var: the 6: for do 7: Sample start do 9: Choose a according to (cid:15) -greedy policy derived

Original:  Take 
Revised: Q s a = i Q ( s , i ) 10: Take action a , observe r , s

Original: 11: a (cid:48)∗  arg max a (cid:48) (cid:80) Ni =

Revised: 11: a (cid:48)∗ = arg max a (cid:48) (cid:80) Ni =

Original: b  ( s (cid:48) , a  i  12: for i ∈ { 0 , 
Revised: b i Q ( s (cid:48) , a (cid:48) , i ) 12: for i ∈ { 0 , . . .

Original: . , N  do 13:
Revised: , N } do 13:

Original: Q ( s, a, i ) ← Q ( s, a, i ) + α [ r ib + γQ ( s (cid:48) , a (cid:48)∗ , i ) − Q ( s, a, i )] 14: s ← s (cid:48)
Revised: Q ( s, a, i ) ← Q ( s, a, i ) + α [ r ib + γQ ( s (cid:48) , a (cid:48)∗ , i ) − Q ( s, a, i )] 14: s ← s (cid:48)

Original: D D ERIVATION OF THE S PECTRAL D ECOMPOSITION F ORMULA
Revised: D D ERIVATION OF THE S PECTRAL D ECOMPOSITION F ORMULA

Original: Recall from Section 4.1 that the spectral representation of a number is found by laying buckets of exponentially increasing width on the number line, then calculating the proportion of each bucket that is ﬁlled.
Revised: Recall from Section 4.1 that the spectral representation of a number is found by laying buckets of exponentially increasing width on the number line, then calculating the proportion of each bucket that is ﬁlled.

Original: To calculate the proportion of the i th bucket that is ﬁlled by a non-negative number r , note that the sum of all the preceding buckets’ widths is equal to:
Revised: To calculate the proportion of the i th bucket that is ﬁlled by a non-negative number r , note that the sum of all the preceding buckets’ widths is equal to:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the amount leftover to ﬁll the i th bucket is:
Revised: Therefore, the amount leftover to ﬁll the i th bucket is:

Original: [Equation]
Revised: [Equation]

Original: To calculate the proportion of the i th bucket that is ﬁlled, we must divide this quantity by the width of the i th bucket (which is equal to b i ) then clamp the result to the allowable range of values:
Revised: To calculate the proportion of the i th bucket that is ﬁlled, we must divide this quantity by the width of the i th bucket (which is equal to b i ) then clamp the result to the allowable range of values:

Original: [Equation]
Revised: [Equation]

Original: Finally, to convert this into an odd function so that the representation of − r is equal to the negated representation of r , we replace r with | r | and multiply the result by sign ( r ) .
Revised: Finally, to convert this into an odd function so that the representation of − r is equal to the negated representation of r , we replace r with | r | and multiply the result by sign ( r ) .

Original: This yields the formula for the i th element of the spectral representation:
Revised: This yields the formula for the i th element of the spectral representation:

Original: [Equation]
Revised: [Equation]

Original: E P ROOFS E. P ROOF OF E QUATION
Revised: E P ROOFS E. P ROOF OF E QUATION

Original: Starting with the standard deﬁnition of the action-value function, then replacing the reward at each time step with its spectral decomposition, we have:
Revised: Starting with the standard deﬁnition of the action-value function, then replacing the reward at each time step with its spectral decomposition, we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since there is no interaction between the summation indices, we can rearrange (15) into:
Revised: Since there is no interaction between the summation indices, we can rearrange (15) into:

Original: [Equation]
Revised: [Equation]

Original: Finally, using the linearity of expectation, we can take the expectation operator inside and write:
Revised: Finally, using the linearity of expectation, we can take the expectation operator inside and write:

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF P ROPOSITION
Revised: E. P ROOF OF P ROPOSITION

Original: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .
Revised: Proposition 1 In the tabular setting, Spectral Q-learning behaves identically to standard Qlearning, provided that the rewards are bound in magnitude to no more than ( b N +1 − 1) / ( b − 1) .

Original: We prove this statement via induction.
Revised: We prove this statement via induction.

Original: The ﬁrst point to observe is that, provided the action-value estimates of the two algorithms agree at a given point in time, their (cid:15) -greedy policies at that time will also be equivalent.
Revised: The ﬁrst point to observe is that, provided the action-value estimates of the two algorithms agree at a given point in time, their (cid:15) -greedy policies at that time will also be equivalent.

Original: Therefore, assuming that the action-value functions agree at t = 

Revised: Therefore, assuming that the action-value functions agree at t =

Original: (e.g. they are both initialised to zero), all we need to prove is the inductive step, i.e. that agreement in action-values at time t implies agreement in action-values at time t + 1 .
Revised: (e.g. they are both initialised to zero), all we need to prove is the inductive step, i.e. that agreement in action-values at time t implies agreement in action-values at time t + 1 .

Original: To distinguish between the action-value estimates of the two algorithms at different time steps, we use the following notation:
Revised: To distinguish between the action-value estimates of the two algorithms at different time steps, we use the following notation:

Original: • Let ˆ Q ( s, a ; t ) denote the action-value estimates under standard Q-learning after t updates.
Revised: • Let ˆ Q ( s, a ; t ) denote the action-value estimates under standard Q-learning after t updates.

Original: • Let Q ( s, a ; t ) =
Revised: • Let Q ( s, a ; t ) =

Original: (cid:80) Ni =
Revised: (cid:80) Ni =

Original: b i Q ( s, a, i ; t ) denote the action-values estimates under Spectral Q-learning after t updates.
Revised: b i Q ( s, a, i ; t ) denote the action-values estimates under Spectral Q-learning after t updates.

Original: For the induction step, we need to show that Q ( s, a ; t +1)
Revised: For the induction step, we need to show that Q ( s, a ; t +

Original: =
Revised: =

Original: ˆ Q ( s, a; t + 1) ∀ s ∈S , a ∈ A given that Q ( s, a ; t ) =
Revised: ˆ Q s a t + 1) ∀ s ∈

Revised: S a ∈

Revised: A

Revised: given that

Original:  ˆ Q ( s, a ; t ) ∀ s ∈ S , a ∈ A .
Revised: Q ( s, a ; t ) = ˆ Q ( s, a ; t ) ∀ s ∈ S , a ∈ A .

Original: To begin, note that the standard Q-learning backup expressed under the above notation is:
Revised: To begin, note that the standard Q-learning backup expressed under the above notation is:

Original: [Equation]
Revised: [Equation]

Original: (19) where a (cid:48)∗ = arg max a (cid:48)
Revised: (19) where a (cid:48)∗ = arg max a (cid:48)

Original: ˆ Q ( s (cid:48) , a (cid:48)∗ ; t ) is the greedy action for the next state.
Revised: ˆ Q ( s (cid:48) , a (cid:48)∗ ; t ) is the greedy action for the next state.

Original: Since we assume equal action-values at time t , Spectral Q-learning will calculate the same greedy action for s (cid:48) .
Revised: Since we assume equal action-values at time t , Spectral Q-learning will calculate the same greedy action for s (cid:48) .

Original: Hence, the backup for each frequency of the spectral action-value function is:
Revised: Hence, the backup for each frequency of the spectral action-value function is:

Original: [Equation]
Revised: [Equation]

Original: After performing this update on each frequency
Revised: After performing this update on each frequency

Original: [Equation]
Revised: [Equation]

Original: the function at time t + 1 becomes:
Revised: the function at time t + 1 becomes:

Original: [Equation]
Revised: [Equation]

Original: Since the induction base case assumes that ˆ Q ( s [Equation] a ; t ) =
Revised: Since the induction base case assumes that ˆ Q ( s, a ; t ) =

Original: Q ( s
Revised: Q ( s

Original: [Equation]
Revised: [Equation]

Original: a ; t ) ∀ s ∈ S
Revised: a ; t ) ∀ s ∈ S

Original: [Equation]
Revised: [Equation]

Original: a ∈ A
Revised: a ∈ A

Original: [Equation]
Revised: [Equation]

Original: we can rewrite this as:
Revised: we can rewrite this as:

Original: [Equation]
Revised: [Equation]

Original: Now, all that remains to establish equality with the standard Q-learning update in (19) is to show that (cid:80) Ni =

Revised: Now, all that remains to establish equality with the standard Q-learning update in (19) is to show that (cid:80) Ni =

Original: b i r ib = r .
Revised: b i r ib = r .

Original: That is, we need to show that the partial sum over indices 0 to N fully captures the reward, or, equivalently, that r ib = 0 for all
Revised: That is, we need to show that the partial sum over indices 0 to N fully captures the reward, or, equivalently, that r ib = 0 for all

Original: i > = N + 1 .
Revised: i > = N + 1 .

Original: From the spectral decomposition formula (13) and the proposition’s statement that the rewards are bound to no more than ( b N +1 − 1) / ( b − 1) , we have: 
Revised: From the spectral decomposition formula (13) and the proposition’s statement that the rewards are bound to no more than ( b N +1 − 1) / ( b − 1) , we have: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, if i > = N + 1 , it implies that ( b N +1 − i − 1) ≤ 0 , so the whole expression in (30) becomes 0.
Revised: Now, if i > = N + 1 , it implies that ( b N +1 − i − 1) ≤ 0 , so the whole expression in (30) becomes 0.

Original: Thus (cid:80) Ni =

Revised: Thus (cid:80) Ni =

Original: b i r ib
Revised: b i r ib

Original: =
Revised: =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: i =

Revised: i =

Original: b i r ib =
Revised: b i r ib =

Original: r and the proof is complete.
Revised: r and the proof is complete.

Original: (cid:4)
Revised: (cid:4)