Original: B EYOND T ARGET N ETWORKS : I MPROVING D EEP QLEARNING WITH F UNCTIONAL R EGULARIZATION
Revised: B EYOND T ARGET N ETWORKS : I MPROVING D EEP QLEARNING WITH F UNCTIONAL R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Much of the recent successes in Deep Reinforcement Learning have been based on minimizing the squared Bellman error.
Revised: Much of the recent successes in Deep Reinforcement Learning have been based on minimizing the squared Bellman error.

Original: However, training is often unstable due to fast-changing target Q -values, and target networks are employed to stabilize training by using an additional set of lagging parameters.
Revised: However, training is often unstable due to fast-changing target Q -values, and target networks are employed to stabilize training by using an additional set of lagging parameters.

Original: Despite their advantages, target networks can inhibit the propagation of newly-encountered rewards which may ultimately slow down training.
Revised: Despite their advantages, target networks can inhibit the propagation of newly-encountered rewards which may ultimately slow down training.

Original: In this work, we address this issue by augmenting the squared Bellman error with a functional regularizer.
Revised: In this work, we address this issue by augmenting the squared Bellman error with a functional regularizer.

Original: Unlike target networks, the regularization we propose here is explicit and enables us to use up-to-date parameters as well as control the regularization.
Revised: Unlike target networks, the regularization we propose here is explicit and enables us to use up-to-date parameters as well as control the regularization.

Original: This leads to a faster yet more stable training method.
Revised: This leads to a faster yet more stable training method.

Original: Across a range of Atari environments, we demonstrate empirical improvements over target-network based methods in terms of both sample efﬁciency and performance.
Revised: Across a range of Atari environments, we demonstrate empirical improvements over target-network based methods in terms of both sample efﬁciency and performance.

Original: In summary, our approach provides a fast and stable alternative to replace the standard squared Bellman error.
Revised: In summary, our approach provides a fast and stable alternative to replace the standard squared Bellman error.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In practice, Deep Q -learning (DQL) methods demonstrate instability due to being trained with offpolicy data, function-approximations, and bootstrapping (Sutton & Barto, 2018; Van Hasselt et al., 2018).
Revised: In practice, Deep Q -learning (DQL) methods demonstrate instability due to being trained with offpolicy data, function-approximations, and bootstrapping (Sutton & Barto, 2018; Van Hasselt et al., 2018).

Original: The Bellman error, which is used to train DQL methods, is computed with target Q -values estimated by a Deep Neural Network (DNN) with constantly changing parameters which results in instability.
Revised: The Bellman error, which is used to train DQL methods, is computed with target Q -values estimated by a Deep Neural Network (DNN) with constantly changing parameters which results in instability.

Original: Target networks were proposed as a solution to stabilize training by performing an implicit regularization.
Revised: Target networks were proposed as a solution to stabilize training by performing an implicit regularization.

Original: They do so by using an additional set of parameters which can be updated in two different ways.
Revised: They do so by using an additional set of parameters which can be updated in two different ways.

Original: The ﬁrst way, which is a popular method for visual tasks such as Atari, is to periodically update the parameters to estimate the target Q -values (Mnih et al., 2013).
Revised: The ﬁrst way, which is a popular method for visual tasks such as Atari, is to periodically update the parameters to estimate the target Q -values (Mnih et al., 2013).

Original: The second way, popular in robotics and control tasks, is to parametrize the target network using a moving average of the parameters (Lillicrap et al., 2015).
Revised: The second way, popular in robotics and control tasks, is to parametrize the target network using a moving average of the parameters (Lillicrap et al., 2015).

Original: Stabilizing the squared Bellman error using target networks parametrized by an additional set of weights has become standard practice in DQL algorithms achieving state-of-the-art performance in a variety of difﬁcult tasks (Mnih et al., 2013; Lillicrap et al., 2015; Abdolmaleki et al., 2018b;
Revised: Stabilizing the squared Bellman error using target networks parametrized by an additional set of weights has become standard practice in DQL algorithms achieving state-of-the-art performance in a variety of difﬁcult tasks (Mnih et al., 2013; Lillicrap et al., 2015; Abdolmaleki et al., 2018b;

Original: Haarnoja et al., 2018a; Fujimoto et al., 2018; Hausknecht & Stone, 2015; Van Hasselt et al., 2016).
Revised: Haarnoja et al., 2018a; Fujimoto et al., 2018; Hausknecht & Stone, 2015; Van Hasselt et al., 2016).

Original: Despite the impressive performance of target networks, using additional sets of parameters to estimate the target Q -values can be problematic.
Revised: Despite the impressive performance of target networks, using additional sets of parameters to estimate the target Q -values can be problematic.

Original: First, due to lagging parameters, newly encountered rewards may not be quickly propagated through the state space, ultimately slowing down the training.
Revised: First, due to lagging parameters, newly encountered rewards may not be quickly propagated through the state space, ultimately slowing down the training.

Original: Second, while this can be partially addressed by using a moving-average (Polyak averaging) of parameters, such averaging procedure can be seen as a type of weight-regularization which too can slow down the updates.
Revised: Second, while this can be partially addressed by using a moving-average (Polyak averaging) of parameters, such averaging procedure can be seen as a type of weight-regularization which too can slow down the updates.

Original: The weight-regularization is known to be insufﬁcient for DNNs and do not necessarily improves stability (Benjamin et al., 2018).
Revised: The weight-regularization is known to be insufﬁcient for DNNs and do not necessarily improves stability (Benjamin et al., 2018).

Original: The goal of this paper is to address these issues by using up-to-date Q -values in the square Bellman error and using explicit Q -values regularization by functional regularization.
Revised: The goal of this paper is to address these issues by using up-to-date Q -values in the square Bellman error and using explicit Q -values regularization by functional regularization.

Original: We propose an alternative to the squared Bellman error based on functional regularization which, as we show, enables us to go beyond the traditional target networks.
Revised: We propose an alternative to the squared Bellman error based on functional regularization which, as we show, enables us to go beyond the traditional target networks.

Original: Our approach allows us to use the up-to-date parameters to compute the Bellman error while stabilizing the training by functionally regularizing against a periodically updated prior network (see Fig. 1).
Revised: Our approach allows us to use the up-to-date parameters to compute the Bellman error while stabilizing the training by functionally regularizing against a periodically updated prior network (see Fig. 1).

Original: Using up-to-date parameters addresses the issue of slow propagation of newly encountered rewards.
Revised: Using up-to-date parameters addresses the issue of slow propagation of newly encountered rewards.

Original: Since the regularization is explicit, it enables us to tune the regularization level for a given task, which is not possible with
Revised: Since the regularization is explicit, it enables us to tune the regularization level for a given task, which is not possible with

Original: [Table] [Table]
Revised: [Table] [Table]

Original: R (
Revised: R (

Original: [Figure]
Revised: [Figure]

Original: standard approaches that rely on implicit regularization based on target networks.
Revised: standard approaches that rely on implicit regularization based on target networks.

Original: We validate our new objective on the Four Rooms environment (Sutton et al., 1999) with DNNs function approximation and show that we can approximate the true value function and learn quickly.
Revised: We validate our new objective on the Four Rooms environment (Sutton et al., 1999) with DNNs function approximation and show that we can approximate the true value function and learn quickly.

Original: Finally, we demonstrate that DQL methods with the FR Bellman error outperform popular algorithms based on the Bellman error on a subset of the Atari suite (Bellemare et al., 2013) which is the most common benchmark for DQL methods.
Revised: Finally, we demonstrate that DQL methods with the FR Bellman error outperform popular algorithms based on the Bellman error on a subset of the Atari suite (Bellemare et al., 2013) which is the most common benchmark for DQL methods.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Preliminaries We consider the general case of a Markov Decision Process (MDP) deﬁned by {S , A , P, R, γ, µ } , where S and A respectively denote the ﬁnite state and action spaces, P : S×A×
Revised: Preliminaries We consider the general case of a Markov Decision Process (MDP) deﬁned by {S , A , P, R, γ, µ } , where S and A respectively denote the ﬁnite state and action spaces, P : S×A×

Original: S → R represent the environment transition dynamics, where P ( ·| s , a ) is the distribution of the next state taking action a in state s .
Revised: S → R represent the environment transition dynamics, where P ( ·| s , a ) is the distribution of the next state taking action a in state s .

Original: R : S×A → R denotes the reward function, γ ∈ [0 , 1) is the discount factor, and µ is the initial state distribution.
Revised: R : S×A → R denotes the reward function, γ ∈ [0 , 1) is the discount factor, and µ is the initial state distribution.

Original: We use an MDP to represent the sequential interaction between an agent and the environment, where at each round t the agent observes its current state s t , selects an action a t which results in a reward R ( s t , a t ) and the next state s t +1 ∼ P ( ·| s t , a t ) .
Revised: We use an MDP to represent the sequential interaction between an agent and the environment, where at each round t the agent observes its current state s t , selects an action a t which results in a reward R ( s t , a t ) and the next state s t +1 ∼ P ( ·| s t , a t ) .

Original: The reinforcement learning (RL) objective involves deﬁning an agent that maximizes the expected discounted sum of rewards E P,π (cid:80) t γ t R ( s t , a t ) , (with s 0 ∼ µ ), by means of a policy π ( a | s ) , that given a state selects in expectation the best action.
Revised: The reinforcement learning (RL) objective involves deﬁning an agent that maximizes the expected discounted sum of rewards E P,π (cid:80) t γ t R ( s t , a t ) , (with s 0 ∼ µ ), by means of a policy π ( a | s ) , that given a state selects in expectation the best action.

Original: Bellman optimality equations are deﬁned as:
Revised: Bellman optimality equations are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where the optimal policy is deﬁned as π ∗ ( a | s )
Revised: where the optimal policy is deﬁned as π ∗ ( a | s )

Original: =
Revised: =

Original: δ ( a − arg max a Q ∗ ( s , a )) , δ ( · ) is the Dirac delta function, and we use π ∗ to evaluate the expectations in Eqs.
Revised: δ ( a − arg max a Q ∗ ( s , a )) , δ ( · ) is the Dirac delta function, and we use π ∗ to evaluate the expectations in Eqs.

Original: (1) and (2) (see (Watkins, 1989)).
Revised: (1) and (2) (see (Watkins, 1989)).

Original: Note that the value function can be obtained as V ∗ ( s t ) = max a Q ∗ ( s t , a ) .
Revised: Note that the value function can be obtained as V ∗ ( s t ) = max a Q ∗ ( s t , a ) .

Original: Q -value Estimation We can estimate the Q -value for each state-action pair by formulating it as a regression problem, e.g., treating the right hand side of Eq.
Revised: Q -value Estimation We can estimate the Q -value for each state-action pair by formulating it as a regression problem, e.g., treating the right hand side of Eq.

Original: (1) as the target.
Revised: (1) as the target.

Original: However, we generally do not have access to the optimal policy or to samples from the optimal return.
Revised: However, we generally do not have access to the optimal policy or to samples from the optimal return.

Original: Fortunately, it is possible to learn from arbitrary policy by using the Q -learning algorithm
Revised: Fortunately, it is possible to learn from arbitrary policy by using the Q -learning algorithm

Original: (Watkins, 1989).
Revised: (Watkins, 1989).

Original: Speciﬁcally, we can minimize the one-step Mean Squared Bellman Error for a given transition ( s t , a , r t , s t +1 )
Revised: Speciﬁcally, we can minimize the one-step Mean Squared Bellman Error for a given transition ( s t , a , r t , s t +1 )

Original: (Sutton, 1988):
Revised: (Sutton, 1988):

Original: [Equation]
Revised: [Equation]

Original: where π ( a | s )
Revised: where π ( a | s )

Original: =
Revised: =

Original: δ ( a − arg max a Q ( s , a )) , and the loss is then averaged over multiple transitions.
Revised: δ ( a − arg max a Q ( s , a )) , and the loss is then averaged over multiple transitions.

Original: This approach of using future value estimates as regression targets is referred to as bootstrapping .
Revised: This approach of using future value estimates as regression targets is referred to as bootstrapping .

Original: Sometimes, the target is expressed using the Bellman operator T Q ( s t , a t ) =
Revised: Sometimes, the target is expressed using the Bellman operator T Q ( s t , a t ) =

Original: R ( s t , a t ) + γ E s t +1 , a t +1 ∼ P,π [ Q ( s t +1 , a t +1 )] .
Revised: R ( s t , a t ) + γ E s t +1 , a t +1 ∼ P,π [ Q ( s t +1 , a t +1 )] .

Original: Function Approximation
Revised: Function Approximation

Original: The original Q-learning algorithm by Watkins (1989), estimates each state-action pair separately, is a prohibitive task in real-world applications.
Revised: The original Q-learning algorithm by Watkins (1989), estimates each state-action pair separately, is a prohibitive task in real-world applications.

Original: Instead, it is possible to approximate Q by means of a parametrized function Q θ (e.g., a deep network), which can be learned by minimizing Eq.
Revised: Instead, it is possible to approximate Q by means of a parametrized function Q θ (e.g., a deep network), which can be learned by minimizing Eq.

Original: (3) with an iterative algorithm such as gradient descent.
Revised: (3) with an iterative algorithm such as gradient descent.

Original: Unfortunately, combining off-policy data, function approximation, and bootstrapping makes learning unstable and potentially diverges (Van Hasselt et al., 2018; Sutton & Barto, 2018; Achiam et al., 2019).
Revised: Unfortunately, combining off-policy data, function approximation, and bootstrapping makes learning unstable and potentially diverges (Van Hasselt et al., 2018; Sutton & Barto, 2018; Achiam et al., 2019).

Original: The problem arises when the parameters of the Q -network are updated to better approximate the Qvalue of a state-action pair at the cost of worsening the approximation of other Q -values, including the ones used as targets.
Revised: The problem arises when the parameters of the Q -network are updated to better approximate the Qvalue of a state-action pair at the cost of worsening the approximation of other Q -values, including the ones used as targets.

Original: One approach to stabilize learning involves estimating the regression targets with a target network
Revised: One approach to stabilize learning involves estimating the regression targets with a target network

Original: Q θ , a copy of Q θ where θ is periodically updated only after a certain number of iterations (Mnih et al., 2013).
Revised: Q θ , a copy of Q θ where θ is periodically updated only after a certain number of iterations (Mnih et al., 2013).

Original: This causes regression targets to no longer directly depend on the most recent estimate of θ .
Revised: This causes regression targets to no longer directly depend on the most recent estimate of θ .

Original: With this, the Mean Squared Projected Bellman Error (we will refer to this loss as simply the
Revised: With this, the Mean Squared Projected Bellman Error (we will refer to this loss as simply the

Original: Squared Bellman Error as we only use functional approximation) loss for a given state transition is:
Revised: Squared Bellman Error as we only use functional approximation) loss for a given state transition is:

Original: [Equation]
Revised: [Equation]

Original: A common strategy to stabilize learning is to update the target parameters only after a ﬁxed number of training iterations i.e., periodic update (Mnih et al., 2013).
Revised: A common strategy to stabilize learning is to update the target parameters only after a ﬁxed number of training iterations i.e., periodic update (Mnih et al., 2013).

Original: In between these updates, the target is using the lagging parameters, and the newly encountered rewards are not immediately reﬂected in the estimated target Q -values.
Revised: In between these updates, the target is using the lagging parameters, and the newly encountered rewards are not immediately reﬂected in the estimated target Q -values.

Original: The delay in the reward propagation can, however, slow down training.
Revised: The delay in the reward propagation can, however, slow down training.

Original: For instance, consider a Markov chain environment of N states with deterministic transitions, in which all states have reward of 0 except the terminal, right-most state which has a reward of 1 .
Revised: For instance, consider a Markov chain environment of N states with deterministic transitions, in which all states have reward of 0 except the terminal, right-most state which has a reward of 1 .

Original: Consider using tabular Q -values and the corresponding target Q -values, which are updated every H training steps.
Revised: Consider using tabular Q -values and the corresponding target Q -values, which are updated every H training steps.

Original: Even with access to all the state-action pairs, this scheme will require NH training steps to propagate the right-most reward through the chain, i.e., to the state N steps away.
Revised: Even with access to all the state-action pairs, this scheme will require NH training steps to propagate the right-most reward through the chain, i.e., to the state N steps away.

Original: We can observe that the learning speed would be a direct function of H .
Revised: We can observe that the learning speed would be a direct function of H .

Original: An alternative to hard-updating is the use of a moving average of the weights to parametrize the target Q -network as done with Polyak’s averaging (”the soft update”)
Revised: An alternative to hard-updating is the use of a moving average of the weights to parametrize the target Q -network as done with Polyak’s averaging (”the soft update”)

Original: (Lillicrap et al., 2015).
Revised: (Lillicrap et al., 2015).

Original: In this case, the target network’s weights are updated as θ ← (1 − τ ) θ
Revised: In this case, the target network’s weights are updated as θ ← (1 − τ ) θ

Original: + τ θ , with τ ∈ (0 , 1) .
Revised: + τ θ , with τ ∈ (0 , 1) .

Original: However, θ is not used to estimate the target Q -values, thus, our regression targets are no longer up-to-date and learning is slowed.
Revised: However, θ is not used to estimate the target Q -values, thus, our regression targets are no longer up-to-date and learning is slowed.

Original: Weight Regularization in DQN Soft updates can be seen as a simple weight regularization where the update of ¯ θ ← θ is delayed by adding an L 2 regularizer τ (cid:107) θ − ¯ θ (cid:107) 2 .
Revised: Weight Regularization in DQN Soft updates can be seen as a simple weight regularization where the update of ¯ θ ← θ is delayed by adding an L 2 regularizer τ (cid:107) θ − ¯ θ (cid:107) 2 .

Original: Essentially, we do not want the value of ¯ θ to abruptly change.
Revised: Essentially, we do not want the value of ¯ θ to abruptly change.

Original: This approach to improving stability is common in nonstationary time-series analysis (Brown, 1959; Holt et al., 1960; Gardner Jr, 1985), online learning (Cesa-Bianchi & Lugosi, 2006), and continual learning (Kirkpatrick et al., 2017; Nguyen et al., 2017).
Revised: This approach to improving stability is common in nonstationary time-series analysis (Brown, 1959; Holt et al., 1960; Gardner Jr, 1985), online learning (Cesa-Bianchi & Lugosi, 2006), and continual learning (Kirkpatrick et al., 2017; Nguyen et al., 2017).

Original: Recent attempts to go beyond target networks employ preconditioning methods which can also be seen as weight-regularization methods (Knight & Lerner, 2018; Achiam et al., 2019).
Revised: Recent attempts to go beyond target networks employ preconditioning methods which can also be seen as weight-regularization methods (Knight & Lerner, 2018; Achiam et al., 2019).

Original: For example, natural-gradient descent used in Knight & Lerner (2018) employs the Kullback-Leibler (KL) for Gaussian distributions N ( q | Q θ ( s, a ) , 1) over the Q-values q , giving rise to a quadratic weightregularizer.
Revised: For example, natural-gradient descent used in Knight & Lerner (2018) employs the Kullback-Leibler (KL) for Gaussian distributions N ( q | Q θ ( s, a ) , 1) over the Q-values q , giving rise to a quadratic weightregularizer.

Original: Another approach of Achiam et al.
Revised: Another approach of Achiam et al.

Original: (2019) uses a different preconditioner which can also be seen as a quadratic weight-regularization.
Revised: (2019) uses a different preconditioner which can also be seen as a quadratic weight-regularization.

Original: The computation in these approaches is challenging and so far have only been applied to small networks.
Revised: The computation in these approaches is challenging and so far have only been applied to small networks.

Original: A more serious problem with weight regularization is its ineffectiveness in DNNs.
Revised: A more serious problem with weight regularization is its ineffectiveness in DNNs.

Original: For neural networks, the network outputs depend on the weights in a complex way and the exact value of the weights may not matter much.
Revised: For neural networks, the network outputs depend on the weights in a complex way and the exact value of the weights may not matter much.

Original: What ultimately matters is the network outputs (Benjamin et al., 2018), and it is better to directly regularize those.
Revised: What ultimately matters is the network outputs (Benjamin et al., 2018), and it is better to directly regularize those.

Original: Deep Q -learning algorithms
Revised: Deep Q -learning algorithms

Original: Deep Q -network ( DQN ) (Mnih et al., 2013) has been the ﬁrst DQL algorithm to solve Atari games from images.
Revised: Deep Q -network ( DQN ) (Mnih et al., 2013) has been the ﬁrst DQL algorithm to solve Atari games from images.

Original: To learn the Q -values, DQN minimizes the squared
Revised: To learn the Q -values, DQN minimizes the squared

Original: Bellman error where the target Q -values are estimated by a target network with a periodically updated set of parameters.
Revised: Bellman error where the target Q -values are estimated by a target network with a periodically updated set of parameters.

Original: Polyak DQN is a DQL algorithm that also minimizes the squared Bellman error but uses a target network using a moving average of the parameters.
Revised: Polyak DQN is a DQL algorithm that also minimizes the squared Bellman error but uses a target network using a moving average of the parameters.

Original: Double DQN decouples the action selection from the action evaluation by using different network to do both and to provide a target during training and counter the maximum bias.
Revised: Double DQN decouples the action selection from the action evaluation by using different network to do both and to provide a target during training and counter the maximum bias.

Original: It has been shown an effective way to stabilize
Revised: It has been shown an effective way to stabilize

Original: DQN and prevent (soft-)divergence (Van Hasselt et al., 2016).
Revised: DQN and prevent (soft-)divergence (Van Hasselt et al., 2016).

Original:  D EEP Q - LEARNING WITH F UNCTIONAL R EGULARIZATION
Revised:  D EEP Q - LEARNING WITH F UNCTIONAL R EGULARIZATION

Original: In practice, deep Q -learning methods are known to be unstable,  not approximate the true value function, and  sometimes even (soft-)diverge (Van Hasselt, 2010; Van Hasselt et al., 2016; 2018).
Revised: In practice, deep Q -learning methods are known to be unstable, to not approximate the true value function, and to sometimes even (soft-)diverge (Van Hasselt, 2010; Van Hasselt et al., 2016; 2018).

Original: A multitude of solutions have been proposed to stabilize training, but in this work we will mainly focus on the stabilizing role played by target networks.
Revised: A multitude of solutions have been proposed to stabilize training, but in this work we will mainly focus on the stabilizing role played by target networks.

Original: Target networks are parametrized by an additional set of lagging parameters.
Revised: Target networks are parametrized by an additional set of lagging parameters.

Original: Intuitively, since the lagging parameters are used to estimate the target Q -value, they can be interpreted as a form of implicit prior , preventing the Q -value estimates from changing too quickly.
Revised: Intuitively, since the lagging parameters are used to estimate the target Q -value, they can be interpreted as a form of implicit prior , preventing the Q -value estimates from changing too quickly.

Original: In this section, we use this insight to introduce the functional regularized squared Bellman error, a regularized loss to stabilize training while using up-to-date target values.
Revised: In this section, we use this insight to introduce the functional regularized squared Bellman error, a regularized loss to stabilize training while using up-to-date target values.

Original: In this process, we help formalize the notion of target networks as forms of functional priors.
Revised: In this process, we help formalize the notion of target networks as forms of functional priors.

Original: 3. F UNCTIONALLY R EGULARIZED S QUARED B ELLMAN E RROR
Revised: 3. F UNCTIONALLY R EGULARIZED S QUARED B ELLMAN E RROR

Original: Target networks have coupled roles: providing target Q -values used for learning, and stabilizing training.
Revised: Target networks have coupled roles: providing target Q -values used for learning, and stabilizing training.

Original: We can decouple these two roles by using up-to-date parameters to estimate the target
Revised: We can decouple these two roles by using up-to-date parameters to estimate the target

Original: Q -values, and using a functional prior to regularize the Q -values.
Revised: Q -values, and using a functional prior to regularize the Q -values.

Original: Following Benjamin et al.
Revised: Following Benjamin et al.

Original: , we proposed using an L 2 functional regularizer (FR) which penalizes the divergence between the current estimate of Q-value function Q θ ( s , a ) and a lagging version of it Q
Revised: , we proposed using an L 2 functional regularizer (FR) which penalizes the divergence between the current estimate of Q-value function Q θ ( s , a ) and a lagging version of it Q

Original: θ ( s , a ) .
Revised: θ ( s , a ) .

Original: This gives us the following loss function:
Revised: This gives us the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where the up-to-date parameter θ is used in the Squared Bellman Error along with a FR loss to stabilize θ by regularizing the current state-action pair only.
Revised: where the up-to-date parameter θ is used in the Squared Bellman Error along with a FR loss to stabilize θ by regularizing the current state-action pair only.

Original: The expectation above is taken over the transition and policy, and κ > 0 is the regularization parameter.
Revised: The expectation above is taken over the transition and policy, and κ > 0 is the regularization parameter.

Original: By stop grad , we indicate a function that prevents gradients from propagating into the target Q -value estimate.
Revised: By stop grad , we indicate a function that prevents gradients from propagating into the target Q -value estimate.

Original: Critically, unlike Eq.
Revised: Critically, unlike Eq.

Original: (4), the target Q -value estimates are supplied by the up-to-date Q -network, with the functional prior now separately serving to stabilize the estimate.
Revised: (4), the target Q -value estimates are supplied by the up-to-date Q -network, with the functional prior now separately serving to stabilize the estimate.

Original: This decoupling allows us to quickly propagate newly encountered rewards and maintain stability.
Revised: This decoupling allows us to quickly propagate newly encountered rewards and maintain stability.

Original: As with a target network, we can update the functional prior periodically to control the stability of the Q -value estimates.
Revised: As with a target network, we can update the functional prior periodically to control the stability of the Q -value estimates.

Original: Overall, l FR is arguably similar in complexity to l target , requiring only an additional function evaluation for Q θ ( s t +1 , a t +1 ) and an additional hyper-parameter, κ , which was not difﬁcult to tune in all of our experiments (see Section 4).
Revised: Overall, l FR is arguably similar in complexity to l target , requiring only an additional function evaluation for Q θ ( s t +1 , a t +1 ) and an additional hyper-parameter, κ , which was not difﬁcult to tune in all of our experiments (see Section 4).

Original: As will be discussed in the subsequent sectionSection 3.2, target networks implicitly make a ﬁxed trade-off between speed and stability, whereas κ in the FR squared Bellman error provides additional ﬂexibility and can be task dependent.
Revised: As will be discussed in the subsequent Section 3.2, target networks implicitly make a ﬁxed trade-off between speed and stability, whereas κ in the FR squared Bellman error provides additional ﬂexibility and can be task dependent.

Original: Functional regularization  canalso be seen as a Kullback Leibler (KL) divergence between two Gaussian processes whose mean functions are the Q -values at different state-action pairs and whose covariance function is the identity.
Revised: The functional regularization between Q θ and Q ¯ θ can be seen as an approximation to the KullbackLeibler (KL) divergence between two Gaussian processes where we assume identity covariance for these two processes.

Revised: Since the covariance matrices might not be the identity in practice, further improvement might be possible by estimating the covariances at the cost of additional computation, e.g. (Pan et al., 2020).

Original: This interpretation draws a link with other popular regularization methods in policy optimization inspired by trust-region algorithms (Schulman et al., 2015a; Abdolmaleki et al., 2018b;a) where the quantity of interest, the policy, is KL regularized towards its past values.
Revised: This interpretation draws a link with other popular regularization methods in policy optimization inspired by trust-region algorithms (Schulman et al., 2015a; Abdolmaleki et al., 2018b;a) where the quantity of interest, the policy, is KL regularized towards its past values.

Original: In the same spirit, for value-based methods, the FR Squared Bellman Error regularizes the Q function estimates towards their past values.
Revised: In the same spirit, for value-based methods, the FR Squared Bellman Error regularizes the Q function estimates towards their past values.

Original: 3. I NTERPRETING T ARGET N ETWORKS THROUGH F UNCTIONAL R EGULARIZATION
Revised: 3. I NTERPRETING T ARGET N ETWORKS THROUGH F UNCTIONAL R EGULARIZATION

Original: Given the similar regularization role accomplished by target networks and functional priors, it is reasonable to assume that target networks can be interpreted through the lens of FR.
Revised: Given the similar regularization role accomplished by target networks and functional priors, it is reasonable to assume that target networks can be interpreted through the lens of FR.

Original: To validate this assumption, we compared the gradients, ∇ θ l ( θ ) , arising from each case (Eq.
Revised: To validate this assumption, we compared the gradients, ∇ θ l ( θ ) , arising from each case (Eq.

Original: (4) & Eq.
Revised: (4) & Eq.

Original: (5)) which allowed us to see how these learning procedures differed.
Revised: (5)) which allowed us to see how these learning procedures differed.

Original: For a complete derivation, see
Revised: For a complete derivation, see

Original: Appendix A. To simplify notation, we deﬁne the Q -value estimate at time t as Q ( t ) θ ≡ Q θ ( s t , a t ) , the corresponding expected change in the Q -value estimate as ∆ Q ( t ) θ ≡ E P,π [ Q ( t +1) θ | s t , a t ] − Q ( t ) θ , and the reward as R ( t ) ≡ R ( s t , a t ) .
Revised: Appendix A. To simplify notation, we deﬁne the Q -value estimate at time t as Q ( t ) θ ≡ Q θ ( s t , a t ) , the corresponding expected change in the Q -value estimate as ∆ Q ( t ) θ ≡ E P,π [ Q ( t +1) θ | s t , a t ] − Q ( t ) θ , and the reward as R ( t ) ≡ R ( s t , a t ) .

Original: Then, for a given state-action pair, we can compare the two gradients:
Revised: Then, for a given state-action pair, we can compare the two gradients:

Original: [Equation]
Revised: [Equation]

Original: Replacing ∆ Q ( t ) θ with ∆ Q ( t ) θ and κ with γ we recover the gradient of l target ( θ ) :
Revised: Replacing ∆ Q ( t ) θ with ∆ Q ( t ) θ and κ with γ we recover the gradient of l target ( θ ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By comparing the gradients, we see that, despite the differences in their loss formulations, target networks and FR result in nearly identical gradients, with two key exceptions.
Revised: By comparing the gradients, we see that, despite the differences in their loss formulations, target networks and FR result in nearly identical gradients, with two key exceptions.

Original: The ﬁrst distinction, underlined in orange, is the estimated change in the Q -value.
Revised: The ﬁrst distinction, underlined in orange, is the estimated change in the Q -value.

Original: With target networks, this term is supplied by the lagging network parameters, θ , and is therefore not up-to-date, whereas with
Revised: With target networks, this term is supplied by the lagging network parameters, θ , and is therefore not up-to-date, whereas with

Original: FR, it is supplied by the up-to-date parameters.
Revised: FR, it is supplied by the up-to-date parameters.

Original: Thus, illustrating why FR can propagate updated value estimates faster.
Revised: Thus, illustrating why FR can propagate updated value estimates faster.

Original: The second distinction, underlined in blue, is the weighting on the difference between the lagging and up-to-date Q -value estimates.
Revised: The second distinction, underlined in blue, is the weighting on the difference between the lagging and up-to-date Q -value estimates.

Original: With target networks, this is the discount factor, γ , whereas with FR, this is a separate hyper-parameter.
Revised: With target networks, this is the discount factor, γ , whereas with FR, this is a separate hyper-parameter.

Original: γ is usually part of the problem deﬁnition and thus cannot be tuned for a speciﬁc task.
Revised: γ is usually part of the problem deﬁnition and thus cannot be tuned for a speciﬁc task.

Original: Using the gradient in Eq.
Revised: Using the gradient in Eq.

Original: (7), and deﬁning ˆ Q ( t ) θ , θ ≡ Q ( t ) θ + ∆ Q ( t ) θ , we can derive a loss function, (cid:101) l target , that yields equivalent gradients as l target , but is written in the FR form:
Revised: (7), and deﬁning ˆ Q ( t ) θ , θ ≡ Q ( t ) θ + ∆ Q ( t ) θ , we can derive a loss function, (cid:101) l target , that yields equivalent gradients as l target , but is written in the FR form:

Original: [Equation]
Revised: [Equation]

Original: From this perspective, target networks effectively perform a special, restricted form of Gaussian FR, with p ( Q θ ) =
Revised: From this perspective, target networks effectively perform a special, restricted form of Gaussian FR, with p ( Q θ ) =

Original: N ( Q θ , γ − 1 ) , i.e., a precision weight that is not under direct control.
Revised: N ( Q θ , γ − 1 ) , i.e., a precision weight that is not under direct control.

Original: In contrast, l FR allows us to separately adjust this weight to trade-off between stability and learning speed.
Revised: In contrast, l FR allows us to separately adjust this weight to trade-off between stability and learning speed.

Original: Overall, this derivation illustrates the similarities between the squared Bellman error with target networks and the FR squared Bellman error.
Revised: Overall, this derivation illustrates the similarities between the squared Bellman error with target networks and the FR squared Bellman error.

Original: It also highlights the advantages of using the FR squared Bellman error over square Bellman error combined with target networks: up-to-date target Q -value estimates and control over the regularization through κ .
Revised: It also highlights the advantages of using the FR squared Bellman error over square Bellman error combined with target networks: up-to-date target Q -value estimates and control over the regularization through κ .

Original: 3. C OMPARISON WITH P OLYAK UPDATING
Revised: 3. C OMPARISON WITH P OLYAK UPDATING

Original: Polyak’s updating is a common technique used to trade-off between speed and stability for control problems.
Revised: Polyak’s updating is a common technique used to trade-off between speed and stability for control problems.

Original: Interestingly, if we assume the target network is an estimate in weight space of the latest Q -network, we can show that Polyak’s updates implicitly performs parameter regularization on the weight estimate.
Revised: Interestingly, if we assume the target network is an estimate in weight space of the latest Q -network, we can show that Polyak’s updates implicitly performs parameter regularization on the weight estimate.

Original: To better understand the implicit regularization, let us consider a situation in which after each gradient step (indexed by i ), we identiﬁed the target network as the solution to the problem:
Revised: To better understand the implicit regularization, let us consider a situation in which after each gradient step (indexed by i ), we identiﬁed the target network as the solution to the problem:

Original: [Equation]
Revised: [Equation]

Original: (8) whereby θ i is the latest set of weights of the Q -network and θ i is the previous instance of the target network, and τ is the exponential averaging step size.
Revised: (8) whereby θ i is the latest set of weights of the Q -network and θ i is the previous instance of the target network, and τ is the exponential averaging step size.

Original: Minimizing θ is obtained by computing the gradient of the problem and equating it 0 , leading to θ
Revised: Minimizing θ is obtained by computing the gradient of the problem and equating it 0 , leading to θ

Original: i +1 =
Revised: i +1 =

Original: (1 − τ ) θ i
Revised: (1 − τ ) θ i

Original: + τ θ , which is exactly Polyak’s updating.
Revised: + τ θ , which is exactly Polyak’s updating.

Original: This derivation illustrates that Polyak’s update is indeed a form of weight regularization
Revised: This derivation illustrates that Polyak’s update is indeed a form of weight regularization

Original: [Figure]
Revised: [Figure]

Original: with respect to the most recent target network weights.
Revised: with respect to the most recent target network weights.

Original: Weight regularization does not, however, guarantee that the output of the regularized network matches the previous target network.
Revised: Weight regularization does not, however, guarantee that the output of the regularized network matches the previous target network.

Original: In fact, while this technique has found success in control problems (Lillicrap et al., 2015; Haarnoja et al., 2018a; Fujimoto et al., 2018), periodically updating the parameters is usually preferred for complex
Revised: In fact, while this technique has found success in control problems (Lillicrap et al., 2015; Haarnoja et al., 2018a; Fujimoto et al., 2018), periodically updating the parameters is usually preferred for complex

Original: DNN architectures (Mnih et al., 2013; Hausknecht & Stone, 2015; Hessel et al., 2018; Kapturowski et al., 2018; Parisotto et al., 2020).
Revised: DNN architectures (Mnih et al., 2013; Hausknecht & Stone, 2015; Hessel et al., 2018; Kapturowski et al., 2018; Parisotto et al., 2020).

Original:  R ESULTS
Revised:  R ESULTS

Original: We compared the performance of our objective (FR-Squared Bellman Error) and other algorithms minimizing the squared Bellman error and using target networks in the Four Rooms environment and the subset of Atari games (Bellemare et al., 2013) used by Mnih et al.
Revised: We compared the performance of our objective (FR-Squared Bellman Error) and other algorithms minimizing the squared Bellman error and using target networks in the Four Rooms environment and the subset of Atari games (Bellemare et al., 2013) used by Mnih et al.

Original: The experimental code is based on DQN Zoo (Quan & Ostrovski, 2020), implemented in Jax (Bradbury et al., 2018) and
Revised: The experimental code is based on DQN Zoo (Quan & Ostrovski, 2020), implemented in Jax (Bradbury et al., 2018) and

Original: Haiku (Hennigan et al., 2020).
Revised: Haiku (Hennigan et al., 2020).

Original: Additional experimental details can be found in the Appendix.
Revised: Additional experimental details can be found in the Appendix.

Original: 4. F OUR R OOMS 4.1. E XPERIMENTAL S ET -U P
Revised: 4. F OUR R OOMS 4.1. E XPERIMENTAL S ET -U P

Original: In order to better understand the performance of the FR squared Bellman error, we compared our objective to a suite of algorithms and hyper-parameters minimizing the squared Bellman error and using a target network in the Four Rooms environment (Sutton et al., 1999) with neural approximation.
Revised: In order to better understand the performance of the FR squared Bellman error, we compared our objective to a suite of algorithms and hyper-parameters minimizing the squared Bellman error and using a target network in the Four Rooms environment (Sutton et al., 1999) with neural approximation.

Original: Four Rooms is a grid world environment comprised of four rooms divided by walls with a single gap in each wall allowing an agent to move between room-.
Revised: Four Rooms is a grid world environment comprised of four rooms divided by walls with a single gap in each wall allowing an agent to move between rooms.

Original: In this environment, the agent’s goal is to reach a square in the upper-left room near the right door, and the agent’s initial position is drawn randomly from all other cells excluding the walls position.
Revised: In this environment, the agent’s must reach a goal state in the upper-left room near the right door, and the agent’s initial position is drawn randomly from all other cells excluding the walls position.

Original: The agent’s position is given through a one-hot encoding.
Revised: The agent’s position is given through a one-hot encoding.

Original: The agent is capable of taking actions in four directions: up, down, left and right.
Revised: The agent is capable of taking actions in four directions: up, down, left and right.

Original: See Fig.
Revised: See Fig.

Original: 6 in the Appendix for a graphical representation of the environment and the optimal discounted return.
Revised: 12 in the Appendix for a graphical representation of the environment and the optimal discounted return.

Original: The advantage of using this environment over more complex ones such as Atari games, is that the underlying transition matrix is available, and the true optimal value function can be computed using tabular methods (which will be referred to as V ∗ ).
Revised: The advantage of using this environment over more complex ones such as Atari games, is that the underlying transition matrix is available, and the true optimal value function can be computed using tabular methods (which will be referred to as V ∗ ).

Original: We can then compare the agent’s parametrized value estimate V θ ( s ) = max a Q θ ( s , a ) with the optimal one V ∗ ( s ) to assess the accuracy of the agent’s Q -values.
Revised: We can then compare the agent’s parametrized value estimate V θ ( s ) = max a Q θ ( s , a ) with the optimal one V ∗ ( s ) to assess the accuracy of the agent’s Q -values.

Original: The discount factor is 0.99, the reward is always 0 except at the goal state where every action leads to a reward of 1.
Revised: The discount factor is 0.99, the reward is always 0 except at the goal state where every action leads to a reward of 1.

Original: Speciﬁcally all actions from the goal state receive a reward of
Revised: Speciﬁcally all actions from the goal state receive a reward of

Original: [Figure]
Revised: [Figure]

Original: The episode terminates either once the agent reach the goal or after 500 steps.
Revised: The episode terminates either once the agent reach the goal or after 500 steps.

Original: The environment is implemented using Easy MDP (Ahmed, 2020).
Revised: The environment is implemented using Easy MDP (Ahmed, 2020).

Original: The Q -value is parametrized by a Multi-Layers
Revised: The Q -value is parametrized by a Multi-Layers

Original: Perceptron (MLP) with 2 layers of 128 hidden units each.
Revised: Perceptron (MLP) with 2 layers of 128 hidden units each.

Original: The shared hyper-parameters (learning rate and batch size) were tuned on 5 seeds for Deep Q -Network (DQN) (Mnih et al., 2013) to maximize success rate, and then use for all the other methods.
Revised: The shared hyper-parameters (learning rate and batch size) were tuned on 5 seeds for Deep Q -Network (DQN) (Mnih et al., 2013) to maximize success rate, and then use for all the other methods.

Original: The ﬁnal performances are reported on 20 different seeds after 100k gradient steps (and as many transitions).
Revised: The ﬁnal performances are reported on 20 different seeds after 100k gradient steps (and as many transitions).

Original: All hyper parameters are available in Table 1.
Revised: All hyper parameters are available in Table 1.

Original:  R ESULTS
Revised: 4.1. R ESULTS

Original: We compared the performance of the FR square Bellman error combined with the DQN algorithm (FR DQN) to DQN and Polyak DQN.
Revised: We compared the performance of the FR square Bellman error combined with the DQN algorithm (FR DQN) to DQN, Polyak DQN, and Deep Mellow (Kim et al., 2019).

Original: As observed in Fig.
Revised: As observed in Fig.

Original: 2 (a), the target network update period greatly affects performance of algorithms minimizing the squared Bellman error.
Revised: 2 (a), the target network update period greatly affects performance of algorithms minimizing the squared Bellman error.

Original: On the one hand, frequently updating the target network leads to faster initial performance since the reward is propagated through the space faster.
Revised: On the one hand, frequently updating the target network leads to faster initial performance since the reward is propagated through the space faster.

Original: On the other hand, updating the network too frequently can lead to instability as observed in Fig.
Revised: On the other hand, updating the network too frequently can lead to instability as observed in Fig.

Original: For example, τ of 0.05 initially outperforms τ of 0.01, but becomes unstable, and never reaches optimal performance.
Revised: For example, τ of 0.05 initially outperforms τ of 0.01, but becomes unstable, and never reaches optimal performance.

Original: The FR squared Bellman error combined with the DQN algorithm (FR DQN) bypasses this issue, by using the previous Q -values estimates exclusively for regularization which allows it to use the most up-to-date estimate for learning.
Revised: The FR squared Bellman error combined with the DQN algorithm (FR DQN) bypasses this issue, by using the previous Q -values estimates exclusively for regularization which allows it to use the most up-to-date estimate for learning.

Original: Additionally, Q -learning methods should converge to the optimal value function V ∗ , but in complex settings like Atari games, it is impossible to compute V ∗ and monitor the distance to the optimum.
Revised: Additionally, Q -learning methods should converge to the optimal value function V ∗ , but in complex settings like Atari games, it is impossible to compute V ∗ and monitor the distance to the optimum.

Original: In the Four Rooms setting, however, measuring the accuracy of their approximation is tractable.
Revised: In the Four Rooms setting, however, measuring the accuracy of their approximation is tractable.

Original: Interestingly, even if the Bellman error loss is roughly the same for FR DQN and Polyak DQN (Fig.
Revised: Interestingly, even if the Bellman error loss is roughly the same for FR DQN and Polyak DQN (Fig.

Original: 2 (b)), FR DQN approximates the true value function with much greater accuracy than Polyak
Revised: 2 (b)), FR DQN approximates the true value function with much greater accuracy than Polyak

Original: DQN (Fig. 2 (c)).
Revised: DQN (Fig. 2 (c)).

Revised: We also note that Deep Mellow does not recover the true value function.

Revised: This failure is not surprising as Deep Mellow is biased by using the mellowmax operator instead of the max .

Original: This is of particular interest since the squared Bellman error is the metric that is usually monitored in practice and might not be an accurate measure of the accuracy of the Q -value.
Revised: This is of particular interest since the squared Bellman error is the metric that is usually monitored in practice and might not be an accurate measure of the accuracy of the Q -value.

Original: 4. A TARI 4.2. E XPERIMENTAL S ET -U P
Revised: 4. A TARI 4.2. E XPERIMENTAL S ET -U P

Original: The Arcade Learning Environment (Bellemare et al., 2013) provides a set of benchmark Atari games for evaluating Reinforcement Learning algorithms.
Revised: The Arcade Learning Environment (Bellemare et al., 2013) provides a set of benchmark Atari games for evaluating Reinforcement Learning algorithms.

Original: These games represent a good benchmark as  they possess rich visual observations and diverse reward functions.
Revised: These games represent a good benchmark as [Figure] they possess rich visual observations and diverse reward functions.

Original: Complex DQL methods currently hold state-of-the-art results in this benchmark (Hessel et al., 2018).
Revised: Complex DQL methods currently hold state-of-the-art results in this benchmark (Hessel et al., 2018).

Original: As previously mentioned, complex DQL methods show state-of-the-art results in this benchmark task.

Original: But in order to un- [Figure] derstand the effect of replacing the squared Bellman error with the FR squared Bellman error, we ﬁrst compared our approach with the standard DQN (Mnih et al., 2013) and Polyak DQN.
Revised: But in order to understand the effect of replacing the squared Bellman error with the FR squared Bellman error, we ﬁrst compared our approach with the standard DQN (Mnih et al., 2013) and Polyak DQN.

Original: Then we showed that we can easily combine the FR squared Bellman error objective with orthogonal improvements such as Double DQN (Van Hasselt et al., 2016).
Revised: Then we showed that we can easily combine the FR squared Bellman error objective with orthogonal improvements such as Double DQN (Van Hasselt et al., 2016).

Original: We stress that the algorithms based on the FR squared Bellman error, e.g., FR DQN and FR Double
Revised: We stress that the algorithms based on the FR squared Bellman error, e.g., FR DQN and FR Double

Original: DQN, are simple modiﬁcations from the DQN Zoo library (Quan & Ostrovski, 2020).
Revised: DQN, are simple modiﬁcations from the DQN Zoo library (Quan & Ostrovski, 2020).

Original: We tuned the hyper-parameter for each method (except DQN as they have already been tuned by the authors of the
Revised: We tuned the hyper-parameter for each method (except DQN as they have already been tuned by the authors of the

Original: DQN Zoo package) by performing experiments on 2 seeds (only 1 set of global hyper-parameters used for every tasks).
Revised: DQN Zoo package) by performing experiments on 2 seeds (only 1 set of global hyper-parameters used for every tasks).

Original: Once the best set of hyper-parameters for each method have been selected, we used 10 additional seeds (and exclude the 2 initial seeds) to compare the algorithms.
Revised: Once the best set of hyper-parameters for each method have been selected, we used 10 additional seeds (and exclude the 2 initial seeds) to compare the algorithms.

Original: All hyperparameters we used are displayed in Table 2 in the appendix.
Revised: All hyperparameters we used are displayed in Table 2 in the appendix.

Original: We used the same suite of representative environments from Mnih et al.
Revised: We used the same suite of representative environments from Mnih et al.

Original: Each seed took 48 hours to run on our resources for a total of 14400 hours of GPU usage.
Revised: Each seed took 48 hours to run on our resources for a total of 14400 hours of GPU usage.

Original: 4.2. R ESULTS
Revised: 4.2. R ESULTS

Original: Comparison with DQN and Polyak DQN The comparison between FR DQN,DQN and Polyak
Revised: Comparison with DQN, Polyak DQN, and Deep Mellow The comparison between FR DQN,

Original: DQN  on Atari performance results are presented in Fig.
Revised: DQN and Polyak DQN on Atari performance results are presented in Fig.

Original: Across all environments, FR DQN matches or exceeds the performance of DQN and Polyak DQN in terms of sample efﬁciency and ﬁnal performance at 100 M steps.
Revised: Across all environments, FR DQN matches or exceeds the performance of DQN and Polyak DQN in terms of sample efﬁciency and ﬁnal performance at 100 M steps.

Original: FR DQN’s ability to use up-to-date target Q -values also plays a signiﬁcant role in improving performance over Target Networks.
Revised: FR DQN’s ability to use up-to-date target Q -values also plays a signiﬁcant role in improving performance over Target Networks.

Original: Polyak DQN is quite unstable and does not constantly match the performance of DQN.
Revised: Polyak DQN and Deep Mellow are quite unstable and do not constantly match the performance of DQN.

Revised: We hypothesise that the temperature of Deep Mellow has to be ﬁnely tune to each games in order to outperform

Original:  Comparison with Double DQN Since the introduction of Target Networks, there has been numerous techniques to stabilize the training and improve the performance of DQL methods (Hessel et al., 2018).
Revised: DQN as reported in Kim et al. Comparison with Double DQN Since the introduction of Target Networks, there has been numerous techniques to stabilize the training and improve the performance of DQL methods (Hessel et al., 2018).

Original: While it is outside of the scope of this paper to combine FR DQN with each and every DQN improvements, we show that it is fairly easy to combine the FR squared Bellman error with double DQN (Van Hasselt et al., 2016).
Revised: While it is outside of the scope of this paper to combine FR DQN with each and every DQN improvements, we show that it is fairly easy to combine the FR squared Bellman error with double DQN (Van Hasselt et al., 2016).

Original: The results presented in Fig.
Revised: The results presented in Fig.

Original: 4 clearly demonstrates that the FR squared Bellman error can be combined with DDQN to match or achieves even higher performances on this subset of Atari games with the exception of Seaquest.
Revised: 4 clearly demonstrates that the FR squared Bellman error can be combined with DDQN to match or achieves even higher performances on this subset of Atari games with the exception of Seaquest.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Multiple previous works have investigated how to improve value estimation through various constraints and regularization.
Revised: Multiple previous works have investigated how to improve value estimation through various constraints and regularization.

Original: Farahmand et al.
Revised: Farahmand et al.

Original: (2009) is perhaps the closest work to our own, functionally regularizing the L 2 norm of the Q -value estimate, i.e., penalizing κ || Q θ || 2 .
Revised: (2009) is perhaps the closest work to our own, functionally regularizing the L 2 norm of the Q -value estimate, i.e., penalizing κ || Q θ || 2 .

Original: Farahmand et al.
Revised: Farahmand et al.

Original: (2009)’s approach can be interpreted as using a ﬁxed Gaussian prior, whereas we used a periodically updated set of parameters to provide a moving prior.
Revised: (2009)’s approach can be interpreted as using a ﬁxed Gaussian prior, whereas we used a periodically updated set of parameters to provide a moving prior.

Original: However, penalising the magnitude of the Qvalues would not prevent the algorithm to converge to V ∗ if κ does not tend towards 0.
Revised: However, penalising the magnitude of the Qvalues would not prevent the algorithm to converge to V ∗ if κ does not tend towards 0.

Original: Shao et al.
Revised: Shao et al.

Original: (2020) proposed adding an additional backward Squared Bellman Error loss to the next Q -value to stabilize training and remove the need for a target network.
Revised: (2020) proposed adding an additional backward Squared Bellman Error loss to the next Q -value to stabilize training and remove the need for a target network.

Original: Other works have sought to regularize the Q -value estimator by constraining parameter updates, e.g., through regularization (Farebrother et al., 2018), conjugate gradient methods (Schulman et al., 2015b), pre-conditioning the gradient updates (Knight & Lerner, 2018; Achiam et al., 2019), or using Kalman ﬁltering (Shashua & Mannor, 2019; 2020).
Revised: Other works have sought to regularize the Q -value estimator by constraining parameter updates, e.g., through regularization (Farebrother et al., 2018), conjugate gradient methods (Schulman et al., 2015b), pre-conditioning the gradient updates (Knight & Lerner, 2018; Achiam et al., 2019), or using Kalman ﬁltering (Shashua & Mannor, 2019; 2020).

Original: As argued in Sections 2 and 3, functional regularization, as opposed to parameter regularization, is more appropriate in combination with DNNs, because parameter regularization does not necessarily imply that the Q -value estimates remain stable during training.
Revised: As argued in Sections 2 and 3, functional regularization, as opposed to parameter regularization, is more appropriate in combination with DNNs, because parameter regularization does not necessarily imply that the Q -value estimates remain stable during training.

Original: Other previous works have sought to address orthogonal issues with DQL algorithms, and, in principle, these could be combined with the FR squared Bellman error.
Revised: Other previous works have sought to address orthogonal issues with DQL algorithms, and, in principle, these could be combined with the FR squared Bellman error.

Original: For instance, Kim et al.
Revised: For instance, Kim et al.

Original: (2019) removed the need for target network on Atari games by using the mellow max operator as an alternative to the max operator used in bootstrapping.
Revised: (2019) removed the need for target network on Atari games by using the mellow max operator as an alternative to the max operator used in bootstrapping.

Original: This new operator is used with the squared Bellman error and could easily be used in combination with FR squared Bellman error.
Revised: This new operator is used with the squared Bellman error and could easily be used in combination with FR squared Bellman error.

Original: As mentionned above,
Revised: As mentionned above,

Original: Double Deep Q -learning (Van Hasselt, 2010; Van Hasselt et al., 2016) addresses positive bias in the value estimate (Thrun & Schwartz, 1993) by using two Q -networks, with each target network providing target values for the other.
Revised: Double Deep Q -learning (Van Hasselt, 2010; Van Hasselt et al., 2016) addresses positive bias in the value estimate (Thrun & Schwartz, 1993) by using two Q -networks, with each target network providing target values for the other.

Original: Fujimoto et al.
Revised: Fujimoto et al.

Original: (Fujimoto et al., 2018) extend and apply this technique to continuous control, instead using the minimum of the two target networks in the target value.
Revised: (Fujimoto et al., 2018) extend and apply this technique to continuous control, instead using the minimum of the two target networks in the target value.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed the FR Squared Bellman Error as an alternative to the Squared Bellman
Revised: In this paper, we proposed the FR Squared Bellman Error as an alternative to the Squared Bellman

Original: Error.
Revised: Error.

Original: The Bellman error requires target networks to stabilize the training of DQL methods due to the non-stationary nature of the regression problem.
Revised: The Bellman error requires target networks to stabilize the training of DQL methods due to the non-stationary nature of the regression problem.

Original: Like the squared Bellman error combined with target networks, the FR squared Bellman error uses a lagging set of parameters to stabilize estimation.
Revised: Like the squared Bellman error combined with target networks, the FR squared Bellman error uses a lagging set of parameters to stabilize estimation.

Original: An important distinction however is that the lagging set is decoupled from target value estimation, instead taking the form of a functional prior.
Revised: An important distinction however is that the lagging set is decoupled from target value estimation, instead taking the form of a functional prior.

Original: In this way, we can 1) use up-to-date target value estimates, thereby propagating reward information more quickly, and 2) separately control the degree of stabilization via the FR weight, κ .
Revised: In this way, we can 1) use up-to-date target value estimates, thereby propagating reward information more quickly, and 2) separately control the degree of stabilization via the FR weight, κ .

Original: Indeed, by comparing the gradients resulting from the squared Bellman error with target networks to the FR squared Bellman error, we noted that target networks implicitly perform a form of FR, but with lagging target values and a non-independent weight ( κ = γ ).
Revised: Indeed, by comparing the gradients resulting from the squared Bellman error with target networks to the FR squared Bellman error, we noted that target networks implicitly perform a form of FR, but with lagging target values and a non-independent weight ( κ = γ ).

Original: This helps illuminate why target networks stabilize training in practice, while also highlighting their drawbacks.
Revised: This helps illuminate why target networks stabilize training in practice, while also highlighting their drawbacks.

Original: We also demonstrated the stability of FR DQN in the Four Rooms environment by showing that it can quickly recover the optimal value function, and show that we achieve higher performance than
Revised: We also demonstrated the stability of FR DQN in the Four Rooms environment by showing that it can quickly recover the optimal value function, and show that we achieve higher performance than

Original: DQN and Polyak DQN.
Revised: DQN and Polyak DQN.

Original: Then we showed that FR DQN consistently match or outperform DQN and
Revised: Then we showed that FR DQN consistently match or outperform DQN and

Original: Polyak DQN on a subset of the Atari suite.
Revised: Polyak DQN on a subset of the Atari suite.

Original: Finally we showed that FR DQN can be combined with DQN to further improve performance.
Revised: Finally we showed that FR DQN can be combined with DQN to further improve performance.

Original: Thus, we conclude that FR Bellman error provides a conceptually simple, easy to implement, and effective objective for improving DQL methods.
Revised: Thus, we conclude that FR Bellman error provides a conceptually simple, easy to implement, and effective objective for improving DQL methods.

Original: While the FR Bellman error requires an additional weighting parameter, κ , we noted that the performance does not change dramatically for small change in κ .
Revised: While the FR Bellman error requires an additional weighting parameter, κ , we noted that the performance does not change dramatically for small change in κ .

Original: We see κ as providing needed ﬂexibility in setting the degree of regularization, unlike Target Networks, which cannot separately control this aspect.
Revised: We see κ as providing needed ﬂexibility in setting the degree of regularization, unlike Target Networks, which cannot separately control this aspect.

Original: In our experiments, we used a ﬁxed κ throughout training, however, future work could investigate methods for automatically adjusting κ during training (Haarnoja et al., 2018b) or conditioning it on individual state-action pairs.
Revised: In our experiments, we used a ﬁxed κ throughout training, however, future work could investigate methods for automatically adjusting κ during training (Haarnoja et al., 2018b) or conditioning it on individual state-action pairs.

Original: Nevertheless, the FR Squared Bellman Error objective that we have presented here performs well across a wide range of environments, providing a drop-in replacement for the Squared Bellman Error.
Revised: Nevertheless, the FR Squared Bellman Error objective that we have presented here performs well across a wide range of environments, providing a drop-in replacement for the Squared Bellman Error.

Original: A A PPENDIX A. Q - LEARNING WITH T ARGET N ETWORK
Revised: A A PPENDIX A. Q - LEARNING WITH T ARGET N ETWORK

Original: Letting R ( t ) ≡ R ( s t , a t ) , Q ( t ) ≡ Q ( s t , a t ) and ∆ Q ( t ) θ ≡ E [ Q ( t +1)
Revised: Letting R ( t ) ≡ R ( s t , a t ) , Q ( t ) ≡ Q ( s t , a t ) and ∆ Q ( t ) θ ≡ E [ Q ( t +1)

Original: θ − Q ( t ) θ | s t , a t ] =
Revised: θ − Q ( t ) θ | s t , a t ] =

Original: E
Revised: E

Original: [ Q ( t +1) θ | s t , a t ] − Q ( t ) θ ∇ θ l target ( θ )
Revised: [ Q ( t +1) θ | s t , a t ] − Q ( t ) θ ∇ θ l target ( θ )

Original: = − (cid:0)
Revised: = − (cid:0)

Original: R ( t ) + γ E P,π [ Q ( t +1) θ ] − Q ( t ) θ (cid:1) ∇ θ Q ( t ) θ = − (cid:0) R ( t ) + γ ( Q ( t ) θ + ∆ Q ( t ) θ ) − Q ( t ) θ (cid:1) ∇ θ Q ( t ) θ = − (cid:0) R ( t )
Revised: R ( t ) + γ E P,π [ Q ( t +1) θ ] − Q ( t ) θ (cid:1) ∇ θ Q ( t ) θ = − (cid:0) R ( t ) + γ ( Q ( t ) θ + ∆ Q ( t ) θ ) − Q ( t ) θ (cid:1) ∇ θ Q ( t ) θ = − (cid:0) R ( t )

Original: + γ ( Q ( t ) θ + ∆ Q ( t ) θ ) − (1 −
Revised: + γ ( Q ( t ) θ + ∆ Q ( t ) θ ) − (1 −

Original: γ + γ ) Q ( t ) 
Revised: γ + γ ) Q ( t ) θ

Revised: [Equation]

Original: θ (cid:1)
Revised: [Equation] A. C ONVERGENCE OF F UNCTIONALLY REGULARIZED VALUE ITERATION Let us consider the value iteration algorithm for the state-action value function in the tabular setting.

Original: ∇ θ Q ( t  )θ =− (cid:0) R( t ) + γQ ( t ) θ
Revised: [Equation] t where T π ( Q ) = R + γP π Q

Original: +
Revised: It is known that this algorithm converges linearly as this update is a contraction. We show in this section that value iteration with functional regularization enjoys similar properties.

Original: γ ∆ Q ( t ) θ − (1 − γ ) Q( t )θ − γQ ( t ) θ (cid:1) ∇ θ Q ( t ) θ = − (cid:0) R ( t ) + γ ∆ Q( t ) θ − (1 − γ ) Q ( t ) θ (cid:1) ∇ θ Q ( t ) θ + γ (cid:0) Q ( t ) θ − Q t θ (cid:1) ∇ θ Q ( t ) θ = − (cid:0) R ( t ) + γ∆ Q ( t ) θ − Q ( t ) θ + γQ ( t )
Revised: Theorem 1 (Convergence of functionally regularized value iteration) . Value iteration with learning rate η and with functional regularization (cid:96) ( Q, ¯ Q ) = κ 2 (cid:107) Q − ¯ Q (cid:107) 2 where ¯ Q is updated every T steps to the current Q value converges linearly for κ ≥ 0 , η ≤ 1 1+ γ + κ and T > log(1+2 κ

Original: θ (cid:1)
Revised: − γ ) logρ . Proof. For this proof, we need to show that Q will converge to Q π . For this we split the problem in two 1) on one hand we show that Q converges to Q πκ ( ¯ Q ) , the best FR-regularized Q value and 2) that Q πκ ( ¯ Q ) converges to Q π depending on ¯ Q .

Revised: First step Q and Q πκ ( ¯ Q ) :

Revised: Our loss function is

Revised: [Equation]

Revised: where ¯ Q is a prior state-action value function that will be update every T steps.

Revised: The update for FR-regularized loss would be

Revised: [Equation]

Revised: Therefore, equating Q t =

Revised: Q t +1 , the ﬁxed point is

Revised: [Equation]

Revised: Let us look at the Euclidean distance

Original: ∇ θ Q ( t ) θ + γ (cid:0) Q ( t ) θ − Q ( t ) θ (cid:1) ∇ θ Q ( t ) θ = − (cid:0) R ( t )
Revised: [Equation] This is guaranteed to be a contraction for the spectral radius ρ = max (cid:107) x [Equation] 1 x

Revised: [Equation]

Original:  + γ ( Q ( t ) θ + ∆ Q ( t ) θ )  − Q
Revised: (cid:0) I − η ((1 + κ ) I − γP π ) (cid:1) x < 1 . As the spectral radius of P π is 1 (Perron-Frobenius theorem), we can deduce that η < 2 1+ γ + κ is sufﬁcient to get the spectral radius I − η ((1 + κ ) I − γP π ) bounded by 1 .

Revised: Thus we have

Original:  (  t ) θ (cid:1) ∇ θ Q  ( t ) θ
Revised: [Equation] ≤ ρ T (cid:107) Q t − Q πκ ( Q t ) (cid:107) Second step Q πκ ( ¯ Q ) and Q π : First let us show that Q πκ ( ¯ Q ) = Q π

Original: +γ (cid:0) Q ( t )θ −  Q  ( t ) θ (cid:1) ∇ θ Q  ( t ) θ
Revised: + ( I + κ − γP π ) − 1 κ ( ¯ Q − Q π ) We have Q πκ ( ¯ Q ) = ((1+ κ ) I − γP π ) − 1 ( R + κ ¯ Q ) and Q π = ( I − γP π ) − 1 R .

Revised: Using A − 1 − B − 1 =

Original: A. Q - LEARNING WITH F UNCTIONAL R EGULARIZATION
Revised: A − 1 ( B − A ) B − 1 , we get [Equation] Putting it all together: Let us deﬁne the FR Bellman operator Q (cid:55)→ T TFR [ Q ] which consists into updating the prior in FR to the current Q -value then doing T steps of FR update on Q .

Revised: We have

Original: ∇ θ l FR ( θ ) =− (cid:0) R( t ) + γ E P,π [ Q ( t +1)
Revised: T T FR [ Q π ] = Q π as Q π = R + γP π Q π . (cid:107)T [Equation]

Revised: [Equation]

Revised: ) (cid:107) + (cid:107) Q πκ ( Q t ) − Q

Revised: [Equation]

Revised: (cid:107) ≤

Revised: ρ T (cid:107) Q t − Q πκ ( Q t ) (cid:107) + (cid:107) ( I + κ − γP

Revised: [Equation]

Original: θ ] −  Q( t ) θ (cid:1) ∇ θ Q  ( t ) θ + κ(cid:0) Q t θ − Q ( t ) θ (cid:1) ∇ Q ( t ) θ = −(cid:0) R ( t ) + γ ( Q( t ) θ + ∆ Q( t) θ ) − Q ( t ) θ (cid:1) ∇ θ Q ( t ) θ
Revised: ) − 1 κ (cid:107) 2 (cid:107) Q t − Q π (cid:107) ≤ ρ T (cid:107) (cid:0) I − ( I + κ − γP π ) − 1 κ (cid:1)(cid:0) Q π − Q t (cid:1) (cid:107) + µ (cid:107) Q t − Q π (cid:107) ≤ (cid:0) ρ [Equation] µ

Revised: [Equation]

Revised: Q

Original: + κ (cid:0) Q  ( t )θ − Q t θ (cid:1) ∇ Q ( t ) θ
Revised: [Equation] − Q [Equation] With µ = Spectral radius (cid:0) ( I + κ − γP π ) − 1 κ (cid:1) ≤ κ 1 − γ + κ < 1 as 1 − γ > 0 .

Revised: Thus, for

Revised: [Equation]

Revised: we have that (cid:0) ρ T (1+ µ )+ µ (cid:1) < 1 therefore our operator is a contraction, thus the methods converges linearly.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: 1e

Revised: [Table]

Revised: [Figure]

Original: B A PPENDIX B. F OUR R OOMS B. A TARI
Revised: B A PPENDIX B. F OUR R OOMS B. A TARI

Original: Most of the hyper-parameters have been kept from the defaut DQN and Double DQN respectively.
Revised: Most of the hyper-parameters have been kept from the defaut DQN and Double DQN respectively.

Original: Here is a list of the tuning that has been performed and the resulting best hyper-parameters.
Revised: Here is a list of the tuning that has been performed and the resulting best hyper-parameters.

Original: C A DDITIONAL E XPERIMENTAL D ETAILS D R ETURN AND OPTIMAL VALUE FUNCTION ERROR

Original: [Figure] [Table]
Revised: [Table] [Table] [Table]

Revised: - 4e 4e -

Original: [Table]
Revised: [Table]

Revised: C A DDITIONAL E XPERIMENTAL D ETAILS D R ETURN AND OPTIMAL VALUE FUNCTION ERROR