Faster Deep Reinforcement Learning with Slower Online Network
Abstract
Deep reinforcement learning algorithms often use two networks for value function optimization: an online network, and a target network that tracks the online network with some delay.
Using two separate networks enables the agent to hedge against issues that arise when performing bootstrapping.
In this paper we endow two popular deep reinforcement learning algorithms, namely DQN and
Rainbow, with updates that incentivize the online network to remain in the proximity of the target network.
This improves the robustness of deep reinforcement learning in presence of noisy updates.
The resultant agents, called DQN Pro and
Rainbow Pro, exhibit signiﬁcant performance improvements over their original counterparts on the Atari benchmark demonstrating the effectiveness of this simple idea in deep reinforcement learning.
The code for our paper is available here:
Github.com/amazon-research/fast-rl-with-slow-updates.
Introduction
An important competency of reinforcement-learning (RL) agents is learning in environments with large state spaces like those found in robotics (Kober et al., 2013),
dialog systems (Williams et al., 2017), and games (Tesauro, 1994; Silver et al., 2017).
Recent breakthroughs in deep RL have demonstrated that simple approaches such as Q-learning (Watkins & Dayan, 1992) can surpass human-level performance in challenging environments when equipped with deep neural networks for function approximation (Mnih et al., 2015).
Two components of a gradient-based deep RL agent are its objective function and optimization procedure.
The optimization procedure takes estimates of the gradient of the objective with respect to network parameters and updates the parameters accordingly.
In DQN (Mnih et al., 2015), for example, the objective function is the empirical expectation of the temporal difference (TD) error (Sutton, 1988) on a buffered set of environmental interactions (Lin, 1992), and variants of stochastic gradient descent are employed to best minimize this objective function.
A fundamental difﬁculty in this context stems from the use of bootstrapping.
Here, bootstrapping refers to the dependence of the target of updates on the parameters of the neural network, which is itself continuously updated during training.
Employing bootstrapping in RL stands in contrast to supervised-learning techniques and Monte-Carlo RL (Sutton & Barto, 2018), where the target of our gradient updates does not depend on the parameters of the neural network.
Mnih et al.
(2015) proposed a simple approach to hedging against issues that arise when using bootstrapping, namely to use a target network in value-function optimization.
The target network is updated periodically, and tracks the online network with some delay.
While this modiﬁcation constituted a major step towards combating misbehavior in Q-learning (Lee & He, 2019; Kim et al., 2019; Zhang et al., 2021), optimization instability is still prevalent (van Hasselt et al., 2018).
Our primary contribution is to endow DQN and Rainbow (Hessel et al., 2018) with a term that ensures the parameters of the online-network component remain in the proximity of the parameters of the target network.
Our theoretical and empirical results show that our simple proximal updates can remarkably increase robustness to noise without incurring additional computational or memory costs.
In particular, we present comprehensive experiments on the Atari benchmark (Bellemare et al., 2013) where proximal updates yield signiﬁcant improvements, thus revealing the beneﬁts of using this simple technique for deep RL.
Background and Notation
RL is the study of the interaction between an environment and an agent that learns to maximize reward through experience.
The Markov Decision Process (Puterman, 1994), or MDP, is used to mathematically deﬁne the RL problem.
An MDP is speciﬁed by the tuple h S , A , R , P ,  i , where S is the set of states and A is the set of actions.
The functions R : S ⇥ A ! [Equation]
P [Equation] S
[Equation] A ⇥ S ! [0 , 1] denote the reward and transition dynamics of the MDP.
Finally, a discounting factor  is used to formalize the intuition that short-term rewards are more valuable than those received later.
The goal in the RL problem is to learn a policy, a mapping from states to a probability distribution over actions, ⇡ : S ! P ( A ) , that obtains high sums of future discounted rewards.
An important concept in RL is the state value function.
Formally, it denotes the expected discounted sum of future rewards when committing to a policy ⇡ in a state s : v ⇡ ( s ) :
= E ⇥ P 1 t {+=
 +} t R t  S
= s, ⇡ ⇤ .
We deﬁne the Bellman operator T ⇡ as follows:
[Equation]
, which we can write compactly as: T ⇡ v := R ⇡ +  P ⇡ v , where ⇥
R ⇡ ⇤ ( s ) = P a 2 A ⇡ ( a | s ) R ( s, a ) and ⇥
P ⇡ v ⇤ ( s )
[Equation] P [Equation]
⇡
[Equation]
P s
[Equation]
S P ( s, a, s 0 ) v ( s 0 ) . We also denote: ( T ⇡ ) n v := T ⇡ · · · T ⇡ | {z } n compositions v .
Notice that v ⇡ is the unique ﬁxed-point of ( T ⇡ ) n for all natural numbers n , meaning that v ⇡ =
( T ⇡ ) n v ⇡ , for all n .
Deﬁne v ? as the optimal value of a state, namely: v ? ( s ) :
= max ⇡ v ⇡ ( s ) , and ⇡ ? as a policy that achieves v ? ( s ) for all states.
We deﬁne the Bellman Optimality Operator T ? : ⇥
[Equation]
whose ﬁxed point is v ? .
These operators are at the heart of many planning and RL algorithms including Value Iteration (Bellman, 1957) and Policy Iteration (Howard, 1960).
Proximal Bellman Operator
In this section, we introduce a new class of Bellman operators that ensure that the next iterate in planning and RL remain in the vicinity of the previous iterate.
To this end, we deﬁne the Bregman
Divergence generated by a convex function f :
[Equation]
Examples include the l p norm generated by f ( v ) = 12 k v k 2 p and the Mahalanobis Distance generated by f ( v ) = 12 h v, Qv i for a positive semi-deﬁnite matrix Q .
We now deﬁne the Proximal Bellman Operator ( T ⇡ c,f ) n :
[Equation]
where c 2 (0 , 1 ) .
Intuitively, this operator encourages the next iterate to be in the proximity of the previous iterate, while also having a small difference relative to the point recommended by the original Bellman Operator.
The parameter c could, therefore, be thought of as a knob that controls the degree of gravitation towards the previous iterate.
Our goal is to understand the behavior of Proximal Bellman Operator when used in conjunction with the Modiﬁed Policy Iteration (MPI) algorithm (Puterman, 1994; Scherrer et al., 2015).
Deﬁne G v as the greedy policy with respect to v .
At a certain iteration k , Proximal Modiﬁed Policy Iteration (PMPI) proceeds as follows:
[Equation]
The pair of updates above generalize existing algorithms.
Notably, with c ! 1 and general n we get
MPI, with c ! 1 and n = 1 the algorithm reduces to Value Iteration, and with c ! 1 and n = 1 we have a reduction to Policy Iteration.
For ﬁnite c , the two extremes of n , namely n = 1 and n = 1 , could be thought of as the proximal versions of Value Iteration and Policy Iteration, respectively.
To analyze this approach, it is ﬁrst natural to ask if each iteration of PMPI could be thought of as a contraction so we can get sound and convergent behavior in planning and learning.
For n > 1 ,
Scherrer et al.
(2015) constructed a contrived MDP demonstrating that one iteration of MPI can unfortunately expand.
As PMPI is just a generalization of MPI, the same example from Scherrer et al.
(2015) shows that PMPI can expand.
In the case of n = 1 , we can rewrite the pair of equations (2) and (3) in a single update as follows: v k T ? c,f v k  1 .
When c ! 1 , standard proofs can be employed to show that the operator is a contraction (Littman & Szepesvári, 1996).
We now show that
T ? c,f is a contraction for ﬁnite values of c .
See our appendix for proofs.
Theorem 1.
The Proximal Bellman Optimality Operator T ? c,f is a contraction with ﬁxed point v ? .
Therefore, we get convergent behavior when using T ? c,f in planning and RL.
The addition of the proximal term is fortunately not changing the ﬁxed point, thus not negatively affecting the ﬁnal solution.
This could be thought of as a form of regularization that vanishes in the limit; the algorithm converges to v ? even without decaying 1 /c .
Going back to the general n  1 case, we cannot show contraction, but following previous work (Bertsekas & Tsitsiklis, 1996; Scherrer et al., 2015), we study error propagation in PMPI in presence of additive noise where we get a noisy sample of the original Bellman Operator ( T ⇡ k ) n v k  1 + ✏ k .
The noise can stem from a variety of reasons, such as approximation or estimation error.
For simplicity, we restrict the analysis to D f ( v 0 , v ) =
|| v 0  v || 22 , so we rewrite update (3) as:
[Equation]
which can further be simpliﬁed to:
[Equation]
where  = 11+ c .
This operator is a generalization of the operator proposed by Smirnova & Dohmatob (2020) who focused on the case of n = 1 .
To build some intuition, notice that the update is multiplying error ✏ k by a term that is smaller than one, thus better hedging against large noise.
While the update may slow progress when there is no noise, it is entirely conceivable that for large enough values of ✏ k , it is better to use non-zero  values.
In the following theorem we formalize this intuition.
Our result leans on the theory provided by Scherrer et al.
and could be thought of as a generalization of their theorem for non-zero  values.
Theorem 2.
Consider the PMPI algorithm speciﬁed by:
[Equation]
Deﬁne the Bellman residual b k := v k  T ⇡ k +1 v k , and error terms x k :=
( I   P ⇡ k ) ✏ k and y k :=
P ⇡ ⇤ ✏ k .
After k steps:
[Equation]
• where d k   P ⇡ ⇤ d k  1   (1   ) y k  1 +  b k  [Equation]
[Equation] [Equation] P ⇡
[Equation] [Equation]
[Equation] k • s k   (1   )(  P ⇡ k ) n +  I  ( I   P ⇡ k )  1 b k • b k   (1   )(  P ⇡ k ) n +  I  b k  1 + (1   ) x k + ✏ 0 k +
The bound provides intuition as to how the proximal Bellman Operator can accelerate convergence in the presence of high noise.
For simplicity, we will only analyze the effect of the ✏ noise term, and ignore the ✏ 0 term.
We ﬁrst look at the Bellman residual, b k .
Given the Bellman residual in iteration k  1 , b k  1 , the only inﬂuence of the noise term ✏ k on b k is through the (1   ) x k , term, and we see that b k decreases linearly with larger  .
The analysis of s k is slightly more involved but follows similar logic.
The bound for s k can be decomposed into a term proportional to b k  1 and a term proportional to  b k  1 , where both are multiplied with positive semi-deﬁnite matrices.
Since b k  1 itself linearly decreases with  , we conclude that larger  decreases the bound quadratically.
The effect of  on the bound for d k is more complex.
The terms  y k  1 and [Equation]
introduce a linear decrease of the bound on d k with  , while the term  ( I  [Equation]
introduces a quadratic dependence whose curvature depends on I  P [Equation] ) j .
This complex dependence on  highlights the trade-off between noise reduction and magnitude of updates.
To understand this trade-off better, we examine two extreme cases for the magnitude on the noise.
When the noise is very large, we may set  = 1 , equivalent to an inﬁnitely strong proximal term.
It is easy to see that for  = 1 , the values of d k and s k remain unchanged, which is preferable to the increase they would suffer in the presence of very large noise.
On the other extreme, when no noise is present, the x k and y k terms in Theorem 2 vanish, and the bounds on d k and s k can be minimized by setting  = 0 , i.e. without noise the proximal term should not be used and the original Bellman update performed.
Intermediate noise magnitudes thus require a value of  that balances the noise reduction and update size.
Deep Q-Network with Proximal Updates
We now endow DQN-style algorithms with proximal updates.
Let h s, a, r, s {+
 i+} denote a buffered tuple of interaction.
Deﬁne the following objective function:
[Equation]
Our proximal update is deﬁned as follows:
[Equation]
This algorithm closely resembles the standard proximal-point algorithm (Rockafellar, 1976; Parikh &
Boyd, 2014) with the important caveat that the function h is now taking two vectors as input.
At each iteration, we hold the ﬁrst input constant while optimizing over the second input.
In the optimization literature, the proximal-point algorithm is well-studied in contexts where an analytical solution to (7) is available.
With deep learning no closed-form solution exists, so we approximately solve (7) by taking a ﬁxed number of descent steps using stochastic gradients.
Speciﬁcally, starting each iteration with w = w t , we perform multiple w updates w
[Equation]  .
We end the iteration by setting w t + w .
To make a connection to standard deep RL, the online weights w could be thought of as the weights we maintain in the interim to solve (7) due to lack of a closed-form solution.
Also, what is commonly referred to as the target network could better be thought of as just the previous iterate in the above proximal-point algorithm.
Observe that the update can be written as: w [Equation]
Notice the intuitively appealing form: we ﬁrst compute a convex combination of w t and w , based on the hyper-parameters ↵ and ˜ c , then add the gradient term to arrive at the next iterate of w .
If w t and w are close, the convex combination is close to w itself and so this DQN with proximal update
( DQN Pro ) would behave similarly to the original DQN.
However, when w strays too far from w t , taking the convex combination ensures that w gravitates towards the previous iterate w t .
The gradient signal from minimizing the squared TD error (6) should then be strong enough to cancel this default gravitation towards w t .
The update includes standard DQN as a special case when ˜ c ! 1 .
The pseudo-code for DQN is presented in the Appendix.
The difference between DQN and DQN Pro is minimal (shown in gray), and corresponds with a few lines of code in our implementation.
Experiments
In this section, we empirically investigate the effectiveness of proximal updates in planning and reinforcement-learning algorithms.
We begin by conducting experiments with PMPI in the context of approximate planning, and then move to large-scale RL experiments in Atari.
5.1 PMPI Experiments
We now focus on understanding the empirical impact of adding the proximal term on the performance of approximate PMPI.
To this end, we use the pair of update equations:
[Equation]
For this experiment, we chose the toy 8 ⇥ 8 Frozen Lake environment from Open AI
Gym (Brockman et al., 2016), where the transition and reward model of the environment is available to the planner.
Using a small environment allows us to understand the impact of the proximal term in the simplest and most clear setting.
Note also that we arranged the experiment so that the policy greediﬁcation step G v k  1 8 k is error-free, so we can solely focus on the interplay between the proximal term and the error caused by imperfect policy evaluation.
We applied 100 iterations of PMPI, then measured the quality of the resultant policy ⇡ := ⇡ 100 as deﬁned by the distance between its true value and that of the optimal policy, namely k V ?  V ⇡ k 1 .
We repeated the experiment with different magnitudes of error, as well as different values of the  parameter.
Results are averaged over 30 random seeds with n = 1 (left) and n = 3 (right) .
From Figure 1, it is clear that the ﬁnal performance exhibits a U-shape with respect to the parameter  .
It is also noticable that the best-performing  is shifting to the right side (larger values) as we increase the magnitude of noise.
This trend makes sense, and is consistent with what is predicted by
Theorem 2: As the noise level rises, we have more incentive to use larger (but not too large)  values to hedge against it.
5.2 Atari Experiments
In this section, we evaluate the proximal (or Pro) agents relative to their original DQN-style counterparts on the Atari benchmark (Bellemare et al., 2013), and show that endowing the agent with the proximal term can lead into signiﬁcant improvements in the interim as well as in the ﬁnal performance.
We next investigate the utility of our proposed proximal term through further experiments.
Please see the Appendix for a complete description of our experimental pipeline.
5.2.1 Setup
We used 55 Atari games (Bellemare et al., 2013) to conduct our experimental evaluations.
Following
Machado et al.
(2018) and Castro et al.
(2018), we used sticky actions to inject stochasticity into the otherwise deterministic Atari emulator.
Our training and evaluation protocols and the hyper-parameter settings follow those of the Dopamine baseline (Castro et al., 2018).
To report performance, we measured the undiscounted sum of rewards obtained by the learned policy during evaluation.
We further report the learning curve for all experiments averaged across 5 random seeds.
We reiterate that we used the exact same hyper-parameters for all agents to ensure a sound comparison.
Our Pro agents have a single additional hyper-parameter ˜ c .
We did a minimal random search ongames to tune ˜ c .
Figure 2 visualizes the performance of Pro agents as a function of ˜ c .
In light of this result, we set ˜ c
= 0 .
2 for DQN Pro and ˜ c
= 0 .
05 for Rainbow Pro.
We used these values of ˜ c for all 55 games, and note that we performed no further hyper-parameter tuning at all.
5.2.2 Results
The ﬁrst question is whether endowing the DQN agent with the proximal term can yield signiﬁcant improvements over the original DQN.
Figure 3 (top) shows a comparison between DQN and DQN
Pro in terms of the ﬁnal performance.
In particular, following standard practice (Wang et al., 2016;
Dabney et al., 2018; van Seijen et al., 2019), for each game we compute:
[Figure]
Bars shown in red indicate the games in which we observed better ﬁnal performance for DQN Pro relative to DQN, and bars in blue indicate the opposite.
The height of a bar denotes the magnitude of this improvement for the corresponding benchmark; notice that the y-axis is scaled logarithmically.
We took human and random scores from previous work (Nair et al., 2015; Dabney et al., 2018).
It is clear that DQN Pro dramatically improves upon DQN.
We defer to the Appendix for full learning curves on all games tested.
Can we fruitfully combine the proximal term with some of the existing algorithmic improvements in DQN?
To answer this question, we build on the Rainbow algorithm of Hessel et al.
(2018) who successfully combined numerous important algorithmic ideas in the value-based RL literature.
We present this result in Figure 3 (bottom).
Observe that the overall trend is for Rainbow Pro to yield large performance improvements over Rainbow.
Additionally, we measured the performance of our agents relative to human players.
To this end, and again following previous work (Wang et al., 2016; Dabney et al., 2018; van Seijen et al., 2019), for each agent we compute the human-normalized score: [Figure] [Equation] [Figure]
[Equation]
In Figure 4 (left), we show the median of this score for all agents, which Wang et al.
(2016) and Hessel et al.
(2018) argued is a sensible quantity to track.
We also show per-game learning curves with standard error in the Appendix.
We make two key observations from this ﬁgure.
First, the very basic DQN Pro agent is capable of achieving human-level performance (1.0 on the y-axis) after 120 million frames.
Second, the
Rainbow Pro agent achieves 220 percent human-normalized score after only 120 million frames.
5.2.3 Additional Experiments
Our purpose in endowing the agent with the proximal term was to keep the online network in the vicinity of the target network, so it would be natural to ask if this desirable property can manifest itself in practice when using the proximal term.
In Figure 4, we answer this question afﬁrmatively by plotting the magnitude of the update to the target network during synchronization.
Notice that we periodically synchronize online and target networks, so the proximity of the online and target network should manifest itself in a low distance between two consecutive target networks.
Indeed, the results demonstrate the success of the proximal term in terms of obtaining the desired proximity of online and target networks.
While using the proximal term leads to signiﬁcant improvements, one may still wonder if the advantage of DQN Pro over DQN is merely stemming from a poorly-chosen period hyper-parameter in the original DQN, as opposed to a truly more stable optimization in DQN Pro.
To refute this hypothesis, we ran DQN with various settings of the period hyper-parameter { 2000 , 4000 , 8000 , 12000 } .
This set included the default value of the hyper-parameter (8000) from the original paper (Mnih et al., 2015), but also covered a wider set of settings.
Additionally, we tried an alternative update strategy for the target network, referred to as Polyak averaging, which was popularized in the context of continuous-action RL (Lillicrap et al., 2015): ✓ ⌧ w + (1  ⌧ ) ✓ .
For this update strategy, too, we tried different settings of the ⌧ hyper-parameter, namely { 0 . 05 , 0 .
005 , 0 . 0005 } , which includes the value 0 .
005 used in numerous papers (Lillicrap et al., 2015; Fujimoto et al., 2018; Asadi et al., 2021).
Figure 5 presents a comparison between DQN Pro and DQN with periodic and Polyak target updates for various hyper-parameter settings of period and ⌧ .
It is clear that DQN Pro is consistently outperforming the two alternatives regardless of the speciﬁc values of period and ⌧ , thus clearly demonstrating that the improvement is stemming from a more stable optimization procedure leading to a better interplay between the two networks.
Finally, an alternative approach to ensuring lower distance between the online and the target network is to anneal the step size based on the number of updates performed on the online network since the last online-target synchronization.
In this case we performed this experiment in 4 games where we knew proximal updates provide improvements based on our DQN Pro versus DQN resulst in
Figure 3.
In this case we linearly decreased the step size from the original DQN learning rate ↵ to ↵ 0 ⌧ ↵ where we tuned ↵ 0 using random search.
Annealing indeed improves DQN, but DQN
Pro outperforms the improved version of DQN.
Our intuition is that Pro agents only perform small updates when the target network is far from the online network, but naively decaying the learning rate can harm progress when the two networks are in vicinity of each other.
Discussion
In our experience using proximal updates in the parameter space were far superior than proximal updates in the value space.
We believe this is because the parameter-space deﬁnition can enforce the proximity globally, while in the value space one can only hope to obtain proximity locally and on a batch of samples.
One may hope to use natural gradients to enforce value-space proximity in a more principled way, but doing so usually requires signiﬁcantly more computational resources Knight &
Lerner (2018).
This is in contrast to our proximal updates which add negligible computational cost in the simple form of taking a dimension-wise weighted average of two weight vectors.
In addition, for a smooth (Lipschitz) Q function, performing parameter-space regularization guarantees function-space regularization.
Concretely: 8 s, 8 a | Q ( s, a ; ✓ )  Q ( s, a ; ✓ 0 ) |  L ||
✓  ✓ 0 || , where L is the Lipschitz constant of Q .
Moreover, deep networks are Lipschitz (Neyshabur et al., 2015;
Asadi et al., 2018), because they are constructed using compositions of Lipschitz functions (such as
ReLU, convolutions, etc) and that composition of Lipschitz functions is Lipschitz.
So performing value-space updates may be an overkill.
Lipschitz property of deep networks has successfully been leveraged in other contexts, such as in generative adversarial training Arjovsky et al.
A key selling point of our result is simplicity, because simple results are easy to understand, implement, and reproduce.
We obtained signiﬁcant performance improvements by adding just a few lines of codes to the publicly available implementations of DQN and Rainbow Castro et al.
Related Work
The introduction of proximal operators could be traced back to the seminal work of Moreau (1962, 1965), Martinet (1970) and Rockafellar (1976), and the use of the proximal operators has since expanded into many areas of science such as signal processing (Combettes & Pesquet, 2009), statistics and machine learning (Beck & Teboulle, 2009;
Polson et al., 2015; Reddi et al., 2015), and convex optimization (Parikh & Boyd, 2014; Bertsekas, 2011b,a).
In the context of RL, Mahadevan et al.
(2014) introduced a proximal theory for deriving convergent off-policy algorithms with linear function approximation.
One intriguing characteristic of their work is that they perform updates in primal-dual space, a property that was leveraged in sample complexity analysis (Liu et al., 2020) for the proximal counterparts of the gradient temporal-difference algorithm (Sutton et al., 2008).
Proximal operators also have appeared in the deep RL literature.
For instance, Fakoor et al.
(2020b) used proximal operators for meta learning, and Maggipinto et al.
(2020) improved TD3 (Fujimoto et al., 2018) by employing a stochastic proximal-point interpretation.
The effect of the proximal term in our work is reminiscent of the use of trust regions in policy-gradient algorithms (Schulman et al., 2015, 2017;
Wang et al., 2019; Fakoor et al., 2020a; Tomar et al., 2021).
However, three factors differentiate our work: we deﬁne the proximal term using the value function, not the policy, we enforce the proximal term in the parameter space, as opposed to the function space, and we use the target network as the previous iterate in our proximal deﬁnition.
Conclusion and Future work
We showed a clear advantage for using proximal terms to perform slower but more effective updates in approximate planning and reinforcement learning.
Our results demonstrated that proximal updates lead to more robustness with respect to noise.
Several improvements to proximal methods exist, such as the acceleration algorithm (Nesterov, 1983; Li & Lin, 2015), as well as using other proximal terms (Combettes & Pesquet, 2009), which we leave for future work.
Acknowledgment
We thank Lihong Li, Pratik Chaudhari, and Shoham Sabach for their valuable insights in different stages of this work.
[-numUpdates ← numUpdates +14: if numUpdates % period = 
-]
[-= 
-]