Faster Deep Reinforcement Learning with Slower Online Network
Abstract
Deep reinforcement learning algorithms often use two networks for value function optimization: an online network, and a target network that tracks the online network with some delay.
Using two separate networks enables the agent to hedge against issues that arise when performing bootstrapping.
In this paper we endow two popular deep reinforcement learning algorithms, namely DQN and Rainbow, with updates that incentivize the online network to remain in the proximity of the target network.
This improves the robustness of deep reinforcement learning in presence of noisy updates.
The resultant agents, called DQN Pro and Rainbow Pro, exhibit significant performance improvements over their original counterparts on the Atari benchmark demonstrating the effectiveness of this simple idea in deep reinforcement learning. The code for our paper is available here: Github.com/amazon-research/fast-rl-with-slow-updates.
Introduction
An important competency of reinforcement-learning (RL) agents is learning in environments with large state spaces like those found in robotics (Kober et al., 2013), dialog systems (Williams et al., 2017), and games (Tesauro, 1994; Silver et al., 2017).
Recent breakthroughs in deep RL have demonstrated that simple approaches such as Q-learning (Watkins & Dayan, 1992) can surpass human-level performance in challenging environments when equipped with deep neural networks for function approximation (Mnih et al., 2015).
Two components of a gradient-based deep RL agent are its objective function and optimization procedure.
The optimization procedure takes estimates of the gradient of the objective with respect to network parameters and updates the parameters accordingly.
In DQN (Mnih et al., 2015), for example, the objective function is the empirical expectation of the temporal difference (TD) error (Sutton, 1988) on a buffered set of environmental interactions (Lin, 1992), and variants of stochastic gradient descent are employed to best minimize this objective function.
A fundamental difficulty in this context stems from the use of bootstrapping.
Here, bootstrapping refers to the dependence of the target of updates on the parameters of the neural network, which is itself continuously updated during training.
Employing bootstrapping in RL stands in contrast to supervised-learning techniques and Monte-Carlo RL (Sutton & Barto, 2018), where the target of our gradient updates does not depend on the parameters of the neural network.
Mnih et al.
(2015) proposed a simple approach to hedging against issues that arise when using bootstrapping, namely to use a target network in value-function optimization.
The target network is updated periodically, and tracks the online network with some delay.
While this modification constituted a major step towards combating misbehavior in Q-learning (Lee & He, 2019; Kim et al., 2019; Zhang et al., 2021), optimization instability is still prevalent (van Hasselt et al., 2018).
Our primary contribution is to endow DQN and Rainbow (Hessel et al., 2018) with a term that ensures the parameters of the online-network component remain in the proximity of the parameters of the target network.
Our theoretical and empirical results show that our simple proximal updates can remarkably increase robustness to noise without incurring additional computational or memory costs.
In particular, we present comprehensive experiments on the Atari benchmark (Bellemare et al., 2013) where proximal updates yield significant improvements, thus revealing the benefits of using this simple technique for deep RL.
Background and Notation
RL is the study of the interaction between an environment and an agent that learns to maximize reward through experience.
The Markov Decision Process (Puterman, 1994), or MDP, is used to mathematically define the RL problem.
An MDP is specified by the tuple ⟨S , A , R , P , γ ⟩ , where S is the set of states and A is the set of actions.
The functions R : S ×
A → R and P : S ×
A × S → [0 , 1] denote the reward and transition dynamics of the MDP.
Finally, a discounting factor γ is used to formalize the intuition that short-term rewards are more valuable than those received later.
The goal in the RL problem is to learn a policy, a mapping from states to a probability distribution over actions, π : S → P ( A ) , that obtains high sums of future discounted rewards.
An important concept in RL is the state value function.
Formally, it denotes the expected discounted sum of future rewards when committing to a policy π in a state s : v π ( s ) :
= E (cid:2) (cid:80) ∞ t =0 γ t R t (cid:12)(cid:12) S
= s, π (cid:3) .
We define the Bellman operator T π as follows:
[Equation]
which we can write compactly as: T π v :=
R π + γP π v , where (cid:2) R π (cid:3) ( s ) =
(cid:80) a ∈A π ( a | s ) R ( s, a ) and (cid:2) P π v (cid:3) ( s ) =
(cid:80) a ∈A π ( a | s ) (cid:80) s ′ ∈S P ( s, a, s ′ ) v ( s ′ ) .
We also denote: ( T π ) n v := T π · ·
[Equation] T [Equation] n compositions v .
Notice that v π is the unique fixed-point of ( T π ) n for all natural numbers n , meaning that v π = ( T π ) n v π , for all n .
Define v ⋆ as the optimal value of a state, namely: v ⋆ ( s ) :
= max π v π ( s ) , and π ⋆ as a policy that achieves v ⋆ ( s ) for all states.
We define the Bellman Optimality Operator T ⋆ :
[Equation]
whose fixed point is v ⋆ .
These operators are at the heart of many planning and RL algorithms including Value Iteration (Bellman, 1957) and Policy Iteration (Howard, 1960).
Proximal Bellman Operator
In this section, we introduce a new class of Bellman operators that ensure that the next iterate in planning and RL remain in the vicinity of the previous iterate.
To this end, we define the Bregman
Divergence generated by a convex function f :
[Equation]
− v ⟩ . Examples include the l p norm generated by f ( v ) = 12 ∥ v ∥ 2 p and the Mahalanobis Distance generated by f ( v ) = 12 ⟨ v, Qv ⟩ for a positive semi-definite matrix Q .
We now define the Proximal Bellman Operator ( T πc,f ) n :
[Equation]
where c ∈ (0 , ∞ ) .
Intuitively, this operator encourages the next iterate to be in the proximity of the previous iterate, while also having a small difference relative to the point recommended by the original Bellman Operator.
The parameter c could, therefore, be thought of as a knob that controls the degree of gravitation towards the previous iterate.
Our goal is to understand the behavior of Proximal Bellman Operator when used in conjunction with the Modified Policy Iteration (MPI) algorithm (Puterman, 1994; Scherrer et al., 2015).
Define G v as the greedy policy with respect to v .
At a certain iteration k , Proximal Modified Policy Iteration (PMPI) proceeds as follows:
[Equation]
The pair of updates above generalize existing algorithms.
Notably, with c → ∞ and general n we get
MPI, with c → ∞ and n = 1 the algorithm reduces to Value Iteration, and with c → ∞ and n = ∞ we have a reduction to Policy Iteration.
For finite c , the two extremes of n , namely n = 1 and n = ∞ , could be thought of as the proximal versions of Value Iteration and Policy Iteration, respectively.
To analyze this approach, it is first natural to ask if each iteration of PMPI could be thought of as a contraction so we can get sound and convergent behavior in planning and learning.
For n > 1 ,
Scherrer et al.
(2015) constructed a contrived MDP demonstrating that one iteration of MPI can unfortunately expand.
As PMPI is just a generalization of MPI, the same example from Scherrer et al.
(2015) shows that PMPI can expand.
In the case of n = 1 , we can rewrite the pair of equations (2) and (3) in a single update as follows: v k ← T ⋆c,f v k − 1 .
When c → ∞ , standard proofs can be employed to show that the operator is a contraction (Littman & Szepesvári, 1996).
We now show that
T ⋆c,f is a contraction for finite values of c .
See our appendix for proofs.
Theorem 1.
The Proximal Bellman Optimality Operator T ⋆c,f is a contraction with fixed point v ⋆ .
Therefore, we get convergent behavior when using T ⋆c,f in planning and RL.
The addition of the proximal term is fortunately not changing the fixed point, thus not negatively affecting the final solution.
This could be thought of as a form of regularization that vanishes in the limit; the algorithm converges to v ⋆ even without decaying 1 /c .
Going back to the general n ≥ 1 case, we cannot show contraction, but following previous work (Bertsekas & Tsitsiklis, 1996; Scherrer et al., 2015), we study error propagation in PMPI in presence of additive noise where we get a noisy sample of the original Bellman Operator ( T π k ) n v k − 1 + ϵ k .
The noise can stem from a variety of reasons, such as approximation or estimation error.
For simplicity, we restrict the analysis to D f ( v ′ , v ) =
|| v ′ − v || 22 , so we rewrite update (3) as:
[Equation]
which can further be simplified to:
[Equation]
where β
= 11+ c .
This operator is a generalization of the operator proposed by Smirnova & Dohmatob (2020) who focused on the case of n = 1 .
To build some intuition, notice that the update is multiplying error ϵ k by a term that is smaller than one, thus better hedging against large noise.
While the update may slow progress when there is no noise, it is entirely conceivable that for large enough values of ϵ k , it is better to use non-zero β values.
In the following theorem we formalize this intuition.
Our result leans on the theory provided by Scherrer et al. and could be thought of as a generalization of their theorem for non-zero β values.
Theorem 2.
Consider the PMPI algorithm specified by:
[Equation]
[Equation]
Define the Bellman residual b k := v k − T π k +1 v k , and error terms x k :=
( I − γP π k )
ϵ k
and y
k :=
γP π ∗ ϵ k .
After k steps:
[Equation]
• where d k ≤ γP π ∗ d k − 1 − (cid:0) (1 − β ) y k − 1 + βb k − 1 (cid:1) + (1 − β ) (cid:80) n − 1 j =1 ( γP π k ) j b k − 1 + ϵ ′ k • s k ≤ (cid:0) (1 − β )( γP π k ) n + βI (cid:1) ( I − γP π k ) − 1 b k −• b k ≤ (cid:0) (1 − β )( γP π k ) n + βI (cid:1)
b
k − 1 + (1 − β ) x k + ϵ ′ k
+
[-b k − 1 + (1 − β ) x k + ϵ ′ k +
-]
The bound provides intuition as to how the proximal Bellman Operator can accelerate convergence in the presence of high noise.
For simplicity, we will only analyze the effect of the ϵ noise term, and ignore the ϵ ′ term.
We first look at the Bellman residual, b k .
Given the Bellman residual in iteration k − 1 , b k − 1 , the only influence of the noise term ϵ k on b k is through the (1 − β ) x k , term, and we see that b k decreases linearly with larger β .
The analysis of s k is slightly more involved but follows similar logic.
The bound for s k can be decomposed into a term proportional to b k − 1 and a term proportional to βb k − 1 , where both are multiplied with positive semi-definite matrices.
Since b k − 1 itself linearly decreases with β , we conclude that larger β decreases the bound quadratically.
The effect of β on the bound for d k is more complex.
The terms βy k − 1 and (cid:80) n − 1 j =1 ( γP π k )
j b k −introduce a linear decrease of the bound on d k with β , while the term β ( I − (cid:80) n − 1 j =1 ( γP π k ) j ) b k −introduces a quadratic dependence whose curvature depends on I − (cid:80) n − 1 j =1 ( γP π k ) j .
This complex dependence on β highlights the trade-off between noise reduction and magnitude of updates.
To understand this trade-off better, we examine two extreme cases for the magnitude on the noise.
When the noise is very large, we may set β = 1 , equivalent to an infinitely strong proximal term.
It is easy to see that for β = 1 , the values of d k and s k remain unchanged, which is preferable to the increase they would suffer in the presence of very large noise.
On the other extreme, when no noise is present, the x k and y k terms in Theorem 2 vanish, and the bounds on d k and s k can be minimized by setting β = 0 , i.e. without noise the proximal term should not be used and the original Bellman update performed.
Intermediate noise magnitudes thus require a value of β that balances the noise reduction and update size.
Deep Q-Network with Proximal Updates
We now endow DQN-style algorithms with proximal updates.
Let ⟨ s, a, r, s ′ ⟩ denote a buffered tuple of interaction.
Define the following objective function:
[Equation]
Our proximal update is defined as follows:
[Equation]
This algorithm closely resembles the standard proximal-point algorithm (Rockafellar, 1976; Parikh &
Boyd, 2014) with the important caveat that the function h is now taking two vectors as input. At each iteration, we hold the first input constant while optimizing over the second input. In the optimization literature, the proximal-point algorithm is well-studied in contexts where an analytical solution to (7) is available. With deep learning no closed-form solution exists, so we approximately solve (7) by taking a fixed number of descent steps using stochastic gradients.
Specifically, starting each iteration with w = w t , we perform multiple w updates w ← w − α (cid:0) ∇ 2 h ( w t , w )
+ 1˜ c ( w − w t ) (cid:1) . We end the iteration by setting w t +1 ← w .
To make a connection to standard deep RL, the online weights w could be thought of as the weights we maintain in the interim to solve (7) due to lack of a closed-form solution. Also, what is commonly referred to as the target network could better be thought of as just the previous iterate in the above proximal-point algorithm.
Observe that the update can be written as: w ← (cid:0) 1 − ( α/ ˜ c ) (cid:1) · w + ( α/ ˜ c ) · w t − α ∇ 2 h ( w t , w ) .
Notice the intuitively appealing form: we first compute a convex combination of w t and w , based on the hyper-parameters α and ˜ c , then add the gradient term to arrive at the next iterate of w .
If w t and w are close, the convex combination is close to w itself and so this DQN with proximal update
( DQN Pro ) would behave similarly to the original DQN.
However, when w strays too far from w t , taking the convex combination ensures that w gravitates towards the previous iterate w t .
The gradient signal from minimizing the squared TD error (6) should then be strong enough to cancel this default gravitation towards w t .
The update includes standard DQN as a special case when ˜ c → ∞ .
The pseudo-code for DQN is presented in the Appendix.
The difference between DQN and DQN Pro is minimal (shown in gray), and corresponds with a few lines of code in our implementation.
Experiments
In this section, we empirically investigate the effectiveness of proximal updates in planning and reinforcement-learning algorithms.
We begin by conducting experiments with PMPI in the context of approximate planning, and then move to large-scale RL experiments in Atari.
5.1 PMPI Experiments
We now focus on understanding the empirical impact of adding the proximal term on the performance of approximate PMPI.
To this end, we use the pair of update equations:
[Equation]
For this experiment, we chose the toy 8 × 8 Frozen Lake environment from Open AI
Gym (Brockman et al., 2016), where the transition and reward model of the environment is available to the planner.
Using a small environment allows us to understand the impact of the proximal term in the simplest and most clear setting.
Note also that we arranged the experiment so that the policy greedification step G v k − 1 ∀ k is error-free, so we can solely focus on the interplay between the proximal term and the error caused by imperfect policy evaluation.
We applied 100 iterations of PMPI, then measured the quality of the resultant policy π :=
π 100 as defined by the distance between its true value and that of the optimal policy, namely ∥ V ⋆ − V π ∥ ∞ .
We repeated the experiment with different magnitudes of error, as well as different values of the β parameter.
[Figure]
From Figure 1, it is clear that the final performance exhibits a U-shape with respect to the parameter β .
It is also noticable that the best-performing β is shifting to the right side (larger values) as we increase the magnitude of noise.
This trend makes sense, and is consistent with what is predicted by
Theorem 2: As the noise level rises, we have more incentive to use larger (but not too large) β values to hedge against it.
5.2 Atari Experiments
In this section, we evaluate the proximal (or Pro) agents relative to their original DQN-style counterparts on the Atari benchmark (Bellemare et al., 2013), and show that endowing the agent with the proximal term can lead into significant improvements in the interim as well as in the final performance.
We next investigate the utility of our proposed proximal term through further experiments.
Please see the Appendix for a complete description of our experimental pipeline.
5.2.1 Setup
We used 55 Atari games (Bellemare et al., 2013) to conduct our experimental evaluations.
Following
Machado et al.
(2018) and Castro et al.
(2018), we used sticky actions to inject stochasticity into the otherwise deterministic Atari emulator.
Our training and evaluation protocols and the hyper-parameter settings follow those of the Dopamine baseline (Castro et al., 2018).
To report performance, we measured the undiscounted sum of rewards obtained by the learned policy during evaluation.
We further report the learning curve for all experiments averaged across 5 random seeds.
We reiterate that we used the exact same hyper-parameters for all agents to ensure a sound comparison.
Our Pro agents have a single additional hyper-parameter ˜ c .
We did a minimal random search ongames to tune ˜ c .
Figure 2 visualizes the performance of Pro agents as a function of ˜ c .
In light of this result, we set ˜ c
= 0 .
2 for DQN Pro and ˜ c
= 0 .
05 for Rainbow Pro.
We used these values of ˜ c for all 55 games, and note that we performed no further hyper-parameter tuning at all.
5.2.2 Results
The first question is whether endowing the DQN agent with the proximal term can yield significant improvements over the original DQN.
Figure 3 (top) shows a comparison between DQN and DQN
Pro in terms of the final performance.
In particular, following standard practice (Wang et al., 2016;
Dabney et al., 2018; van Seijen et al., 2019), for each game we compute:
[Equation]
Bars shown in red indicate the games in which we observed better final performance for DQN Pro relative to DQN, and bars in blue indicate the opposite.
The height of a bar denotes the magnitude of this improvement for the corresponding benchmark; notice that the y-axis is scaled logarithmically.
We took human and random scores from previous work (Nair et al., 2015; Dabney et al., 2018).
It is clear that DQN Pro dramatically improves upon DQN.
We defer to the Appendix for full learning curves on all games tested.
Can we fruitfully combine the proximal term with some of the existing algorithmic improvements in DQN?
To answer this question, we build on the Rainbow algorithm of Hessel et al.
(2018) who successfully combined numerous important algorithmic ideas in the value-based RL literature.
We present this result in Figure 3 (bottom).
Observe that the overall trend is for Rainbow Pro to yield large performance improvements over Rainbow.
Additionally, we measured the performance of our agents relative to human players.
To this end, and again following previous work (Wang et al., 2016; Dabney et al., 2018; van Seijen et al., 2019), for each agent we compute the human-normalized score:
[Equation]
In Figure 4 (left), we show the median of this score for all agents, which Wang et al.
(2016) and Hessel et al.
(2018) argued is a sensible quantity to track.
We also show per-game learning curves with standard error in the Appendix.
H e r o
D e m o n A tt a c k
J o u r n e y E
s c a p e A ss a u l t C a r n i v a l F r ee w a y K r u ll P
o n g
A li e n B o w li n g
R o b o t a n k
Y a r s R e v e n g e
C h
o
pp e r C o mm a n d
M o n t e z u m a R e v e n g e A s t e r o i d s B e r z
e
r
k
A t l a n t i s S
o l a r i s
C r a z y C li m b e r
M s P a c m a n
B a tt l e Z o n e P
i
t
f a ll
F r o s t b i t e
R o a d R unn e r
T u t a n k h a m
P r i v a t e E y e
R i v e rr a i d Q b e r t P h
o e n i x V e n t u r e
B a n k H
e
i s t
A m i d a r G r a v
i t a r
K un g F u M a s t e r
F i s h
i n g D e r b y
S e a q u e s t G o p h
e r
I c e H o c k e y
A s t e r i x
B e a m R i
d
e r
Z a xx o n P oo y a n
V i
d e o P i n b a ll
A i r R a i d
U p N D
o w n
J a m e s b o n d
B r e a k o u t B o x i n g
S t a r G unn e r
Sp a c e I n v a d e r s
K a n g a r oo E n d u r o
W i z a r d O f
W o r
T i m e P il o t
N a m e T h i s
G a m e - 50% -
[Table]
G r a v i t a r
B a n k H
e
i s t
P r i v a t e E y e
M o n t e z u m a R e v e n g e H e r o P i t
f a ll F r o s t b
i t e A
m i
d a r A t l a n t
i s
K a n g a r oo B o w li n g
C h
o
pp e r C o mm a n d K r u ll
R i v e rr a i d
J a m e s b o n d
V e n t
u
r e
J o u r n e y E
s c a p e C a r n i v a l S
o l a r i s
A s t e r o i d s
F i s h
i n g D e r b y
B a tt l e Z o n e
B o x i n g
V i
d e o P i n b a ll
M s P a c m a n Q b e r t
F r ee w a y
U p N D
o w n E n d u r o R
o b
o t a n k
R o a d R unn e r
T u t a n k h a m
C r a z y C li m b e r
B r e a k o u t P oo y a n B e r z e r k A li e n
Y a r s R e v e n g e P o n g
T i m e P il o t
A ss a u l t
I c e H o c k e y
K un g F u M a s t e r Z a xx o n
D e m o n A tt a c k
N a m e T h i s
G a m e
S t a r G unn e r
B e a m R i
d
e r
S e a q u e s t
A i r R a i d
[Table]
o p h e r
W i z a r
[Figure]
We make two key observations from this figure.
First, the very basic DQN Pro agent is capable of achieving human-level performance (1.0 on the y-axis) after 120 million frames.
Second, the
Rainbow Pro agent achieves 220 percent human-normalized score after only 120 million frames.
5.2.3 Additional Experiments
Our purpose in endowing the agent with the proximal term was to keep the online network in the vicinity of the target network, so it would be natural to ask if this desirable property can manifest itself in practice when using the proximal term.
In Figure 4, we answer this question affirmatively by plotting the magnitude of the update to the target network during synchronization.
Notice that we periodically synchronize online and target networks, so the proximity of the online and target network should manifest itself in a low distance between two consecutive target networks.
Indeed, the results demonstrate the success of the proximal term in terms of obtaining the desired proximity of online and target networks.
While using the proximal term leads to significant improvements, one may still wonder if the advantage of DQN Pro over DQN is merely stemming from a poorly-chosen period hyper-parameter in the original DQN, as opposed to a truly more stable optimization in DQN Pro.
To refute this hypothesis, we ran DQN with various settings of the period hyper-parameter { 2000 , 4000 , 8000 , 12000 } .
This set included the default value of the hyper-parameter (8000) from the original paper (Mnih et al., 2015), but also covered a wider set of settings.
Additionally, we tried an alternative update strategy for the target network, referred to as Polyak averaging, which was popularized in the context of continuous-action RL (Lillicrap et al., 2015): θ ← τw + (1 − τ ) θ .
For this update strategy, too, we tried different settings of the τ hyper-parameter, namely { 0 . 05 , 0 .
005 , 0 . 0005 } , which includes the value 0 .
005 used in numerous papers (Lillicrap et al., 2015; Fujimoto et al., 2018; Asadi et al., 2021).
Figure 5 presents a comparison between DQN Pro and DQN with periodic and Polyak target updates for various hyper-parameter settings of period and τ .
It is clear that DQN Pro is consistently
[Figure]
outperforming the two alternatives regardless of the specific values of period and τ , thus clearly demonstrating that the improvement is stemming from a more stable optimization procedure leading to a better interplay between the two networks.
Finally, an alternative approach to ensuring lower distance between the online and the target network is to anneal the step size based on the number of updates performed on the online network since the last online-target synchronization.
In this case we performed this experiment in 4 games where we knew proximal updates provide improvements based on our DQN Pro versus DQN resulst in
Figure 3.
In this case we linearly decreased the step size from the original DQN learning rate α to α ′ ≪
α where we tuned α ′ using random search.
Annealing indeed improves DQN, but DQN
Pro outperforms the improved version of DQN.
Our intuition is that Pro agents only perform small updates when the target network is far from the online network, but naively decaying the learning rate can harm progress when the two networks are in vicinity of each other.
Discussion
In our experience using proximal updates in the parameter space were far superior than proximal updates in the value space.
We believe this is because the parameter-space definition can enforce the proximity globally, while in the value space one can only hope to obtain proximity locally and on a batch of samples.
One may hope to use natural gradients to enforce value-space proximity in a more principled way, but doing so usually requires significantly more computational resources Knight &
Lerner (2018).
This is in contrast to our proximal updates which add negligible computational cost in the simple form of taking a dimension-wise weighted average of two weight vectors.
In addition, for a smooth (Lipschitz) Q function, performing parameter-space regularization guarantees function-space regularization.
Concretely: ∀ s, ∀ a | Q ( s, a ; θ ) − Q ( s, a ; θ ′ )
| ≤ L || θ − θ ′ || , where L is the Lipschitz constant of Q .
Moreover, deep networks are Lipschitz (Neyshabur et al., 2015;
Asadi et al., 2018), because they are constructed using compositions of Lipschitz functions (such as ReLU, convolutions, etc) and that composition of Lipschitz functions is Lipschitz.
So performing value-space updates may be an overkill.
Lipschitz property of deep networks has successfully been leveraged in other contexts, such as in generative adversarial training Arjovsky et al.
A key selling point of our result is simplicity, because simple results are easy to understand, implement, and reproduce.
We obtained significant performance improvements by adding just a few lines of codes to the publicly available implementations of DQN and Rainbow Castro et al.
Related Work
The introduction of proximal operators could be traced back to the seminal work of Moreau (1962, 1965), Martinet (1970) and Rockafellar (1976), and the use of the proximal operators has since expanded into many areas of science such as signal processing (Combettes & Pesquet, 2009), statistics and machine learning (Beck & Teboulle, 2009;
Polson et al., 2015; Reddi et al., 2015), and convex optimization (Parikh & Boyd, 2014; Bertsekas, 2011b,a).
In the context of RL, Mahadevan et al.
(2014) introduced a proximal theory for deriving convergent off-policy algorithms with linear function approximation.
One intriguing characteristic of their work is that they perform updates in primal-dual space, a property that was leveraged in sample complexity analysis (Liu et al., 2020) for the proximal counterparts of the gradient temporal-difference algorithm (Sutton et al., 2008).
Proximal operators also have appeared in the deep RL literature.
For instance, Fakoor et al.
(2020b) used proximal operators for meta learning, and Maggipinto et al. (2020) improved TD3 (Fujimoto et al., 2018) by employing a stochastic proximal-point interpretation.
The effect of the proximal term in our work is reminiscent of the use of trust regions in policy-gradient algorithms (Schulman et al., 2015, 2017;
Wang et al., 2019; Fakoor et al., 2020a; Tomar et al., 2021).
However, three factors differentiate our work: we define the proximal term using the value function, not the policy, we enforce the proximal term in the parameter space, as opposed to the function space, and we use the target network as the previous iterate in our proximal definition.
Conclusion and Future work
We showed a clear advantage for using proximal terms to perform slower but more effective updates in approximate planning and reinforcement learning.
Our results demonstrated that proximal updates lead to more robustness with respect to noise.
Several improvements to proximal methods exist, such as the acceleration algorithm (Nesterov, 1983; Li & Lin, 2015), as well as using other proximal terms (Combettes & Pesquet, 2009), which we leave for future work.  Acknowledgment We thank Lihong Li, Pratik Chaudhari, and Shoham Sabach for their valuable insights in different stages of this work.
Appendix 10.1 Pseudo-code for DQN Pro
Below, we present the pseudo-code for DQN Pro.
Notice that the difference between DQN and DQN
Pro is minimal (highlighted in gray).
1: Initialize θ , N, period , replay buffer D , α, and ˜ c 2: s ← env.reset(), w ← θ , numUpdates ←3: repeat 4: a ∼ ϵ -greedy (cid:0) Q ( s, · ; w ) (cid:1) 5: s ′ , r ← env.step( s, a ) 6: add ⟨ s, a, r, s ′ ⟩ to D 7: if s ′ is terminal then 8: s ← env.reset() 9: end if 10: for n in {1, ..., N} do 11: sample B = {⟨ s, a, r, s ′ ⟩} , compute ∇ w h ( w ) 12: w ← (cid:0) 1 − ( α/ ˜ c ) (cid:1) w + ( α/ ˜ c ) θ − α ∇ w h ( w ) 13:
{+numUpdates ← numUpdates +14: if numUpdates % period = 
+}
then 15: θ ← w 16: end if 17: end for 18: until convergence
10.2 Implementation Details
Table 1 and 2 show hyper-parameters, computing infrastructure, and libraries used for the experiments in this paper for all games tested.
Our training and evaluation protocols and the hyper-parameter settings closely follow those of the Dopamine baseline.
To report performance results, we measured the undiscounted sum of rewards obtained by the learned policy during evaluation.
[Table]
[Table]
10.3 Proofs
Theorem 1.
The Proximal Bellman Optimality Operator T ⋆c,f is a contraction with fixed point v ⋆ .
We make two assumptions:
f is smooth, or more specifically that its gradient is 1-Lipschitz: ||∇ f ( v 1 ) − ∇ f ( v 2 ) ||
≤
|| v 1 − v 2 || ∀ v 1 , ∀ v 2 .
the value of the parameter c is large, in particular c > 21 − γ .
Proof.
Both terms are convex and differentiable, therefore by setting the gradient to zero, we have:
[Equation]
We can then show:
[Equation]
This implies:
[Equation]
Therefore,
[Equation]
Allowing us to conclude that T ⋆c,f is a contraction (second assumption).
Further, to show that v ⋆ is indeed the fixed point of T ⋆c,f , notice from the original formulation:
[Equation]
that, at point v ∗ setting v ′ = v ⋆ jointly minimizes the first term, because v ⋆ = T ⋆ v ⋆ due to fixedpoint defintion, and it also minimizes the second term because D f ( v ⋆ , v ⋆ )
{+= 
+}
and that Bregman divergence is non-negative.
Therefore, T ⋆c,f v ⋆ = v ⋆ ; v ⋆ is the fixed-point of T ⋆c,f v ⋆ .
Since, T ⋆c,f is a contraction, this fixed point is unique.
Theorem 2.
Consider the PMPI algorithm specified by:
[Equation]
Define the Bellman residual b k := v k − T π k +1 v k , and error terms x k :=
( I − γP π k )
ϵ k
and y
k :=
γP π ∗ ϵ k .
After k steps:
[Equation]
• where d k ≤ γP π ∗ d k − 1 − (cid:0) (1 − β ) y k − 1 + βb k − 1 (cid:1) + (1 − β ) (cid:80) n − 1 j =1 ( γP π k ) j b k − 1 + ϵ ′ k • s k ≤ (cid:0) (1 − β )( γP π k ) n + βI (cid:1) ( I − γP π k ) − 1 b k −• b k ≤ (cid:0) (1 − β )( γP π k ) n + βI (cid:1)
b
k − 1 + (1 − β ) x k + ϵ ′ k
+
We make two assumptions:
we assume ϵ error in policy evaluation step, as already stated in equation (4).
we assume ϵ ′ error in policy greedification step π k ← G ϵ
k
v k − 1 ∀ k .
This means ∀ π T π v k − T π k +1 v k ≤ ϵ ′ k
+1 .
Note that this assumption is orthogonal to the thesis of our paper
[Equation]
but we kept it for generality.
Proof.
Step 0: bound the Bellman residual: b k :=
v k − T π k +1 v k .
b k
[Equation]
(from our assumption
[Equation]
) ( I − γP π k )
ϵ k
(cid:124) (cid:123)(cid:122) (cid:125) x
k + ϵ ′ k
+ = v k − (1 − β )
ϵ k − T π k ( v k − (1 − β )
ϵ k )
+ (1 − β ) x k + ϵ ′ k
+ (from v k − (1 − β )
ϵ k
=
( T π k β ) n v k − 1 ) =
(1 − β )( T π k ) n v k − 1 + βv k − 1 − T π k (cid:0) (1 − β )( T π k ) n v k − 1 + βv k − 1 (cid:1) + (1 − β ) x k + ϵ ′ k
+ (from linearity of T π k ) =
[Equation]
β )(
[Equation]
T
[Equation]
(cid:0) (
[Equation]
β )( T
[Equation]
n v k − 1 (cid:1) + β (cid:0) v k
[Equation]
1 − T
[Equation]
k v k − 1 (cid:1) + (1 − β ) x k + ϵ ′ k
+ = (1 − β ) (cid:16) ( T π k ) n v k − 1 − T π k (cid:0) ( T π k ) n v k − 1 (cid:1)(cid:17)
+ β (cid:0) v k − 1 − T π k v k − 1 (cid:1)
+ (1 − β ) x k + ϵ ′ k
+ = (1 − β ) (cid:16) ( T π k ) n v k − 1 − ( T π k ) n (cid:0) T π k v k − 1 (cid:1)(cid:17) + β (cid:0) v k − 1 − T π k v k − 1 (cid:1)
+ (1 − β ) x k + ϵ ′ k
+ = (1 − β )( γP π k ) n (cid:0) v k − 1 − T π k ( v k −(cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
allowing us to conclude:
[Equation]
1: bound the distance to the optimal value
[Equation]
d k
[Equation]
[Equation]
Additionally we can bound g k +1 as follows:
[Equation]
Allowing us to conclude that:
[Equation]
Step 2: bound the distance between the approximate value and the value of the policy: s k :=
( T π k β ) n
[Equation]
k
[Equation]
[Equation]
Allowing us to conclude that:
[Equation]
Learning curves
We present full learning curves of DQN, DQN Pro, Rainbow, and Rainbow Pro for the 55 Atari games.
All results are averaged over 5 independent seeds.
[Figure]
Motivating Example for Adaptive ˜ c
In this section we specifically look at 4 domains in which Rainbow Pro did significantly better than the original Rainbow, as well 4 domains where Rainbow Pro is underperforming Rainbow.
Note, again, that it is uncommon for Rainbow Pro with the original ˜ c to underperform, but here we have a deeper dive into these cases for a better understanding.
From Figure 11, we observe that by using a slightly larger value of ˜ c , which slightly decreases the incentive for online-target proximity, we can recover from the downside, while still maintaining superior performance on games that are conducive to proximal updates.
This suggest that, while using a fixed ˜ c value is enough to obtain significant performance improvement, adaptively choosing ˜ c would provide us with even more reliable improvements when performing proximal updates.
In this context, a promising idea would be to hinge on the variance of our gradient updates when setting ˜ c .
We leave this promising direction for future work.
Proximal Updates in the Value Space
Our primary contribution was to show the usefulness of performing proximal updates in the parameter space.
That said, we also implemented a version of proximal updates that operated in the value space.
More specifically, in this case we updated the parameters of the online network as follows:
[Equation]
We conducted numerous experiments using variants of this idea (such as using separate replay buffer for each term, performing the update for all actions in buffered states, etc) but we generally found the value-space updates to be ineffective.
As mentioned in the main paper, we believe this is because the parameter-space definition can enforce the proximity globally, while in the value space one can only hope to obtain proximity locally and on a batch of samples.
To perform global updates we may need to compute the natural gradient, which typically requires matrix invasion Knight & Lerner (2018), and thus adding significant computational burden to the original algorithm.
In comparison, the parameter-space version is effective, simple to implement, and capable of enforcing proximity globally due to the Lipschitz property of neural networks.