Faster Deep Reinforcement Learning with Slower Online Network
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep reinforcement learning algorithms often use two networks for value functionoptimization: an online network, and a target network that tracks the online network with some delay.
Using two separate networks enables the agent to hedgeagainst issues that arise when performing bootstrapping.
In this paper we endowtwo popular deep reinforcement learning algorithms, namely DQN and Rainbow,with updates that incentivize the online network to remain in the vicinity of thetarget network.
This improves the robustness of deep reinforcement learning inpresence of noisy updates.
The resultant agents, called DQN Pro and Rainbow Pro,exhibit significant performance improvements over their original counterparts onthe Atari benchmark demonstrating the effectiveness of this simple idea in deepreinforcement learning.
Introduction
An important competency of reinforcement-learning (RL) agents is learning in environments withlarge state spaces like those found in robotics (Kober et al., 2013), dialog systems (Williams et al.,2017), and games (Tesauro, 1994; Silver et al., 2017).
Recent breakthroughs in deep RL havedemonstrated that simple approaches such as Q-learning (Watkins & Dayan, 1992) can surpasshuman-level performance in challenging environments when equipped with deep neural networks forfunction approximation (Mnih et al., 2015).
Two components of a gradient-based deep RL agent are its objective function and optimizationprocedure.
The optimization procedure takes estimates of the gradient of the objective with respect tonetwork parameters and updates the parameters accordingly.
In DQN (Mnih et al., 2015), for example,the objective function is the empirical expectation of the temporal difference (TD) error (Sutton,1988) on a buffered set of environmental interactions (Lin, 1992), and variants of stochastic gradientdescent are employed to best minimize this objective function.
A fundamental difficulty in minimizing this objective function stems from the use of bootstrapping.
In this context, bootstrapping refers to the dependence of the target of updates on the parameters ofthe neural network, which is itself continuously updated during training.
Employing bootstrapping in
RL stands in contrast to supervised-learning techniques and Monte-Carlo RL (Sutton & Barto, 2018),where the target of our gradient updates does not depend on the parameters of the neural network.
Mnih et al.
(2015) proposed a simple approach to hedging against issues that arise when usingbootstrapping, namely to use a target network in value-function optimization.
The target networkis updated periodically, and tracks the online network with some delay.
While this modificationconstituted a major step towards combating misbehavior in Q-learning (Lee & He, 2019; Kim et al.
,2019; Zhang et al., 2021), optimization instability is still prevalent (van Hasselt et al., 2018).
Our primary contribution is to endow DQN and Rainbow (Hessel et al., 2018) agents with a term thatensures the parameters of the online-network component remain in the vicinity of the parametersof the target network.
Our theoretical and empirical results show that this simple modificationcan remarkably increase the robustness of RL to noisy updates without incurring additional computational or memory costs.
In particular, we present comprehensive experiments on the Ataribenchmark (Bellemare et al., 2013) where the new agents yield significant improvements over theiroriginal counterparts, thus revealing the benefits of using this simple technique for deep RL.
Background and Notation
RL is the study of the interaction between an environment and an agent that learns to maximizereward through experience.
The Markov Decision Process (Puterman, 1994), or MDP, is used to mathematically define the RL problem.
An MDP is specified by the tuple ⟨S , A , R , P , γ ⟩ , where S is theset of states and A is the set of actions.
The functions R : S ×
A → R and P : S ×
A × S → [0 , 1]denote the reward and transition dynamics of the MDP.
Finally, by discounting future rewards, γformalizes the intuitive notion that short-term rewards are more valuable than those received later.
The goal in the RL problem is to learn a policy, a mapping from states to a probability distributionover actions, π : S → P ( A ) , that obtains high sums of future discounted rewards.
An importantconcept in RL is the state value function.
Formally, it denotes the expected discounted sum of futurerewards when committing to a policy π in a state s : v π ( s ) :
= E (cid:2) (cid:80) ∞ t =0 γ t R t (cid:12)(cid:12) S
= s, π (cid:3) .
We definethe Bellman operator T π as follows:
[Equation]
s ′ ∈S γ P
s, [Equation] , s ′
[Equation] ) (cid:1) , which we can write compactly as: T π v :=
R π + γP π v , where (cid:2) R π (cid:3) ( s ) =
(cid:80) a ∈A π ( a | s ) R ( s, a )and (cid:2) P π v (cid:3) ( s ) =
(cid:80) a ∈A π ( a | s ) (cid:80) s ′ ∈S P ( s, a, s ′ ) v ( s ′ ) .
We also denote: ( T π ) n v :=
T π · · · T π (cid:124) (cid:123)(cid:122) (cid:125) n compositions v .
Notice that v π is the unique fixed-point of ( T π ) n for all natural numbers n , meaning that v π =( T π ) n v π , for all n .
Define v ⋆ as the optimal value of a state, namely: v ⋆ ( s ) :
= max π v π ( s ) , and π ⋆as a policy that achieves v ⋆ ( s ) for all states.
We define the Bellman Optimality Operator T ⋆ :
[Equation]
whose fixed point is v ⋆ .
These operators are at the heart of many planning and RL algorithmsincluding Value Iteration (Bellman, 1957) and Policy Iteration (Howard, 1960).
Proximal Bellman Operator
In this section, we introduce a new class of Bellman operators that ensure that the next iterate in planning and RL remain in the vicinity of the previous iterate.
To this end, we define the Bregman
Divergence generated by a convex function f :
[Equation]
Examples include the l p norm generated by f ( v ) = 12 ∥ v ∥ 2 p and the Mahalanobis Distance generatedby f ( v ) = 12 ⟨ v, Qv ⟩ for a positive semi-definite matrix Q .
We now define the Proximal Bellman Operator ( T πc,f ) n :
[Equation]
where c ∈ (0 , ∞ ] .
Intuitively, this operator encourages the next iterate to be in the proximity of theprevious iterate, while also having a low error relative to the point recommended by the original
Bellman Operator.
The parameter c could, therefore, be thought of as a knob that controls the degreeof gravitation towards the previous iterate.
Our goal is to understand the behavior of Proximal Bellman Operator when used in conjunction withthe Modified Policy Iteration (MPI) algorithm (Puterman, 1994; Scherrer et al., 2015).
Define G v as the greedy policy with respect to v .
At a certain iteration k , Proximal Modified Policy Iteration(PMPI) proceeds as follows:
[Equation]
The pair of updates above generalize existing algorithms.
Notably, with c → ∞ and general n we get
MPI, with c → ∞ and n = 1 the algorithm reduces to Value Iteration, and with c → ∞ and n = ∞we have a reduction to Policy Iteration.
For finite c , the two extremes of n , namely n = 1 and n = ∞ ,could be thought of as the proximal versions of Value Iteration and Policy Iteration, respectively.
To analyze this approach, it is first natural to ask if each iteration of PMPI could be thought of asa contraction so we can get sound and convergent behavior in planning and learning.
For n > 1 ,
Scherrer et al.
(2015) constructed a contrived MDP demonstrating that one iteration of MPI canunfortunately expand.
As PMPI is just a generalization of MPI, the same example from Scherreret al.
(2015) shows that PMPI can expand.
In the case of n = 1 , we can rewrite the pair of equations(2) and (3) in a single update as follows: v k ← T ⋆c,f v k − 1 .
When c → ∞ , standard proofs can beemployed to show that the operator is a contraction (Littman & Szepesvári, 1996).
We now show that
T ⋆c,f is a contraction for finite values of c .
See our appendix for proofs.
Theorem 1.
The Proximal Bellman Optimality Operator T ⋆ c,f is a contraction with fixed point v ⋆ .
Therefore, we get convergent behavior when using T ⋆c,f in planning and RL.
The addition of theproximal term is fortunately not changing the fixed point, thus not negatively affecting the finalsolution.
This could be thought of as a form of regularization that vanishes in the limit; the algorithmconverges to v ⋆ even without decaying 1 /c .
Going back to the general n ≥ 1 case, we cannot show contraction, but following previous work (Bertsekas & Tsitsiklis, 1996; Scherrer et al., 2015), we study error propagation in PMPI in presence of additive noise where we get a noisy sample of the original Bellman Operator ( T π k ) n v k − 1 + ϵ k .
The noise can stem from a variety of reasons, such as function approximation or sampling error.
For simplicity, we restrict the analysis to D f ( v ′ , v ) =
|| v ′ − v || 22 , so we rewrite update (3) as:
[Equation]
which can further be simplified to:
[Equation]
where β
= 11+ c .
This operator is a generalization of the operator proposed by Smirnova & Dohmatob(2020) who focused on the case of n = 1 .
To build some intuition, notice that the update is multiplyingerror ϵ k by a term that is smaller than one, thus better hedging against large noise.
While the updatemay slow progress when there is no noise, it is entirely conceivable that for large enough values of ϵ k ,it is better to use non-zero β values.
In the following theorem we formalize this intuition.
Our resultleans on the theory provided by Scherrer et al. (2015) and could be thought of as a generalization oftheir theorem for non-zero β values.
Theorem 2.
Consider the PMPI algorithm specified by:
[Equation]
Define the Bellman residual
[Equation]
+1 v k , and error terms x k :=
( I − γP π k )
ϵ k
and y
k :=
γP π ∗ ϵ k .
After k steps:
[Equation]
• where d k ≤ γP π ∗ d k − 1 − (cid:0) 1 − β
[Equation]
y k − 1 + βb k −
(cid:1)
+
1 − β
) (cid:80) n
− j =
( γP π k )
j
b k − 1 + ϵ ′ k
• s k ≤ (cid:0) (1 − β )( γP π k ) n + βI (cid:1) ( I − γP π k ) − 1 b k − 1• b k ≤ (cid:0) (1 − β )( γP π k ) n + βI (cid:1)
b k − 1 + (1 − β ) x k + ϵ ′ k +
The bound provides intuition as to how the proximal Bellman Operator can accelerate convergence inthe presence of high noise.
For simplicity, we will only analyze the effect of the ϵ noise term, andignore the ϵ ′ term.
We first look at the Bellman residual, b k .
Given the Bellman residual in iterationk − 1 , b k − 1 , the only influence of the noise term ϵ k on b k is through the (1 − β ) x k , term, and we seethat b k decreases linearly with larger β .
The analysis of s k is slightly more involved but follows similar logic.
The bound for s k can bedecomposed into a term proportional to b k − 1 and a term proportional to βb k − 1 , where both aremultiplied with positive semi-definite matrices.
Since b k − 1 itself linearly decreases with β , weconclude that larger β decreases the bound quadratically.
The effect of β on the bound for d k is more complex.
The terms βy k − 1 and (cid:80) n − 1 j =1 ( γP π k )
j b k − 1introduce a linear decrease of the bound on d k with β , while the term β ( I − (cid:80) n − 1 j =1 ( γP π k ) j ) b k − 1introduces a quadratic dependence whose curvature depends on I − (cid:80) n − 1 j =1 ( γP π k ) j .
This complexdependence on β highlights the trade-off between noise reduction and magnitude of updates.
Tounderstand this trade-off better, we examine two extreme cases for the magnitude on the noise.
Whenthe noise is very large, we may set β = 1 , equivalent to an infinitely strong proximal term.
It is easyto see that for β = 1 , the values of d k and s k remain unchanged, which is preferable to the increasethey would suffer in the presence of very large noise.
On the other extreme, when no noise is present,the x k and y k terms in Theorem 2 vanish, and the bounds on d k and s k can be minimized by settingβ = 0 , i.e. without noise the proximal term should not be used and the original Bellman updateperformed.
Intermediate noise magnitudes thus require a value of β that balances the noise reductionand update size.
Deep Q-Network with Proximal Updates
Leveraging the insights we built in the previous section, we now endow the DQN-style family ofalgorithms with proximal updates.
Recall that DQN employs an online network (cid:98) Q ( s, a ; w ) , and atarget network (cid:98) Q ( s, a ; θ ) that is used for bootstrapping and is synchronized with the online networkperiodically.
Let tuples ⟨ s, a, r, s ′ ⟩ denote the buffered environmental interactions of the RL agent.
Define the following objective function like that used in DQN:
[Equation]
By drawing an analogy between the target network and the previous iterate in PMPI, we define the following proximal update for online weights:
[Equation]
To solve this problem, we take multiple descent steps using the stochastic gradients of the objective.
Thus, starting each iteration with w = θ , we perform multiple w updates (specified by the periodhyper-parameter)
w ← w − α (cid:0) ∇ h ( w ) + 1˜ c ( w − θ ) (cid:1) .
To end the iteration, akin to the original DQN,we synchronize the two networks by performing θ ← w , before moving to the subsequent iteration.
Observe that the online-network update can equivalently be written as: w ← (cid:0) 1 − ( α/ ˜ c ) (cid:1) · w +( α/ ˜ c ) · θ − α ∇ h ( w ) .
Notice the intuitively appealing form of the update: we first compute a convexcombination of θ and w , based on the hyper-parameters α and ˜ c , then add the gradient term to arriveat the next iterate of w .
If w and θ are close, the convex combination is close to w itself and so this
DQN Pro update would in effect perform an update similar to that of the original DQN.
However,when the online weight w strays too far from the previous target-network iterate, taking the convexcombination ensures that the online network gravitates towards the target network by default.
In otherwords, the gradient signal from minimizing the squared TD error (6) needs to be strong enough tocompensate for the default gravitation towards θ .
The update includes the original DQN update as aspecial case when ˜ c → ∞ .
The pseudo-code for DQN Pro is presented in the Appendix.
The difference between DQN (Mnihet al., 2015) and our DQN Pro is minimal, and is highlighted in gray.
Note that DQN Pro addsnegligible computational cost to DQN.
While the particular form of DQN Pro presented in the
Appendix uses SGD as the optimizer, the idea can be used in conjunction with other optimizers suchas Adam (Kingma & Ba, 2015).
Experiments
In this section, we empirically investigate the effectiveness of introducing the proximal term toplanning and reinforcement-learning algorithms.
We begin by conducting experiments with PMPI inthe context of approximate planning, and then move to large-scale RL experiments in Atari.
5.1 PMPI Experiments
We now focus on understanding the empirical impact of adding the proximal term on the performanceof approximate PMPI.
To this end, we use the pair of update equations:
[Equation]
For this experiment, we chose the toy 8 × 8 Frozen Lake environment from Open AI
Gym (Brockmanet al., 2016), where the transition and reward model of the environment is available to the planner.
Using a small environment allows us to understand the impact of the proximal term in the simplestand most clear setting.
Note also that we arranged the experiment so that the policy greedificationstep G v k − 1 ∀ k is error-free, so we can solely focus on the interplay between the proximal term andthe error caused by imperfect policy evaluation.
We applied 100 iterations of PMPI, then measured the quality of the resultant policy π :=
π 100 asdefined by the distance between its true value and that of the optimal policy, namely ∥ V ⋆ − V π ∥ ∞ .
We repeated the experiment with different magnitudes of error, as well as different values of the βparameter.
[Figure]
From Figure 1, it is clear that the final performance exhibits a U-shape with respect to the parameterβ .
It is also noticable that the best-performing β is shifting to the right side (larger values) as weincrease the magnitude of noise.
This trend makes sense, and is consistent with what is predicted by
Theorem 2: As the noise level rises, we have more incentive to use larger (but not too large) β valuesto hedge against it.
5.2 Atari Experiments
In this section, we evaluate the proximal (or Pro) agents relative to their original DQN-style counterparts on the Atari benchmark (Bellemare et al., 2013), and show that endowing the agent with theproximal term can lead into significant improvements in the interim as well as in the final performance.
We next investigate the utility of our proposed proximal term through further experiments.
Please seethe Appendix for a complete description of our experimental pipeline.
5.2.1 Setup
[Figure]
We used 55 Atari games (Bellemare et al., 2013) to conduct our experimental evaluations.
Following
Machado et al.
(2018) and Castro et al.
(2018), we used sticky actions to inject stochasticity into theotherwise deterministic Atari emulator.
Our training and evaluation protocols and the hyper-parameter settings follow those of the Dopaminebaseline (Castro et al., 2018).
To report performance, we measured the undiscounted sum ofrewards obtained by the learned policy during evaluation.
We further report the learning curvefor all experiments averaged across 5 random seeds.
We reiterate that we used the exact samehyper-parameters for all agents to ensure a sound comparison.
Our Pro agents have a single additional hyper-parameter ˜ c .
We did a minimal random search on 6games to tune ˜ c .
Figure 2 visualizes the performance of Pro agents as a function of ˜ c .
In light of thisresult, we set ˜ c
= 0 .
2 for DQN Pro and ˜ c
= 0 .
05 for Rainbow Pro.
We used these values of ˜ c for all55 games, and note that we performed no further hyper-parameter tuning at all.
5.2.2 Results
The first question is whether endowing the DQN agent with the proximal term can yield significant improvements over the original DQN.
Figure 3 (top) shows a comparison between DQN and DQN
Pro in terms of the final performance.
In particular, following standard practice (Wang et al., 2016;
Dabney et al., 2018; van Seijen et al., 2019), for each game we compute:
[Equation]
Bars shown in red indicate the games in which we observed better final performance for DQN Prorelative to DQN, and bars in blue indicate the opposite.
The height of a bar denotes the magnitude ofthis improvement for the corresponding benchmark; notice that the y-axis is scaled logarithmically.
We took human and random scores from previous work (Nair et al., 2015; Dabney et al., 2018).
It isclear that DQN Pro dramatically improves upon DQN.
We defer to the Appendix for full learningcurves on all games tested. Can we fruitfully combine the proximal term with some of the existing algorithmic improvementsin DQN?
To answer this question, we build on the Rainbow algorithm of Hessel et al.
(2018) whosuccessfully combined numerous important algorithmic ideas in the value-based RL literature.
Wepresent this result in Figure 3 (bottom).
Observe that the overall trend is for Rainbow Pro to yieldlarge performance improvements over Rainbow.
Additionally, we measured the performance of our agents relative to human players.
To this end, and again following previous work (Wang et al., 2016; Dabney et al., 2018; van Seijen et al., 2019), for each agent we compute the human-normalized score: [Figure] [Equation] [Figure]
[Equation]
In Figure 4 (left), we show the median of this score for all agents, which Wang et al.
and Hesselet al.
(2018) argued is a sensible quantity to track.
We also show per-game learning curves withstandard error in the Appendix.
H e r o
D e m o n A tt a c k
J o u r n e y E
s c a p e A ss a u l t C a r n i v a l F r ee w a y K r u ll P
o n g
A li e n B o w li n g
R o b o t a n k
Y a r s R e v e n g e
C h
o
pp e r C o mm a n d
M o n t e z u m a R e v e n g e A s t e r o i d s B e r z
e
r
k
A t l a n t i s S
o l a r i s
C r a z y C li m b e r
M s P a c m a n
B a tt l e Z o n e P
i
t
f a ll
F r o s t b i t e
R o a d R unn e r
T u t a n k h a m
P r i v a t e E y e
R i v e rr a i d Q b e r t P h
o e n i x V e n t u r e
B a n k H
e
i s t
A m i d a r G r a v
i t a r
K un g F u M a s t e r
F i s h
i n g D e r b y
S e a q u e s t G o p h
e r
I c e H o c k e y
A s t e r i x
B e a m R i
d
e r
Z a xx o n P oo y a n
V i
d e o P i n b a ll
A i r R a i d
U p N D
o w n
J a m e s b o n d
B r e a k o u t B o x i n g
S t a r G unn e r
Sp a c e I n v a d e r s
K a n g a r oo E n d u r o
W i z a r d O f
W o r
T i m e P il o t
N a m e T h i s
G a m e - 50% -
[Table]
G r a v i t a r
B a n k H
e
i s t
P r i v a t e E y e
M o n t e z u m a R e v e n g e H e r o P i t
f a ll F r o s t b
i t e A
m i
d a r A t l a n t
i s
K a n g a r oo B o w li n g
C h
o
pp e r C o mm a n d K r u ll
R i v e rr a i d
J a m e s b o n d
V e n t
u
r e
J o u r n e y E
s c a p e C a r n i v a l S
o l a r i s
A s t e r o i d s
F i s h
i n g D e r b y
B a tt l e Z o n e
B o x i n g
V i
d e o P i n b a ll
M s P a c m a n Q b e r t
F r ee w a y
U p N D
o w n E n d u r o R
o b
o t a n k
R o a d R unn e r
T u t a n k h a m
C r a z y C li m b e r
B r e a k o u t P oo y a n B e r z e r k A li e n
Y a r s R e v e n g e P o n g
T i m e P il o t
A ss a u l t
I c e H o c k e y
K un g F u M a s t e r Z a xx o n
D e m o n A tt a c k
N a m e T h i s
G a m e
S t a r G unn e r
B e a m R i
d
e r
S e a q u e s t
A i r R a i d
[Table]
o p h e r
W i z a r
[Figure]
i
[Figure]
We make two key observations from this figure.
First, the very basic DQN Pro agent is capableof achieving human-level performance (1.0 on the y-axis) after 120 million frames.
Second, the
Rainbow Pro agent achieves 220 percent human-normalized score after only 120 million frames.
5.2.3 Additional Experiments
Our purpose in endowing the agent with the proximal term was to keep the online network in thevicinity of the target network, so it would be natural to ask if this desirable property can manifestitself in practice when using the proximal term.
In Figure 4, we answer this question affirmativelyby plotting the magnitude of the update to the target network during synchronization.
Notice thatwe periodically synchronize online and target networks, so the proximity of the online and targetnetwork should manifest itself in a low distance between two consecutive target networks.
Indeed,the results demonstrate the success of the proximal term in terms of obtaining the desired proximityof online and target networks.
While using the proximal term leads to significant improvements, one may still wonder if the advantage of DQN Pro over DQN is merely stemming from a poorly-chosen period hyper-parameter in theoriginal DQN, as opposed to a truly more stable optimization in DQN Pro.
To refute this hypothesis,we ran DQN with various settings of the period hyper-parameter { 2000 , 4000 , 8000 , 12000 } .
Thisset included the default value of the hyper-parameter (8000) from the original paper (Mnih et al.,2015), but also covered a wider set of settings.
Additionally, we tried an alternative update strategy for the target network, referred to as Polyakaveraging, which was popularized in the context of continuous-action RL (Lillicrap et al., 2015), andwhich proceeds by updating the target network as follows:
θ ← τw + (1 − τ ) θ .
For this updatestrategy, too, we tried different settings of the τ hyper-parameter, namely { 0 . 05 , 0 .
005 , 0 . 0005 } ,which includes the value 0 .
005 used in numerous papers (Lillicrap et al., 2015; Fujimoto et al., 2018;
Asadi et al., 2021).
[Figure] [Figure]
hu
m a nno r m a li z e d m e d i a n
Rainbow Pro Rainbow
[Figure]
Figure 5 presents a comparison between DQN Pro and DQN with periodic and Polyak target updatesfor various hyper-parameter settings of period and τ .
It is clear that DQN Pro is consistentlyoutperforming the two alternatives regardless of the specific values of period and τ , thus clearlydemonstrating that the improvement is stemming from a more stable optimization procedure leadingto a better interplay between the two networks.
Finally, an alternative approach to ensuring lower distance between the online and the target networkis to anneal the step size based on the number of updates performed on the online network sincethe last online-target synchronization.
In this case we performed this experiment in 4 games wherewe knew proximal updates provide improvements based on our DQN Pro versus DQN resulst in Figure 3.
In this case we linearly decreased the step size from the original DQN learning rate αto α ′ ≪
α where we tuned α ′ using random search.
Annealing indeed improves DQN, but DQN
Pro outperforms the improved version of DQN.
Our intuition is that Pro agents only perform smallupdates when the target network is far from the online network, but naively decaying the learningrate can harm progress when the two networks are in vicinity of each other.
Discussion
In our experience using proximal updates in the parameter space were far superior than proximalupdates in the value space.
We believe this is because the parameter-space definition can enforce the proximity globally, while in the value space one can only hope to obtain proximity locally and on abatch of samples.
One may hope to use natural gradients to enforce value-space proximity in a moreprincipled way, but doing so usually requires significantly more computational resources Knight &
Lerner (2018).
This is in contrast to our proximal updates which add negligible computational cost inthe simple form of taking a dimension-wise weighted average of two weight vectors.
In addition, for a smooth (Lipschitz) Q function, performing parameter-space regularization guarantees function-space regularization.
Concretely: ∀ s, ∀ a | Q ( s, a ; θ ) − Q ( s, a ; θ ′ )
| ≤ L || θ − θ ′ || ,where L is the Lipschitz constant of Q .
Moreover, deep networks are Lipschitz (Neyshabur et al.
,2015; Asadi et al., 2018), because they are constructed using compositions of Lipschitz functions(such as ReLU, convolutions, etc) and that composition of Lipschitz functions is Lipschitz.
So tryingto perform value-space updates may be an overkill in this case.
Lipschitz property of deep networkshas successfully been leveraged in other contexts, such as in generative adversarial training Arjovskyet al.
A key selling point of our result is simplicity, because simple results are easy to understand, implement,and reproduce.
We obtained significant performance improvements by adding just a few lines ofcodes to the publicly available implementations of DQN and Rainbow Castro et al.
Related Work
The introduction of proximal operators could be traced back to the seminal work of Moreau (1962,1965), Martinet (1970) and Rockafellar (1976), and the use of the proximal operators has sinceexpanded into many areas of science such as signal processing (Combettes & Pesquet, 2009),statistics and machine learning (Beck & Teboulle, 2009;
Polson et al., 2015; Reddi et al., 2015), andconvex optimization (Parikh & Boyd, 2014; Bertsekas, 2011b,a).
In the context of RL, Mahadevan et al.
(2014) introduced a proximal theory for deriving convergentoff-policy algorithms with linear function approximation.
One intriguing characteristic of theirwork is that they perform updates in primal-dual space, a property that was leveraged in samplecomplexity analysis (Liu et al., 2020) for the proximal counterparts of the gradient temporal-differencealgorithm (Sutton et al., 2008).
Proximal operators also have appeared in the deep RL literature.
For instance, Fakoor et al.
(2020b) used proximal operators for meta learning, and Maggipinto et al.(2020) improved TD3 (Fujimoto et al., 2018) by employing a stochastic proximal-point interpretation.
The effect of the proximal term in our work is reminiscent of the use of trust regions in policy-gradientalgorithms (Schulman et al., 2015, 2017;
Wang et al., 2019; Fakoor et al., 2020a; Tomar et al., 2021).
However, three factors differentiate our work: we define the proximal term using the value function,not the policy, we enforce the proximal term in the parameter space, as opposed to the function space,and we use the target network as the previous iterate in our proximal definition.
Conclusion and Future work
We showed a clear advantage for using proximal terms to perform slower but more effective updatesin approximate planning and reinforcement learning.
Our results demonstrated that proximal updateslead to more robustness with respect to noise.
Several improvements to proximal methods exist,such as the acceleration algorithm (Nesterov, 1983; Li & Lin, 2015), as well as using other proximalterms (Combettes & Pesquet, 2009), which we leave for future work.