C AN A LT Q L EARN F ASTER : E XPERIMENTS AND T HEORY
Anonymous authors
Paper under double-blind review
A BSTRACT
Differently from the popular Deep Q-Network (DQN) learning, Alternating Qlearning (AltQ) does not fully ﬁt a target Q-function at each iteration, and is generally known to be unstable and inefﬁcient.
Limited applications of AltQ mostly rely on substantially altering the algorithm architecture in order to improve its performance.
Although Adam appears to be a natural solution, its performance in AltQ has rarely been studied before.
In this paper, we ﬁrst provide a solid exploration on how well AltQ performs with Adam.
We then take a further step to improve the implementation by adopting the technique of parameter restart.
More speciﬁcally, the proposed algorithms are tested on a batch of Atari 2600 games and exhibit superior performance than the DQN learning method.
The convergence rate of the slightly modiﬁed version of the proposed algorithms is characterized under the linear function approximation.
To the best of our knowledge, this is the ﬁrst theoretical study on the Adam-type algorithms in Q-learning.
I NTRODUCTION
Q-learning (Watkins & Dayan, 1992) is one of the most important model-free reinforcement learning (RL) problems, which has received considerable research attention in recent years (Bertsekas &
Tsitsiklis, 1996; Even-Dar & Mansour, 2003; Hasselt, 2010; Lu et al., 2018; Achiam et al., 2019).
When the state-action space is large or continuous, parametric approximation of the Q-function is often necessary.
One remarkable success of parametric Q-learning in practice is its combination with deep learning, known as the Deep Q-Network (DQN) learning (Mnih et al., 2013; 2015).
It has been applied to various applications in computer games (Bhatti et al., 2016), trafﬁc control (Arel et al., 2010), recommendation systems (Zheng et al., 2018;
Zhao et al., 2018), chemistry research (Zhou et al., 2017), etc.
Its on-policy continuous variant (Silver et al., 2014) has also led to great achievements in robotics locomotion (Lillicrap et al., 2016).
The DQN algorithm is performed in a nested-loop manner, where the outer loop follows an one-step update of the Q-function (via the empirical Bellman operator for Q-learning), and the inner loop takes a supervised learning process to ﬁt the updated (i.e., target) Q-function with a neural network.
In practice, the inner loop takes a sufﬁciently large number of iterations under certain optimizer (e.g. stochastic gradient descent (SGD) or Adam) to ﬁt the neural network well to the target Q-function.
In contrast, a conventional Q-learning algorithm runs only one SGD step in each inner loop, in which case the overall Q-learning algorithm updates the Q-function and ﬁts the target Q-function alternatively in each iteration.
We refer to such a Q-learning algorithm with alternating updates as Alternating Q -learning (AltQ).
Although signiﬁcantly simpler in the update rule, AltQ is well known to be unstable and have weak performance (Mnih et al., 2016).
This is in part due to the fact that the inner loop does not ﬁt the target Q-function sufﬁciently well.
To ﬁx this issue, Mnih et al.
(2016) proposed a new exploration strategy and asynchronous sampling schemes over parallel computing units (rather than the simple replay sampling in DQN) in order for the AltQ algorithm to achieve comparable or better performance than DQN.
As another alternative, Knight & Lerner (2018) proposed a more involved natural gradient propagation for AltQ to improve the performance.
All these schemes require more sophisticated designs or hardware support, which may place AltQ less advantageous compared to the popular DQN, even with their better performances.
This motivates us to ask the following ﬁrst question.
• Q1:
Can we design a simple and easy variant of the AltQ algorithm, which uses as simple setup as DQN and does not introduce extra computational burden and heuristics, but still achieves better and more stable performance than DQN ?
In this paper, we provide an afﬁrmative answer by introducing novel lightweight designs to AltQ based on Adam.
Although Adam appears to be a natural tool, its performance in AltQ has rarely been studied yet.
Thus, we ﬁrst provide a solid exploration on how well AltQ performs with Adam (Kingma & Ba
where
the algorithm
is
referred to as AltQ-Adam.
We then take a further step
to improve the implementation of AltQ-Adam by adopting the technique of parameter restart (i
e., restart the initial setting of Adam parameters every a few iterations)
and refer to the new algorithm as AltQ-AdamR.
This is the ﬁrst time that restart is applied for improving the performance of RL algorithms although restart has been used for conventional optimization before.
In a batch of 23 Atari 2600 games, our experiments show that both AltQ-Adam and AltQ-AdamR outperform the baseline performance of DQN by 50% on average.
Furthermore, AltQ-AdamR effectively reduces the performance variance and achieves a much more stable learning process.
In our experiments for the linear quadratic regulator (LQR) problems, AltQ-AdamR converges even faster than the model-based value iteration (VI) solution.
This is a rather surprising result given that the model-based VI has been treated as the performance upper bound for the Q-learning (including DQN) algorithms with target update (Lewis & Vrabie, 2009; Yang et al., 2019).
Regarding the theoretical analysis of AltQ algorithms, their convergence guarantee has been extensively studied (Melo et al., 2008; Chen et al., 2019b).
More references are given in Section 1.1.
However, all the existing studies focus on the AltQ algorithms that take a simple SGD step.
Such theory is not applicable to the proposed AltQ-Adam and AltQ-AdamR that implement the Adam-type update.
Thus, the second intriguing question we address here is described as follows.
Q2:
Can we
provide
the convergence
guarantee
for
AltQ
Adam
and
AltQ
-AdamR
or
their
slightly
modiﬁed
variants (if
these two algorithms
do not always converge
by
nature)?
It is well known in optimization that Adam does not always converge, and instead, a slightly modiﬁed variant AMSGrad proposed in Reddi et al.
(2018) has been widely accepted as an alternative to justify the performance of Adam-type algorithms.
Thus, our theoretical analysis here also focuses on such slightly modiﬁed variants AltQ-AMSGrad and AltQ-AMSGradR of the proposed algorithms.
We show that under the linear function approximation (which is the structure that the current tools for analysis of Q-learning can handle), both AltQ-AMSGrad and
AltQ-AMSGradR converge to the global optimal solution under standard assumptions for Qlearning.
To the best of our knowledge, this is the ﬁrst non-asymptotic convergence guarantee on Q-learning that incorporates Adam-type update and momentum restart.
Furthermore, a slight adaptation of our proof provides the convergence rate for the AMSGrad for conventional strongly convex optimization which has not been studied before and can be of independent interest.
Notations We use (cid:107) x
(cid:107) :
= (cid:107) x (cid:107) 2 = √ x T x to denote the (cid:96) 2 norm of a vector x , and use (cid:107) x (cid:107) ∞ =
max i | x i | to denote the inﬁnity norm.
When x, y are both vectors, x/y, xy, x 2 , √ x are all calculated in the element-wise manner, which will be used in the update of Adam and AMSGrad.
We denote [ n ] = 1 , 2 , . . . , n , and (cid:98) x (cid:99)
∈
Z as the largest integer such that (cid:98) x (cid:99) ≤ x < (cid:98) x
(cid:99)
+ 1 .
1. R ELATED W ORK
Empirical performance of AltQ: AltQ algorithms that strictly follow the alternating updates are rarely used in practice, particularly in comparison with the well-accepted DQN learning and its improved variants of dueling network structure (Wang et al., 2016), double Q-learning (Hasselt, 2010) and variance exploration and sampling schemes (Schaul et al., 2015).
Mnih et al.
(2016) proposed the asynchronous one-step Q-learning that is conceptually close to AltQ with competitive performance against DQN.
However, the algorithm still relies on a slowly moving target network like DQN, and the multi-thread learning also complicates the computational setup.
Lu et al.
(2018) studied the problem of value overestimation and proposed the non-delusional Q-learning algorithm that employed the so-called pre-conditioned Q-networks, which is also computationally complex.
Knight & Lerner (2018) proposed a natural gradient propagation for AltQ to improve the performance, where the gradient implementation is complex.
In this paper, we propose two simple and computationally efﬁcient schemes to improve the performance of AltQ.
Theoretical analysis of AltQ: Since proposed in Watkins & Dayan (1992), Q-learning has aroused great interest in theoretic analysis.
The line of theoretic research of AltQ that are most relevant to our study lies in the Q-learning with function approximation.
A large number of works study Q-learning with linear function approximation such as Bertsekas & Tsitsiklis (1996); Devraj & Meyn (2017);
Zou et al.
; Chen et al.
(2019b)
; Du et al.
(2019), to name a few.
More recently, convergence of
AltQ with neural network parameterization was given in Cai et al.
(2019), which exploits the linear structure of neural networks in the overparamterized regime for analysis.
It is worth noting that all the existing analysis of AltQ with function approximation considers the simple SGD update, whereas our analysis in this paper focuses on the more involved Adam-type updates.
Convergence analysis of Adam: Adam was proposed in Kingma & Ba (2014) and has achieved a great success in training deep neural networks.
Kingma & Ba (2014) and Reddi et al.
(2018) provided regret bounds under the online convex optimization framework for Adam and AMSGrad, respectively.
However, Tran et al.
(2019) pointed out errors in the proofs of the previous two papers and corrected them.
Recently, convergence analysis of Adam and AMSGrad in nonconvex optimization was provided in Zou et al.
; Zhou et al.
; Chen et al. (2019a); Phuong & Phong (2019), in which the Adam-type algorithms were guaranteed to converge to a stationary point.
To the best of our knowledge, our study is the ﬁrst convergence analysis of the Adam-type of algorithms for Q-learning.
P RELIMINARIES
We consider a Markov decision process with a considerably large or continuous state space S ⊂ R M and action space
A ⊂ R N with a non-negative bounded reward function R : S ×
A → [0 , R max ] .
We deﬁne U ( s ) ⊂
A as the admissible set of actions at state s , and π : S → A as a feasible stationary policy.
We seek to solve a discrete-time sequential decision problem with γ ∈ (0 , 1) as follows:
[Equation]
subject to s t +1 ∼ P ( ·| s t ,
[Equation]
t )
[Equation]
Let J (cid:63) ( s ) :
=
J π (cid:63) ( s ) be the optimal value function when applying the optimal policy π (cid:63) .
The corresponding optimal Q-function can be deﬁned as
[Equation]
where s (cid:48) ∼ P ( ·| s, a ) and we use the same notation hereafter when no confusion arises.
In other words, Q (cid:63) ( s, a ) is the reward of an agent who starts from state s and takes action a at the ﬁrst step and then follows the optimal policy π (cid:63) thereafter.
2. A LT Q A LGORITHM
This paper focuses on the Alternating Q-learning (AltQ) algorithm that uses a parametric function ˆ
Q ( s, a ; θ ) to approximate the Q-function with a parameter θ of ﬁnite and relatively small dimension.
The update rule of AltQ-learning is given by
[Equation]
[Equation]
where α t is the step size at time t .
It is immediate from the equations that AltQ performs the update by taking one step temporal target update and one step parameter learning in an alternating fashion.
2. DQN A LGORITHM
As DQN is also included in this work for performance comparison.
We recall the update of DQN in the following as reference.
Differently from AltQ, DQN updates the parameters in a nested loop.
Within the t -th inner loop, DQN ﬁrst obtains the target Q-function as in Equation (5), and then uses a neural network to ﬁt the target Q-function by running Y steps of a certain optimization algorithm as Equation (6).
The update rule of DQN is given as follows.
[Equation]
where Optimizer can be SGD or Adam for example, and Equation (6) is thus a supervised learning process with T ˆ Q ( s, a ; θ 0 t )) as the ”supervisor”.
At the t -th outer loop, DQN performs the so-called target update as
[Equation]
In practice, when one of the momentum-based optimizers is adopted for Equation (6), such as Adam, it is only initialized once at the beginning of the ﬁrst inner loop.
The historical gradient terms then accumulate throughout multiple inner loops with different targets.
While this stabilizes the DQN training empirically, it is still lack of theoretical understanding on how the optimizer affects the training with various moving targets.
As we will discuss in detail in Section 5, the analysis of AltQ with Adam can potentially shed light on such ambiguity and inspire future work for this matter.
Note that AltQ and DQN mainly differ at how the Q-function evolves after each step of sampling.
A fair comparison between the algorithms should be made without introducing dramatic difference on gradient propagation (Knight & Lerner, 2018), policy structure, exploration and sampling strategies (Mnih et al., 2016).
In practice, the vanilla AltQ is often slow in convergence and unstable with high variance.
To improve the performance, we propose to incorporate Adam and restart schemes, which are easy to implement and yield improved performance than DQN.
A CCELERATED A LTERNATING Q- LEARNING A LGORITHMS
In this section, we ﬁrst describe how to incorporate Adam to the AltQ algorithm, and then introduce a novel implementation scheme to improve the performance of AltQ with Adam.
AltQ with Adam-type update We propose a new AltQ algorithm with Adam-type update (AltQAdam) as described in Algorithm 1.
Its update is similar to the well-known Adam (Kingma & Ba, 2014).
The iterations evolve by updating the exponentially decaying average of historical gradients ( m t ) and squared historical gradients ( v t ).
The hyper-parameters β 1 , β 2 are used to exponentially decrease the rate of the moving averages.
The difference between Algorithm 1 and the standard Adam in supervised learning is that in AltQ, there is no ﬁxed target to “supervise” the learning process.
The target is always moving along with iteration t , leading to more noisy gradient estimations.
The proposed algorithm sheds new light on the possibility of using Adam to deal with such unique challenge brought by RL.
AltQ-Adam with momentum restart We also introduce the restart technique to AltQ-Adam and propose AltQ-AdamR as Algorithm 2.
Traditional momentum-based algorithms largely depend on the historical gradient direction.
When part of the historical information is incorrect, the estimation error tends to accumulate.
The restart technique can be employed to deal with this issue.
One way to restart the momentum-based methods is to initialize the momentum at some restart iteration.
That is, at restart iteration r , we reset m r , v r , i.e., m r = 0 , v r = 0 , which yields θ r +
= θ r .
It is an intuitive implementation technique to adjust the trajectory from time to time, and can usually help mitigate the aforementioned problem while keeping fast convergence property.
For the implementation, we execute the restart periodically with a period r .
It turns out that the restart technique can signiﬁcantly improve the numerical performance, which can be seen in Section 4.
E MPIRICAL P ERFORMANCE
We empirically evaluate the proposed algorithms in this section.
The linear quadratic regulator (LQR) is a direct numerical demonstration of the convergence analysis under linear function approximation which will be discussed in the next section.
Atari 2600 game (Bellemare et al., 2013; Brockman et al., 2016), a classic benchmark for DQN evaluations, is also used to show the effectiveness of the proposed algorithms for complicated tasks.
In practice, we also make a small adjustment to the proposed algorithms.
That is, we re-scale the loss term of L ( θ t ) :
=
ˆ Q t ( s, a ; θ t ) − T ˆ Q ( s, a ; θ t )
1: Input: η, θ 1 , β 1 , β 2 , (cid:15), γ, m 0 = 0 , v 0 = 0 .
2: for t = 1 , 2 , . . .
, K do 3:
T ˆ Q ( s, a ; θ t ) =
R ( s, a ) + γ max a (cid:48)
ˆ Q ( s (cid:48) , a (cid:48) ; θ t ) 4: g t = (cid:16) ˆ Q t ( s, a ; θ t ) − T ˆ Q ( s, a ; θ t ) (cid:17)
∂∂θ t ˆ Q t ( s, a ; θ t ) 5: m t = (1 − β 1 ) m t − 1 + β 1 g t 6: v t = (1 − β 2 ) v t − 1 + β 2 g 2 t 7: θ t +1 =
θ t − η m t √ v t + (cid:15) 8: end for 9: Output: θ K
1: Input: η, θ 1 , β 1 , β 2 , (cid:15), γ, m 0 = 0 , v 0 = 0 , r .
2: for t = 1 , 2 , . . .
, K do 3: if mod ( t, r ) = 0 then 4: m t = 0 , v t =5: end if 6:
T ˆ Q ( s, a ; θ t ) =
R ( s, a ) + γ max a (cid:48)
ˆ Q ( s (cid:48) , a (cid:48) ; θ t ) 7: g t = (cid:16) ˆ Q t ( s, a ; θ t ) − T ˆ Q ( s, a ; θ t ) (cid:17)
∂∂θ t ˆ Q t ( s, a ; θ t ) 8: m t = (1 − β 1 ) m t − 1 + β 1 g t 9: v t
=
(1 − β 2 ) v t − 1 + β 2 g 2 t 10: θ t +1 =
θ t − η m t √ v t + (cid:15) 11: end for 12: Output: θ K in Equation (4) as ˜ L ( θ t ) =
˜ τ
2 L ( θ t ) with some scaling factor ˜ τ ∈ (0 , 1] , which is beneﬁcial for stabilizing the learning process.
We ﬁnd that in both experiments, AltQ-AdamR outperforms both AltQ-Adam and DQN in terms of convergence speed and variance reduction.
Compared with DQN in the empirical experiments of
Atari games, under the same hyper-parameter settings, AltQ-Adam and AltQ-AdamR improve the performance of DQN by 50% on average.
4. L INEAR Q UADRATIC R EGULATOR
We numerically validate the proposed algorithms through an inﬁnite-horizon discrete-time LQR problem whose background can be found in Appendix A.1.
A typical model-based solution (with known dynamics), known as the discrete-time algebraic Riccati equation (DARE), is adopted to derive the optimal policy u (cid:63)t = − K (cid:63) x t .
The performance of the learning algorithm is then evaluated at each step of iterate t with the Euclidean norm (cid:107) K t − K (cid:63) (cid:107) .
The performance result for each method is averaged over 10 trials with different random seeds.
All algorithms share the same set of random seeds and are initialized with the same θ 0 .
The hyper-parameters of the learning settings are also consistent and further details are shown in Table 1.
Note that for all the implementations, we also adopt the double Q-update (Hasselt, 2010) to help prevent over-estimations of the Q-value.
The performance results are seen in Figure 1.
Here we highlight main observations from the LQR experiments.
• AltQ-AdamR outperforms DARE In ideal cases where data sampling perfectly emulates the system dynamics and the target is accurately learned in each inner loop, DARE for LQR would become equivalent to the DQN-like update if the neural network is replaced with a parameterzied linear function.
In practice, such ideal conditions are difﬁcult to satisfy, and hence the actual Q-learning with target update is usually far slower (in terms of number of steps of target updates) than DARE.
Note that AltQ-AdamR performs signiﬁcantly well and even converges faster than DARE, and thus implies it is faster than the most well-performing Q-learning with target update.
• AltQ-AdamR outperforms AltQ-Adam Overall, under the same batch sampling scheme and restart period, AltQ-AdamR achieves a faster convergence and smaller variance than AltQ-Adam.
[Table] [Table]
[Figure]
[Table]
[Figure]
We apply the proposed AltQ algorithms to more challenging tasks of deep convolutional neural network playing a group of Atari 2600 games.
The particular DQN we train to compare against adopts the dueling network structure (Wang et al., 2016), double Q-learning setup (Van Hasselt et al., 2016), (cid:15) -greedy exploration and experience replay (Mnih et al., 2013).
Adam is also adopted, without momentum restart, as the optimizer for the inner-loop supervised learning process.
AltQ-Adam and
AltQ-AdamR are implemented using the identical setup of network construction, exploration and sampling strategies.
We test all the three algorithms with a batch of 23 Atari games.
The choice of 10 million steps of iteration is a common setup for benchmark experiments with Atari games.
Although this does not guarantee the best performance in comparison with more time-consuming training with 50 million steps or more, it is sufﬁcient to illustrate different performances among the selected methods.
The software infrastructure is based on the baseline implementation of OpenAI.
Selections of the hyperparameters are listed in Table 2.
We summarize the results in Figure 2.
The overall performance is illustrated by ﬁrst normalizing the return of each method with respect to the results obtained from
DQN, and then averaging the performance of all 23 games to obtain the mean return and standard deviation.
Considering we use a smaller buffer size than common practice, DQN is not consistently showing improved return over all tested games.
Therefore, the self-normalized average return of
DQN in Figure 2 is not strictly increasing from 0 to 100% .
Overall, both AltQ-Adam and AltQ-AdamR achieve signiﬁcant improvement in comparison with the DQN results.
While AltQ-Adam is suffering from a higher variance, periodic restart (AltQAdamR) resolves the issue efﬁciently with an on-par performance on average and far smaller variance.
Speciﬁcally, in terms of the maximum average return, AltQ-Adam and AltQ-AdamR perform no worse then DQN on 17 and 20 games respectively out of the 23 games being evaluated.
C ONVERGENCE A NALYSIS
In this section, we characterize the convergence guarantee for the proposed AltQ-learning algorithms.
Furthermore, like most of the related papers, we focus on convergence analysis under the linear approximation class.
Understanding the analytical behavior in the linear case is an important stepping stone to understand general cases such as deep neural network.
A linear approximation of the
Q-function ˆ Q ( s, a ; θ ) can be written as
[Equation]
where θ ∈ R d , and Φ : S ×
A → R d is a vector function of size d , and the elements of Φ represent the nonlinear kernel (feature) functions.
[Table]
[Table]
M ODIFICATION OF A LGORITHMS
Although Adam has obtained great success as an optimizer in deep learning, it is well known that
Adam by nature is non-convergent even for simple convex loss functions (Reddi et al., 2018).
Instead, a slightly modiﬁed version called AMSGrad (Reddi et al., 2018) is widely used to study the convergence property of the Adam-type algorithms.
Compared with the update rule of Adam, AMSGrad makes the sequence ˆ v t,i increasing along the time step t for each entry i ∈
[ d ] .
Here, we apply the update rule of AMSGrad to the AltQ algorithm and refer to such an algorithm as AltQ-AMSGrad.
Algorithm 3 describes AltQ-AMSGrad in detail, where Π D , ˆ V 1 / 4 t
[Table]
Correspondingly, we introduce AltQ-AMSGradR which applies the same update rule as Algorithm 3, but resets m t , ˆ v t with a period of r , i.e., m t = 0 , ˆ v t = 0 , ∀ t = kr, k
= 1 , 2 , · · · .
: g t = (cid:16) φ T ( s t , a t ) θ t − r ( s t , a t ) − max a (cid:48)
φ T ( s t +1 , a (cid:48) )
θ t (cid:17) φ ( s t , a t ) 5: m t = (1 − β 1 t ) m t − 1 + β 1 t g t 6: v t
=
(1 − β 2 )ˆ v t − 1 + β 2 g 2 t 7: ˆ v t =
max(ˆ v t − 1 , v t ) , ˆ V t =
diag
(ˆ v 1 , . . .
, ˆ v d ) 8: θ t +1 =
Π D , ˆ V 1 / 4 t (cid:16) θ t − α t
ˆ V − 12 t m t (cid:17) 9: end for 10: Output: 1 T (cid:80)
Tt
=1 θ t
5. M AIN RESULTS
Our theoretical analysis here focuses on the slight variants, AltQ-AMSGrad and AltQ-AMSGradR.
Before stating the theorems, we ﬁrst introduce some technical assumptions for our analysis.
Assumption 1.
At each iteration t , the noisy gradient is unbiased and uniformly bounded, i.e. g t = ¯
g t + ξ t with E ξ t = 0 where ¯ g t =
E [ g t ] , and (cid:107) g t (cid:107) <
G ∞ , ∀ t .
Thus (cid:107) g t (cid:107) ∞ <
G ∞ and (cid:107) g t (cid:107) 2 < G 2 ∞ .
Assumption 2.
(Chen et al., 2019b, Lemma 6.7)
The equation ¯ g ( θ ) = 0 has a unique solution θ (cid:63) , which implies that there exists a c > 0 , such that for any θ ∈ R
d we have
[Equation]
Assumption 3.
The domain D ⊂ R d of approximation parameters is a ball originating at θ = 0 with bounded diameter containing θ (cid:63) .
That is, there exists D ∞ , such that (cid:107) θ m − θ n (cid:107) <
D ∞ , ∀ θ m , θ n ∈ D , and θ (cid:63) ∈ D .
Assumption 1 is standard in the theoretical analysis of Adam-type algorithms (Chen et al., 2019a;
Zhou et al., 2018).
Under linear function approximation and given Assumption 3 and bounded r ( · ) ,
Assumption 1 is almost equivalent to the assumption of bounded φ ( · ) which is commonly taken in related RL work (Tsitsiklis & Van Roy, 1997; Bhandari et al., 2018).
Assumption 2 has been proved as a key technical lemma in Chen et al.
(2019b) under certain assumptions.
Such an assumption appears to be the weakest in the existing studies of the theoretic guarantee for Q-learning with function approximation.
We next provide the convergence results of AltQ-AMSGrad and AltQ-AMSGradR under linear function approximation in the following two theorems.
Theorem 1.
(Convergence of AltQ-AMSGrad) Suppose α t = α √ t , β 1 t = β 1 λ t and δ = β 1 /β 2 with δ, λ ∈ (0 , 1) for t = 1 , 2 , . . .
in Algorithm 3.
Given Assumptions 1 ∼ 3, the output of AltQ-AMSGrad satisﬁes:
[Equation]
where B 1 =
[Equation]
In Theorem 1, B 1 , B 2 , B 3 in the bound in Equation (10) are constants and independent of time.
Therefore, under the choice of the stepsize and hyper-parameters in Algorithm 3, AltQ-AMSGrad achieves a convergence rate of O (cid:16) 1 √ T (cid:17) when (cid:80) di
=1 (cid:107) g 1: T,i (cid:107) << √ T which is justiﬁed in Duchi et al.
Remark 1.
Our proof of convergence here has two major differences from that for AMSGrad in Reddi et al. (2018): (a) The two algorithms are quite different.
AltQ-AMSGrad is a Q-learning algorithm alternatively ﬁnding the best policy, whereas AMSGrad is an optimizer for conventional optimization and does not have alternating nature.
(b) Our analysis is on the convergence rate whereas Reddi et al.
(2018) provides regret bound.
In fact, a slight modiﬁcation of our proof also provides the convergence rate of AMSGrad for conventional strongly convex optimization, which can be of independent interest.
Moreover, our proof avoids the theoretical error in the proof in Reddi et al.
(2018) pointed out by Tran et al.
In the following theorem, we provide the convergence result for AltQ-AMSGradR.
Theorem 2.
(Convergence of AltQ-AMSGradR)
Under the same condition of Theorem 1, the output of AltQ-AMSGradR satisﬁes:
[Equation]
where B 1 =
[Equation]
and B [Equation] Theorem 2 indicates that for AltQ-AMSGradR to enjoy a convergence rate of O (cid:16) 1 √ T (cid:17) the restart period r needs to be sufﬁciently large and (cid:80) d i =1 (cid:107) g 1: T,i (cid:107) << √ T .
In practice as demonstrated by the experiments in Section 4, AltQ-AMSGradR typically performs well, not necessarily under the theoretical conditions.
C ONCLUSION
We propose two types of the accelerated AltQ algorithms, and demonstrate their superior performance over the state-of-the-art through a linear quadratic regulator problem and a batch of 23 Atarigames.
Notably, Adam is not the only scheme in the practice for general optimization.
Heavy ball (Ghadimi et al., 2015) and Nesterov (Nesterov, 2013) are also popular momentum-based methods.
When adopting such methods in AltQ-learning for RL problems, however, we tend to observe a less stable learning process than AltQ-Adam.
This is partially caused by the fact that they optimize over a shorter historical horizon of updates than Adam.
Furthermore, the restart scheme provides somewhat remarkable performance in our study.
It is thus of considerable future interest to further investigate the potential of such a scheme.
One possible direction is to develop an adaptive restart mechanism with changing period determined by an appropriately deﬁned signal of restart.
This will potentially relieve the effort in hyper-parameter tuning of ﬁnding a good ﬁxed period.
Supplementary Materials A F URTHER D ETAILS AND R ESULTS ON E XPERIMENTS
We discuss more details on the experiment setup and provide further results that are not included in
Section 4.
A. L INEAR Q UADRATIC R EGULATOR
The linear quadratic regulator (LQR) problem is of great interest for control community where Lewis et al.
applies PQL to both discrete-time problems (Lewis & Vrabie, 2009; Lewis & Vamvoudakis, 2011) and continuous-time problems (Vamvoudakis, 2017; Vrabie et al., 2009).
We empirically validate the proposed algorithms through an inﬁnite-horizon discrete-time LQR problem deﬁned as minimize π
[Equation]
subject to
[Equation]
where u t = π ( x t ) .
A typical model-based solution (with known A and B ) considers the problem backwards in time and iterates a dynamic equation known as the discrete-time algebraic Riccati equation (DARE):
[Equation]
12) with the cost-to-go P being positive deﬁnite.
The optimal policy satisﬁes u (cid:63)t = − K (cid:63) x t with
[Equation]
For experiments, we parameterize a quadratic Q-function with a matrix parameter H in the form of
[Equation]
The corresponding linear policy satisﬁes u = − Kx , and K
= H − 1 uu H ux .
The performance of the learning algorithm is then evaluated at each step of iterate i with the Euclidean norm (cid:107) K i − K (cid:63) (cid:107) 2 .
A. A TARI G AMES
We list detailed experiments of the 23 Atari games evaluated with the proposed algorithms in Figure 3.
All experiments are executed with the same set of two random seeds.
Each task takes about 20-hour of wall-clock time on a GPU instance.
All three methods being evaluated share similar training time.
AltQ-Adam and AltQ-AdamR can be further accelerated in practice with a more memory-efﬁcient implementation considering the target network is not required.
We keep our implementation of proposed algorithms consistent with the DQN we are comparing against.
Other techniques that are not included in this experiment are also compatible with AltQ-Adam and AltQ-AdamR, such like asynchronous exploration (Mnih et al., 2013) and training with decorrelated loss (Mavrin et al., 2019).
Overall, AltQ-Adam signiﬁcantly increases the performance by over 100% in some of the tasks including Asterix, BeamRider, Enduro, Gopher, etc.
However, it also illustrates certain instability with complete failure on Amidar and Assault.
This is mostly caused by the sampling where we are using a relevantly small buffer size with 10% of the common conﬁgured size in Atari games with experience replay.
Notice that those failures tend to appear when the (cid:15) -greedy exploration has evolved to a certain level where the immediate policy is effectively contributing to the accumulated experience.
This potentially ampliﬁes the biased exploration that essentially leads to the observed phenomenon.
Interstingly, AltQ-AdamR that incorporates the restart scheme resolves the problem of high variance of average return brought by AltQ-Adam and provides a more consistent performance across the task domain.
This implies that momentum restart effectively corrects the accumulated error and stabilizes the training process. [Table]
B P ROOF OF T HEOREM
Different from the regret bound for AMSGrad obtained in Reddi et al. (2018), our analysis is on the convergence rate.
In fact, a slight modiﬁcation of our proof also provides the convergence rate for AMSGrad for conventional strongly convex optimization, which can be of independent interest.
Moreover [Table] our proof avoids the theoretical error in the proof in Reddi et al. (2018) pointed out by (Tran et al., 2019).
Before proving the theorems [Table] we ﬁrst provide some useful lemmas [Table]
Lemma 1.
(Zhou et al., 2018, Lemma A.1) Let { g t , m t , ˆ v t } for t = 1 , 2 , . . .
be sequences generated by Algorithm 3 and ¯ g t =
E [ g t ] .
Under Assumption 1, (cid:107) ¯ g t (cid:107) ≤ G ∞ , (cid:107) m t (cid:107) ≤ G ∞ , (cid:107) ˆ v t (cid:107) ≤ G 2 ∞ .
Lemma 2.
(Reddi et al., 2018, Lemma 2) Let { m t , ˆ V t } for t = 1 , 2 , . .
be sequences generated by
Algorithm 3.
Given α t , β 1 t , β 2 as speciﬁed in Theorem 1, we have
[Equation]
Lemma 3.
Let α t = α √ t and β 1 t = β 1 λ t for t = 1 , 2 , . . . .
Then
[Equation]
Proof.
The proof is based on taking the standard sum of geometric sequences.
T
[Equation]
t
[Equation]
With the lemmas above [Equation] we are ready to prove Theorem 1.
Observe that
[Equation]
Clearly Π D , ˆ V 1 / 4 t ( θ (cid:63) )
=
θ (cid:63) due to Assumption 3.
We start from the update of θ t when t ≥ 2 .
[Equation]
[Table]
[Equation]
[-ˆ V − 1 / 4 t m t (cid:13)(cid:13)(cid:13) 
-]
[-+ α t β 1 t (cid:18)α t (cid:13)(cid:13)(cid:13) ˆ V 1 / 4 t ( θ t − θ (cid:63) ) (cid:13)(cid:13)(cid:13) 
-]
where (i) follows from the Cauchy-Schwarz inequality, and (ii) holds because ˆ v t +1 ,i ≥ ˆ v t,i , ∀ t, ∀ i .
Next
[Equation]
we take the expectation over all samples used up to time step t on both sides, which still preserves the inequality.
Since we consider i [Equation] i
[Equation]
d. sampling case, by letting F t be the ﬁltration of all the sampling up to time t , we have
[Equation] [Equation]
Thus we have
[Equation]
[Table]
where (i) follows from Equation (18), (ii) follows due to Assumption 2 and 1 − β 1 t > 0 , (iii) follows from β 1 t < β 1 < 1 and E (cid:107) θ t − θ (cid:63) (cid:107) 2 > 0 , and (iv) follows from (cid:13)(cid:13)(cid:13)
ˆ V 1 / 4 t ( θ t − θ (cid:63) ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)(cid:13) ˆ V 1 / 4 t (cid:13)(cid:13)(cid:13)2 (cid:107) θ t − θ (cid:63) (cid:107) 2 ≤ G ∞ D 2 ∞ by Lemma 1 and Assumption 3.
We note that (iii) is the key step to avoid the error in the proof in Reddi et al. (2018), where we can directly bound 1 − β 1 t , which is impossible in Reddi et al.
By rearranging the terms in the above inequality and taking the summation over time steps
[Equation]
we have
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] where (i) follows from α t < α t − 1 .
With further adjustment of the ﬁrst term in the right hand side of the last inequality
[Equation] we can then bound the sum as [Equation] t ( θ t − θ (cid:63) ) (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) ˆ V 1 / 4 t ( θ t +1 − θ (cid:63) ) (cid:13)(cid:13)(cid:13) 2 (cid:19) +
T (cid:88) t = β 1 t G ∞ D 2 ∞ α t + (1 + β 1 ) T (cid:88) t = α t E (cid:13)(cid:13)(cid:13) ˆ V − 1 / 4 t m t (cid:13)(cid:13)(cid:13)= [Equation]
(cid:13)(cid:13)(cid:13) ˆ V 1 / 4( θ 2 − θ (cid:63) ) (cid:13)(cid:13)(cid:13) [Equation] +
[Equation] (cid:88) t = E [Equation]
 (cid:13)(cid:13)(cid:13) ˆ V 1 / 4 t ( θ t − θ (cid:63) ) (cid:13)(cid:13)(cid:13)α t
[Equation]
(cid:13)(cid:13)(cid:13) ˆ V 1 / 4 t − 1 ( θ t − θ (cid:63) ) (cid:13)(cid:13)(cid:13)α t
[Equation]
 
[Equation]
(cid:13)(cid:13)(cid:13) ˆ V 1 / 4 T ( θ T
[Equation]
[Equation]
θ (cid:63) ) (cid:13) (cid:13)(cid:13)α T +
[Equation]
(cid:88)
[Equation]
β
[Equation]
G
[Equation] D [Equation] ∞
α t
[Equation] [Equation] β 1 ) T (cid:88) t = α t E (cid:13)(cid:13)(cid:13)
ˆ V − 1 / 4 t m t (cid:13)(cid:13)(cid:13)= E (cid:13)(cid:13)(cid:13) ˆ V 1 / 4( θ 2 − θ (cid:63) ) (cid:13)(cid:13)(cid:13)α+ T (cid:88) t =
E (cid:32)(cid:80) di
=1 ˆ v 1 / 2 t,i ( θ t,i − θ (cid:63)i )α t − (cid:80) di
=1 ˆ v 1 / 2 t − 1 ,i ( θ t,i − θ (cid:63)i )α t −(cid:33) −
E (cid:13)(cid:13)(cid:13) ˆ V 1 / 4 T ( θ T +1 − θ (cid:63) ) (cid:13)(cid:13)(cid:13)α T +
T (cid:88) t = β 1 t G ∞ D 2 ∞ α t + (1 + β 1 )
T (cid:88) t = α t E (cid:13)(cid:13)(cid:13)
ˆ V − 1 / 4 t m t (cid:13)(cid:13)(cid:13) 2 . = E (cid:13)(cid:13)(cid:13) ˆ V 1 / 4( θ 2 − θ (cid:63) ) (cid:13) (cid:13) (cid:13)α+
T (cid:88) t = d (cid:88) i = E ( θ t,i − θ (cid:63)i )(cid:32) ˆ v 1 / 2 t,i α t − ˆ v 1 / 2 t − 1 ,i α t −(cid:33) − E (cid:13)(cid:13)(cid:13) ˆ V 1 / 4 T ( θ T +1 − θ (cid:63) ) (cid:13)(cid:13)(cid:13)α T + T (cid:88)
[Equation]
So far we just rearrange the terms in the series sum. Next, we are ready to obtain the upper bound.
[Table]
[Equation]
where (i) follows from Assumption 3 and because ˆ v 1 /
[Equation]
α t > ˆ v [Equation] [Equation]
[Equation]
i
[Equation]
t − [Equation] and (ii) follows from Lemmas 1 - 3.
Finally, applying the Jensen’s inequality yields
[Equation]
We conclude our proof by further applying the bound in Equation (19) to Equation (20).
C P ROOF OF T HEOREM
To prove the convergence for AltQ-AMSGradR
[Equation]
the major technical development beyond the proof of Theorem 1 lies in dealing with the parameter restart.
More speciﬁcally, the moment approximation terms are reset every r steps, i
[Equation]
e., m kr
=
ˆ v kr = 0 for k = 1 , 2 , . . . , which implies θ kr +
= θ kr for k = 1 , 2 , . . . .
For technical convenience
[Equation]
we deﬁne θ
= θ 1 .
Using the arguments similar to Equation (19), in a time window that does not contain a restart (i
[Equation]
e. kr ≤ S ≤ ( k + 1) r − 1 ) we have
[Equation]
where (i) follows from Equation (19) and (ii) follows from θ kr +
=
θ kr due to the deﬁnition of restart
[Equation]
Then we take the summation over the total time steps and obtain
[Equation]
1 − δ ) √ 1 − β(cid:98)
T/r (cid:99) (cid:88) k
= d (cid:88) i =
E (cid:13)(cid:13) g ( k − 1) r
+1: kr − 1 ,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
kr − 1 (cid:88) t =( k − 1) r
+  t + α (1 + β 1 ) (1 − β 1 )(1 − δ ) √ 1 − βd (cid:88) i =
E (cid:13)(cid:13) g (cid:98) T/r (cid:99) r +1: T,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
T (cid:88) t
= (cid:98) T/r (cid:99) r +  t + G ∞ D 2 ∞ (cid:98) T/r (cid:99) (cid:88) k
= kr − 1 (cid:88) t =( k − 1) r + β 1 t α t + G ∞ D 2 ∞
T (cid:88) t
= (cid:98) T/r (cid:99) r + β 1 t α t ≤ (cid:98) T/r (cid:99) (cid:88) k
= (cid:18) G ∞ D 2 ∞ α √ kr + 2 + 4 c
1 − β 1 ) E (cid:107) θ kr − θ (cid:63) (cid:107) 2 (cid:19) + (cid:98) T/r (cid:99) (cid:88) k
= dG ∞ D
2 ∞ α √ kr −+ dG ∞ D 2 ∞ √ T
α + α (1 + β 1 ) (1 − β 1 )(1 − δ ) √ 1 − β(cid:98)
T/r (cid:99) (cid:88) k
= d (cid:88) i =
E (cid:13)(cid:13) g ( k − 1) r
+1: kr − 1 ,i (cid:13)(cid:13) (cid:118) (cid:117) (cid:117)(cid:116) kr − 1 (cid:88) t =( k − 1) r
+  t + α (1 + β 1 ) (1 − β 1 )(1 − δ ) √ 1 − βd (cid:88) i =
E (cid:13)(cid:13) g (cid:98) T/r (cid:99) r +1: T,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
T (cid:88) t
= (cid:98) T/r (cid:99) r +  t + G ∞ D 2 ∞
T (cid:88) t = β 1 t α t .
We can bound the term G ∞ D 2 ∞ (cid:80)
Tt
=1 β 1 t α t by Lemma 3.
Next, we bound another key term in the above inequality.
We ﬁrst observe that ∀ k ≥ 2 , ∀ i ∈ [ d ] ,
[Equation]
where (i) holds due to | g t,i | (cid:113) 1 t > 0 and (ii) follows from the Cauchy-Schwarz inequality.
Then we have
[Equation]
(cid:13)(cid:13) g (cid:98) T/r (cid:99) r
+1:
T,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
T (cid:88) t
= (cid:98) T/r (cid:99) r +  t (i) ≤ (cid:98) T/r (cid:99) (cid:88) k
= d (cid:88) i = | g kr,i | (cid:114)kr + (cid:98) T/r (cid:99) (cid:88) k
= d (cid:88) i =
(cid:13)(cid:13) g ( k − 1) r
+
kr − 1 ,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
kr − 1 (cid:88) t =( k − 1) r
+  t + d (cid:88) i =
(cid:13)(cid:13) g (cid:98) T/r (cid:99) r
+1:
T,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
T (cid:88) t
= (cid:98) T/r (cid:99) r +  t = (cid:98) T/r (cid:99) (cid:88) k
= d (cid:88) i = 
(cid:13)(cid:13) g ( k − 1) r
+1: kr − 1 ,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
kr − 1 (cid:88) t =( k − 1) r
+  t + | g kr,i | (cid:114)kr 
 + d (cid:88) i =
(cid:13)(cid:13) g (cid:98) T/r (cid:99) r
+1:
T,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
T (cid:88) t
= (cid:98) T/r (cid:99) r +  t (ii) ≤ (cid:98) T/r (cid:99) (cid:88) k
= d (cid:88) i =
(cid:13)(cid:13) g (
[Table]
− 1) r
[Table] [Table]
,i (cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
[Table]
t =( k
[Table] [Table]
t + d (cid:88) i =
[Table]
(cid:13)(cid:13)
[Table] [Table]
[Table]
(cid:118)(cid:117)(cid:117)(cid:116)
[Table]
(cid:88)
[Table]
(cid:98)
[Table]
(cid:99)
[Table]
t =
d (cid:88) i =
[Table]
  (cid:98)
[Table]
/r
[Table]
(cid:88) k
[Table]
(cid:13)(cid:13) g (
[Table]
1) r
[Table] [Table]
[Table]
(cid:13)(cid:13) (cid:118)(cid:117)(cid:117)(cid:116)
[Table]
(cid:88) t
[Table]
t +
[Table] [Table]
[Equation]
T/
[Equation]
t 

(iii) ≤ d (cid:88)
i
[Equation] [Equation]
g 1: T,i
[Equation]
t , where (i) follows from | g kr,i | (cid:113) 1 kr , ∀ k ≥ 1 , ∀ i ∈ [ d ] , (ii) follows from Equation (21) and (iii) holds due to the Cauchy-Schwarz inequality.
Then we have
[Table]
[Equation]
where (i) follows from Lemma 2 and Lemma 3.
Finally, applying the Jensen’s inequality and the above bound, we obtain
[Equation]
which concludes the proof
[Equation]