A CCELERATED V ALUE I TERATION VIA A NDERSON M IXING
Anonymous authors
Paper under double-blind review
A BSTRACT
Acceleration for reinforcement learning methods is an important and challenging theme.
We introduce the Anderson acceleration technique into the value iteration, developing an accelerated value iteration algorithm that we call Anderson Accelerated Value Iteration (A2VI).
We further apply our method to the Deep Q-learning algorithm, resulting in the Deep Anderson Accelerated Q-learning (DA2Q) algorithm.
Our approach can be viewed as an approximation of the policy evaluation by interpolating on historical data.
A2VI is more efﬁcient than the modiﬁed policy iteration, which is a classical approximate method for policy evaluation.
We give theoretical analysis of our algorithm and conduct experiments on both toy problems and Atari games.
Both the theoretical and empirical results show the effectiveness of our algorithm.
I NTRODUCTION
In reinforcement learning (Sutton & Barto, 1998), an agent seeks for the optimal policy in a speciﬁc sequential decision problem.
Several algorithms have been proposed over the course of time, including the famous
Q-learning (Watkins & Dayan, 1992), SARSA (Rummery & Niranjan, 1994; Sutton & Barto, 1998), and policy gradient methods (Sutton et al., 2000).
In complicated decision problems where tabular representations are intractable, function approximations are usually used for estimating state-action values (Kaelbling et al., 1996; Sutton & Barto, 1998; Sutton et al., 2000).
Inspired by the success of deep learning, Deep Q-Learning (DQN) (Mnih et al., 2013) and its variants (Bellemare et al., 2017; Schaul et al., 2015; Van Hasselt et al., 2016; Wang et al., 2015) utilize a deep neural network as the value approximator, which has successfully solved end-to-end decision problems such as Atari2000.
The value iteration (VI) Bellman (1957) and policy iteration (PI) (Howard, 1964) are the most classical methods for value updating.
The main difference between them is that PI evaluate the current policy accurately during the iteration while VI does not.
Thanks to the accurate evaluation of the current policy, policy iteration uses signiﬁcantly less policy improvement steps to converge to the optimal value.
Although PI has a faster convergence rate than VI, most of the existing methods employ a rather slow value iteration procedure, because thoroughly evaluating a policy is costly or even intractable under complex environments.
To retain the fast convergence property of policy iteration while reducing its computation overhead, researchers have proposed several modiﬁcations to the original policy iteration (Alla et al., 2015; Puterman, 1994).
The modiﬁed policy iteration (MPI) method (Puterman, 1994) tries to deal with this problem by approximating the solution to policy evaluation via the truncated Neumann expansion of an inverse matrix.
However, this approximation requires extra iterative steps, which is still computationally inefﬁcient for complex decision problems where sampling is costly, compared with the value iteration procedure where the policy iteration step is skipped.
Interpolation methods have been widely used in ﬁrst order optimization problems (Bubeck et al., 2015; Scieur et al., 2016; 2017; Xie et al., 2018).
These methods extract information from historical data and are proven to converge faster than vanilla gradient methods.
However, the interpolation method is not widely applied in reinforcement learning.
The most recent work related to interpolation is the averaged-DQN (Anschel et al., 2016), which calculates the average Q-value over the history and demonstrated that such an operation is effective for variance reduction.
Acceleration in value iteration and policy iteration has attracted researchers’ great attention.
Classical methods for accelerating value iteration include Gauss-Seidel value iteration (Puterman, 1994) and Jacobi value iteration (JAC) (Puterman, 1994).
More recently, Alla et al.
(2015) proposed an acceleration method that switches between a coarse-mesh value iteration and a ﬁne-mesh policy iteration during different stages.
Laurini et al.
(2016) performed a Jacobi-like acceleration method on dynamic programming problems.
In a recent work (Laurini et al., 2017), the value iteration procedure is accelerated by only updating a part of the values.
None of the previous methods have proposed acceleration methods with an application of interpolation.
In this paper, to solve the policy evaluation problem more efﬁciently, we propose an alternative algorithm based on multi-step interpolation.
Explicitly, the solution to the policy evaluation problem is approximately represented by a weighted combination of historical values, whose weights are adaptively updated by an optimization procedure.
To reduce the computational complexity, we resort to the Anderson mixing method (Anderson, 1965; Walker & Ni, 2011; Toth & Kelley, 2015) to do the approximation with only a short length of history.
Our approach ﬁts the gap between value iteration and policy iteration, ending in an updating rule without adding much extra computational complexity to the original value iteration procedure.
We also extend this approach to deep reinforcement learning problems.
The remainder of this paper is organized as follows.
In Section 2, we introduce the foundations of reinforcement learning and present typical value updating algorithms.
In Section 3, we derive the Anderson accelerated methods.
In Section 4, we give a theoretical analysis of the convergence of our method.
In Section 5, we test our method in different environments and empirically show the effectiveness of it.
Finally, we conclude our work in Section 6.
P RELIMINARIES
In this paper we mainly consider a ﬁnite-state and ﬁnite-action scenario in reinforcement learning.
In this case, an Markov Decision Process (MDP) system is deﬁned by a 5-tuple ( S , A , P, r , γ ) , where S is a ﬁnite state space, A is a ﬁnite action space, P ∈ R ( |S|×|A| )
×|S| is the collection of state-to-state transition probabilities, r ∈ R |S|×|A| is the reward matrix, γ is the discount factor.
A policy π ∈ A |S| is a vector of actions at each state.
The transition matrix P π ∈ R |S|×|S| and reward vector r π ∈ R |S| under policy π are deﬁned as P π ( i, j ) =
P (( i, π ( i )) , j ) , r π ( i ) =
r ( i, π ( i )) .
We further deﬁne the value v π ∈ R |S| and the Q-value q π ∈ R |S|×|A| under a given MDP and policy, where each element of v π and q π is deﬁned as
[Equation]
We can verify that q π = r + γP v π .
We deﬁne q π ˜ π ∈ R |S| by q π ˜ π
( i ) =
q π ( i, ˜ π ( i )) , and say a vector to be the maximum among a set if each entry of it is bigger than that of the other vectors.
The values satisfy the
Bellman equation:
[Equation]
The policy π ∗ = argmax π q π is called the optimal policy, whose value or Q-value is denoted as v ∗ or q ∗ .
Note that v ∗ satisﬁes the Bellman optimality equation
[Equation]
Therefore, ﬁnding the optimal policy is equivalent to ﬁnding the ﬁxed point of the operator Γ( v ) .
2. F IXED P OINT I TERATION M ETHODS
Value iteration (VI) is the most widely used and best-understood algorithm for solving Markov decision problems.
It solves the ﬁxed point problem by iterating the following steps repeatedly,
[Equation]
An alternative solution is policy iteration (PI), which maintains both the value v ( t ) and the policy π ( t ) during each iteration.
The procedure alternatively iterates the following two steps:
• Policy evaluation: Find a v ( t ) such that
[Equation]
which can be directly computed by
[Equation]
• Policy improvement: Improve the current policy by
[Equation]
Theoretical analysis has shown that VI enjoys a γ -linear convergence rate (i.e., (cid:107) v ( t ) − v ∗ (cid:107) ∞ ≤ γ (cid:107) v ( t − 1) − v ∗ (cid:107) ∞ ), while PI converges much faster with (cid:107) v ( t ) − v ∗ (cid:107) ∞ ≤ K (cid:107) v ( t − 1) − v ∗ (cid:107) 2 ∞ (Puterman, 1994), where K is some constant related with γ and the given MDP.
Both VI and PI are model-based, because the greedy policy cannot be determined when r and P are unknown.
The VI under q -notation is well-known as Q-learning (Watkins & Dayan, 1992).
We will analyze our method under v -notation, but our analysis also works under the corresponding q -notation.
The main difference between VI and PI is whether the current policy is fully evaluated.
Though PI converges faster than VI, this advantage diminishes under several settings.
In reinforcement learning, we can only access an oracle that returns the reward and next state given the current state and selected action.
Under such a setting, each value iteration step can be performed by estimating Γ( v ) through sampling.
But the policy evaluation step based on equation (2) becomes intractable because it is quite time-consuming to compute ( I − γP π ( t ) )
− 1 .
The modiﬁed policy iteration method (Puterman & Brumelle, 1978) partially solves the problem by setting v t
≈
(Γ π ( t ) ) m t ( v ( t − 1) ) where m t is a (possibly large) integer related to t .
However, this method requires to evaluate a series of values (Γ π ( t ) )
i ( v ( t − 1) ) for i = 1 , 2 , . . .
, m t , which is computationally inefﬁcient.
A NDERSON A CCELERATED V ALUE I TERATION
Based on the observation that full policy evaluation accelerates convergence, we propose an approximate policy evaluation method.
The method aims to approximately solve the policy evaluation problem, circumventing the matrix inversion and iterative procedures mentioned above.
We ﬁrst utilize the linearity of equation (1), deﬁning B π ( v ) =
Γ π ( v ) − v and converting the problem into an equivalent form of solving the equation
B π ( v ) = 0 .
Suppose we have obtained a set of values B π ( v 1 ) , B π ( v 2 ) , . . .
, B π ( v k ) with respect to v 1 , v 2 , . . .
, v k .
Consider to ﬁnd a set of weights α
=
( α 1 , α 2 , . . .
, α k ) T , subject to (cid:80) ki
=1 α i = 1 , which satisﬁes that
[Equation]
Then the combination ˜ v = (cid:80) ki
=
α i v i will satisfy the following relationship:
[Equation]
This relation implies ˜ v can be viewed as an approximate solution to equation (1) provided the sampling estimations are accurate enough.
However, this step needs to keep track of the previous values and recompute Γ π on all of them.
To reduce the huge memory usage and computation, we choose v i from the recent history, i.e., v i = v ( t − i ) , i = 1 , 2 , . . .
, k , and replace B π ( t ) ( v ( t − i ) ) with the previously computed values
B π ( t − i ) ( v ( t − i ) ) .
This modiﬁcation is based on the observation that the recent successive policies do not change sharply and therefore B π ( t − i ) ( v ( t − i ) )
≈ B π ( t ) ( v ( t − i ) ) .
This modiﬁcation approximately solves the policy evaluation problem without model estimation or extra function evaluations.
Another critical issue is that we cannot guarantee the existence of α given that k is small, because the dimension of B π ( v ) is usually much higher than k .
Inspired by the Anderson acceleration technique (Anderson, 1965; Ortega & Rheinboldt, 1970; Walker & Ni, 2011), we instead look for a combination of { B π ( t − i ) ( v ( t − i ) ) } ki =1 ,
[Equation]
where B ( t ) =
( B π ( t − 1) ( v ( t − 1) ) , B π ( t − 2) ( v ( t − 2) ) , . . .
, B π ( t − k ) ( v ( t − k ) )) , Ω = { α | 1 T α = 1 } , Λ is an extra constraint on the values attainable by α .
Typically, Λ can be chosen from the following forms:
• Total space, Λ tot = R k ; • Boxing constraint, Λ box =
{ α | − m 1 ≤ α ≤ m 1 } ; • Convex combination constraint, Λ cvx = { α | 0 ≤ α ≤ 1 } ; • Extrapolation constraint, Λ exp =
{ α | α 1 ≥ 1 , α i ≤ 0 , i = 2 , 3 , . . . , k } .
When the (cid:96) 2 norm is used and Λ = Λ tot , the solution can be written explicitly as α ( t ) =
[( B ( t ) ) (cid:62) B ( t ) ] − 1 1 / 1 (cid:62)
[( B ( t ) ) (cid:62) B ( t ) ] − 1 1 , whose derivation is placed in the appendix.
Note that if we simply set v ( t )
=
(cid:80) ki
=1 α ( t ) i v ( t − i ) , the values will always locate in the subspace expanded by historical values v ( t − 1) , v ( t − 2) , . .
. , v ( t − k ) .
When the solution to equation (1) does not lie in such a subspace, there is no hope for convergence with application of such updating rule directly.
To jump out of the subspace, we perform an extra value iteration step to this combination.
Then we will get the updated value,
[Equation]
3. T HE A LGORITHM
Based on our previous discussion, we present the k -step Anderson Accelerated Value Iteration (A2VI) in
Algorithm 1.
In the ﬁrst k steps, the value is updated according to the original VI.
Otherwise, we perform an interpolation procedure, where the weights are attained from solving the problem (5).
The original value iteration algorithm can be viewed as a special case of our algorithm with k = 1 .
Algorithm
Input: v (0) , P, r , γ, k, T 1: for t = 1 , 2 , . .
, T do 2:
B π ( t − 1) ( v ( t − 1) )
=
max π ( r π + γP π v ( t − 1) ) − v ( t − 1) 3: if t < k then 4: v ( t ) =
max π ( r π + γP π v ( t − 1) ) 5: else 6:
Calculate ( α ( t ) 1 , α ( t ) 2 , . . .
, α ( t ) k ) by solving the optimization problem (5) 7: v ( t ) =
max π (cid:16) r π + γP π (cid:104)(cid:80) ki
=1 α ( t ) i v ( t − i )
(cid:105)(cid:17) 8: end if 9: end for 10: π ( T ) =
argmax π
( r π + γP π v ( T ) ) 11: return v ( T ) , π ( T )
Both Anderson Acceleration (AA) and A2VI have the same spirit of interpolating on historical data.
However,
A2VI does not straightforwardly apply AA to the Bellman optimality equation.
Note that AA has the updating rule v t =
(cid:80) α i B ( v t − i ) , while A2VI exchange the order of the operator sum and B ( · ) due to the motivation from equation (3).
This exchange puts the nonsmooth operator max out of the afﬁne combination, simplifying the theoretical analysis.
We present a geometric explanation on the iterative steps of VI, PI, A2VI under 1-dimensional case in Figure 1.
In value iteration, v ( t ) is attained by making a vertical line at ( v ( t − 1) , 0 ) , ﬁnding its intersection with the function line at ( v ( t − 1) , B ( v ( t − 1) )) , then drawing a line with slope − 1 through ( v ( t − 1) , B ( v ( t − 1) )) and ﬁnding its intersection with the horizon axis at ( v ( t ) , 0 ) ;
In policy iteration, v ( t ) is attained by ﬁrst getting ( v ( t − 1) , B ( v ( t − 1) )) in the same way as value iteration, then calculating the tangent line through ( v ( t − 1) , B ( v ( t − 1) )) and ﬁnding its intersection with the horizon axis.
In Anderson accelerated value iteration, each step is ﬁrst performed in a similar style to policy iteration except that the tangent line is replaced with a secant line.
Then a value iteration step is performed to get v ( t ) .
From the ﬁgure, we can see that VI only utilizes the current value of the Bellman residual, while PI is similar to Newton’s method Puterman & Brumelle (1978), utilizing the gradient information to achieve a faster convergence rate.
Our method serves as an intermediate between them, each step of which is composed of an ordinary value iteration step and a secant step.
In speciﬁc, the replacement of the tangent line to a secant line can be viewed as a quasi-Newton’s method, which is shown computationally more efﬁcient while keeping a fast convergence rate in several particular settings.
Both PI and A2VI converge to the ﬁxed point in a smaller number of steps than VI.
Compared with PI, A2VI is more practical because it approximates the tangent line by a secant line, which circumvents the costly model estimation step.
3. E XTENSION TO M ODEL - FREE L EARNING A LGORITHM
We can rewrite our algorithm under q -notation, and get the Anderson Accelerated Q-Learning (A2Q)
Algorithm shown in the appendix.
Combined with the technique of deep learning, our method can be applied to end-to-end decision problems, resulting in the Deep Anderson Accelerated Q-Learning (DA2Q) Algorithm (Algorithm 2). v * vvv 3 v 4 v 5 v
B v r 1 +( P 1 I ) v r 2 +( P 2 I ) v r 3 +( P 3 I ) v
Value iteration vvvv 4 = v * v
B v r 1 +( P 1 I ) v r 2 +( P 2 I ) v r 3 +( P 3 I ) v
Policy iteration v * vvvv 4 v
B v r 1 +( P 1 I ) v r 2 +( P 2 I ) v r 3 +( P 3 I ) v Bv Bv Bv
Anderson accelerated value iteration
Input: M, N, T, γ, b, B, ε, η, K, C Sample a random minibatch { ( s j j r j s =
D 11: j b do 12: y j =   r j for terminal state r j + γ max a (cid:32) K (cid:88) k
= α k Q ( s (cid:48) j , a ; θ − k ) (cid:33) for non-terminal state 13: end for 14: L ( θ ) = 1 {+b (cid:80) bj =
 ( y j − Q ( s j+} , a j ; θ ))15: θ = θ − η ∂L∂θ 16: s = s +17: if s mod C = 0 then 18:
Assign θ − k
= θ − ( k − 1) for k = K, K − 1 , . . .
Assign θ − 1 = θ .
19: if s ≥ K ( C − 1) then 20:
Sample a random minibatch of transitions { ( s j , a j , r j , s (cid:48) j ) }
Bj =1 from D 21: for j = 1 , 2 , . .
, B do 22: for k = 1 , 2 , . . .
, K do 23:
d kj = (cid:40) r j − Q ( s j , a j ; θ − k ) for terminal state r j + γ max a Q ( s (cid:48)
j , a ; θ − k ) − Q ( s j , a j ; θ − k ) for non-terminal state 24: end for 25: end for 26: end if 27: ( α 1 , α 2 , . .
. , α K )
=
argmin ( α 1 ,α 2 ,...,α K ) (cid:80) Bj =1 ( (cid:80)
Kk
=1 α k d kj ) 2 s.t.
(cid:80)
Kk
=
α k =28: end if 29: end for 30: end for
T HEORETICAL A NALYSIS
We ﬁrst analyze of the local convergence of the A2VI algorithm under boxing constraint.
Our result shows that in a small neighborhood of the optimal value, our algorithm enjoys an exponential convergence rate.
Theorem 1.
For any MDP with a unique optimal policy, there exists some δ > 0 , such that for any initial value v (0) ∈
U δ ( v ∗ ) = { v |(cid:107) v − v ∗ (cid:107) ∞ ≤ δ } , the A2VI algorithm under boxing constraint maintains the following properties:
(i) (cid:107) Γ( v ( t ) ) − v ( t ) (cid:107) ∞ ≤ γ (cid:107) Γ( v ( t − 1) )
− v ( t − 1) (cid:107) ∞ , ∀ t = 1 , 2 , . . .
; (ii) (cid:107) v ( t ) − v ∗ (cid:107) ∞ ≤ γ t 1 − γ (cid:107) Γ( v (0) ) − v (0) (cid:107) ∞ , ∀ t = 1 , 2 , . . . .
Generally, it is difﬁcult to obtain the global convergence rate of A2VI, since the operation max is nonsmooth.
To guarantee the convergence, we introduce a rejection step to the original algorithm.
We say v is monotonic improving if Γ( v ) ≥ v , and denote the set of such values as V B .
We propose the A2VI algorithm with the rejection step, which only differs with Algorithm 1 at line 6.
After calculating α ( t ) , we test whether the afﬁne combination (cid:80) ki =1 α ( t ) i v t − i lies in V B .
If the answer is negative, the interpolation step will be replaced with an ordinary value iteration step.
We put the pseudocode of A2VI with the rejection step in Appendix.
With this modiﬁcation, we can have the following convergence properties.
Theorem 2.
For the A2VI algorithm with the rejection step with Λ = Λ cvx , if v (0) ∈ V B , then we have
[Equation]
Theorem 3.
For the A2VI algorithm with Λ = Λ exp , if v 0 ≥ 0 and v (0) ∈ V B , then we have (a) Monotone improving values, v ( t − 1) ≤ v ( t ) ≤ v ∗ , v ( t ) ∈ V B , ∀ t = 1 , 2 , . . .
(b) γ -linear convergence rate, (cid:107) v ∗ − v ( t ) (cid:107) ∞
≤ γ (cid:107) v ∗ − v ( t − 1) (cid:107) ∞ .
E XPERIMENTS
To validate the effectiveness of our method, we conduct several experiments.
5. E XPERIMENTS ON T OY M ODELS
We ﬁrst test our method on three toy models.
The ﬁrst model is a randomly generated MDP with |S| =and |A| = 50 .
The transition probabilities of the MDP are generated from a uniform distribution on [0 , 1] , and the rewards are generated from a standard normal distribution.
The second model is the N -Chain problem with N = 100 , where a reward of 0 .
1 is given at state 0 and a reward of 1 is given at state N .
At each state, the agent can either choose to move forward or backward, and will move to the selected direction with probability 0 .
9 and to the opposite direction with probability 0 .
The last model is a 20 × 20 Gridworld model, where a reward of 1 is given at state (20 , 20) .
At each state, the agent can choose one of the 4 directions and will move to that direction with probability 0 .
7 , or move to one of the other directions with probability 0 .
1 for each.
We perform the standard value iteration, policy iteration and Anderson accelerated value iteration with/without the rejection step on these models.
In our experiment, each policy iteration step is approximately solved by the modiﬁed policy iteration method with 100 inner iterations.
To compare our method with the averaged updating scheme (Anschel et al., 2016), we further construct and compare our algorithm with the averaged value iteration.
The value of (cid:107) v t − v ∗ (cid:107) w.r.t.
step t is shown in Figure 2, where the results are averaged from 30 independent experiments.
From the results we can see that the policy iteration converges fastest for all of the three models, however, since each of its steps includes 100 inner iterations, the actual computation cost is very high.
Among value iteration methods, the Anderson accelerated value iteration converges fastest.
The acceleration effect is remarkable in randomly generated MDPs, but A2VI slows down at the ﬁrst few steps in the latter two experiments.
However, adding a rejection step solves the problem and attains a faster convergence rate.
Another observation is that in the toy model case, the averaged value iteration cannot be used for acceleration.
[Figure]
D i s t an c e t o O p
t
i m a l V a l ue
Random MDP     1000 Steps
D i s t an c e t o O p
t
i m a l V a l ue
N-Chain
[Figure]
D i
[Figure]
c
[Figure]
p t i
[Figure]
5. E XPERIMENTS ON A TARI G AMES WITH D EEP L EARNING B ASED T ECHNIQUES
To ﬁgure out the performance of our method on complex environments, we apply our method to Atari games from Gym (Brockman et al., 2016), which is a Python API to Arcade Learning Environment (Bellemare et al., 2013).
We compare DA2Q with DQN (Mnih et al., 2013) and Averaged-DQN (Anschel et al., 2016).
Details of the experiment settings are given in Appendix D.
As Figure 3 points out, our algorithm DA2Q obtains a signiﬁcant improvement over both the original DQN algorithm and the Averaged DQN algorithm.
When compared with other interpolation method such as
Averaged-DQN, the overall performance of our method also tends to be stabler, always being superior than other methods among all of the three environments, while the performance of Averaged-DQN varies a lot.
[Figure]
[Figure]
S c
o r e
Boxing
DA2Q, K=2 AVE,
K=2 DQN
Compared with DQN, the extra computational cost is actually low, since the α is updated only once every C steps, which only involves an inversion on a very small-size matrix ( k × k ).
The k target values are computed parallelly in the TensorFlow (Abadi et al., 2016), which cost the same time as in DQN.
Moreover, the extra runtime can be ignored when compared with the costly back propagations and interaction with environments.
C ONCLUSION
We have proposed the Anderson accelerated value iteration method, which is a novel acceleration approach for reinforcement learning.
We have proved the convergence property of our method under certain conditions.
Our algorithm empirically achieves a superior performance on toy models and several Atari games.
Despite the success of our algorithm, several questions remain open.
The convergence analysis for the general case is lacking, and we only provide convergence guarantees but do not give a theoretical analysis of the acceleration effect of A2VI, which we leave for future work.
R A PPENDIX A. A2Q
Here is the pseudocode of A2Q, which is A2VI in q -notation.
Input: q 0 , P, r , γ, k, T 1: for t = 1 , 2 , . .
, T do 2:
B π ( t − 1) ( q t − 1 ) =
vec ( r ) + γP max π q t − 1 π − vec ( q t − 1 ) 3: if t < k then 4: vec ( q t ) = vec ( r ) + γP max π q t − 1 π 5: else 6:
Calculate ( α 1 , α 2 , . . . , α k ) by solving the optimization problem (5) 7: vec ( q t ) = vec ( r ) + γP max π ( (cid:80) ki
=
α i q t − i π ) 8: end if 9: end for 10: π T = arg max
π q
Tπ 11: return q T , π T
B. A2VI WITH THE R EJECTION S TEP
Here is the pseudocode of A2VI with the rejection step, the only difference between this algorithm with algorithm 1 is before the interpolation step, we ﬁrst check whether the afﬁne combination is in V B .
If the answer is negative, then this interpolation step is replaced with an ordinary value iteration step.
Input: v (0) , P, r , γ, k, T 1: for t = 1 , 2 , · · · , T do 2:
B π ( t − 1) ( v
[Equation]
=
max
[Equation]
( r
[Equation]
+ γP π v ( t − 1) ) − v ( t − 1) 3: if t < k then 4: v ( t ) =
max π ( r π + γP π v ( t − 1) ) 5: else 6:
Calculate ( α ( t ) 1 , α ( t ) 2 , · · · , α ( t ) k ) by solving the optimization problem (5) 7: ˜ v = (cid:80) ki
=1 α ( t ) i v ( t − i ) 8:
if max π ( r π + γP π ˜ v ) ≥ ˜ v then 9: v ( t ) = max π ( r π + γP π ˜ v ) 10: else 11: α ( t ) 1 = 1 , α ( t ) i =
for i (cid:54)
=12: v ( t ) =
max π (cid:0) r π + γP π v ( t − 1) (cid:1) 13: end if 14: end if 15: end for 16: π ( T ) = argmax π ( r π + γP π v ( T ) ) 17: return v ( T ) , π ( T )
Lemma 1.
For any MDP whose optimal policy is unique, there exists a δ > 0 and a policy π ∗ such that for any
[Equation]
Proof Because the optimal policy is unique, for any nonoptimal policy π , for any state s such that π ( s ) (cid:54) =
π ∗ ( s ) we have that [Γ π ∗ ( v ∗ )]
s >
[Γ π ( v ∗ )] s , [ · ] s means executing operations on state s .
Denote A ( π ) =
{ s | π ( s ) (cid:54) =
π ∗ ( s ) } .
Suppose the optimal policy is π ∗ , then there exists ε such that
[Equation]
since the optimal policy is unique and the state space and the action space are ﬁnite.
We choose δ =
ε 3 γ , then for any policy π and
any v ∈ U δ ( v ∗ ) we have
[Equation]
Then for any policy π , for any state s ∈ A ( π ) , we have that
[Equation]
which means π does not choose the optimal action in state s .
Therefore, if π selects the optimal action in every state s ∈ S , then we must have π s = π ∗ s , ∀ s ∈ S , which implies π ∗ ∈ arg max π Γ π ( v ) , i.e., Γ( v ) =
r π ∗ + γP π ∗ v .
Lemma 2.
For any given MDP with optimal value v ∗ and any value v , we always have
[Equation]
Proof
[Equation]
Proof of Theorem 1 From lemma 1 we know there exists a policy π and a ˜ δ > 0 such that the optimal
Bellman operator is a linear function on U ˜ δ
( v ∗ ) .
We now set δ sufﬁciently small such that
[Equation]
The result is trivial for the ﬁrst k − 1 steps, which are performed exactly by standard value iteration.
When t > k , we prove the result by induction.
Suppose the conclusion is correct for previous steps, then we have
[Equation]
It follows that
[Equation]
Therefore, (cid:80) ki =1 α ( t ) i v ( t − i ) ∈
U ˜ δ ( v ∗ ) , v ( t ) ∈
U ˜ δ ( v ∗ ) , which implies
[Equation]
Then we can get
[Equation]
Taking norm on both side of the equation and utilizing the deﬁnition of α ( t ) i , i = 1 , 2 , · · · , k , we get
[Equation]
Therefore, we justify property (i).
Property (ii) then follows directly from Lemma 2.
Lemma 3.
For any MDP, suppose the values u and v satisfy u ≥ v , then
[Equation]
Proof As stated in section 2, u ≥ v means that u ( s ) ≥ v ( s ) for any state s .
Suppose ˜ π = arg max π r π + γP π v , therefore for any s we have that
[Equation]
Proof of Theorem 2 On the one hand,
[Equation]
(cid:88)
[Equation]
On the other hand, we denote ˜ π
= argmax π ( r π + γP π (cid:80) ki
=1 α ( t ) i v ( t − i ) ) , then
[Equation]
The above two results shows
[Equation]
Now, consider the rejection step.
First we show that if v ∈ V B then Γ( v ) ∈ V B .
If v ∈ V B , then Γ( v ) ≥ v .
With Lemma 3, we have that Γ(Γ( v )) ≥ Γ( v ) , i.e. Γ( v ) ∈ V B .
Next we show that if v ( i ) ∈ V B for i < t , then v ( t ) ∈ V B .
According to the rejection algorithm, we have v ( t ) =
Γ( (cid:80) ki
=1 α ( t ) i v ( t − i ) ) .
If Γ( (cid:80) ki =1 α ( t ) i v ( t − i ) ) ≥ (cid:80) ki
=1 α ( t ) i v ( t − i ) , then (cid:80) ki
=1 α ( t ) i v ( t − i )
∈ V B and v ( t ) ∈ V B .
If Γ( (cid:80) ki =1 α ( t ) i v ( t − i ) )
< (cid:80)
ki
=1 α ( t ) i v ( t − i ) , then v ( t ) =
Γ( v ( t − 1) )
due to the rejection step.
Since v ( t − 1) ∈ V B , we have that v ( t ) ∈ V B .
Therefore, we also have that B ( (cid:80) ki
=1 α ( k ) i v ( t − i ) ) ≥ 0 .
We have that
[Equation]
≤
[Equation]
≤
[Equation]
The second inequality is due to the deﬁnition of α ( t ) i , i = 1 , 2 , ..., k .
Proof of Theorem 3 We prove the conclusion by induction.
It is evident that the conclusion holds for the ﬁrst k − 1 steps.
Suppose the conclusion holds for the ﬁrst t − 1 steps, then
[Equation]
where ˜ π = arg max π r π + γP π v ( t − 1) .
The second inequality comes from the extrapolation restriction.
The
Third inequality is due to that if v ∈ V B then Γ( v ) ∈ V B , which is shown in Theorem 2.
As shown in Theorem 2, we have that v ( t ) ∈
V B and (cid:80) ki
=1 α ( t ) i v ( t − i )
∈ V B .
Therefore, v ( t ) ≤ v ∗ .
[Equation]
Taking the inﬁnite norm on both sides, we get
[Equation]
which completes the proof.
D. E XPERIMENT D ETAILS D.1 M ODEL A RCHITECTURE AND H YPER - PARAMETERS
For our experiments, we used the DQN(Mnih et al., 2013) architecture, where the Q-value network is composed of 3 convolutional layers, 1 fully connected layer, and 1 output fully connected layer.
Each layer except the ﬁnal layer is followed with a rectiﬁed linear activation(ReLU).
The ﬁrst convolutional layer use8 × 8 ﬁlters with stride 4, the second has 64 4 × 4 ﬁlters with stride 2, and the third convolutional layer has3 × 3 ﬁlters with stride 1.
The fully connected layer consists of 512 units and the ﬁnal layer outputs a single value for each action.
We used the Adam optimizer with learning rate 0.0001 and (cid:15) = 0 . 0015 .
The discount was set to γ = 0 . 99 .
Training is done over 20M or 40M frames.
We updated the target networks everysteps.
The size of experience replay buffer is 100000 tuples, where 32 mini batches were sampled everysteps to update the network.
The exploration policy is ε -greedy policy with ﬁxed ε = 0 .
D.2 P REPROCESSING OF E NVIRONMENTS
We preprocess the environment in the same way as the original DQN paper (Mnih et al., 2013) does.
We utilize the action repeat technique, i.e., each action is repeated for the next four consecutive frames.
The frames are ﬁrstly grey-scaled and then rescaled to the size of 84 × 84 pixels.
Each state is represented by a concatenation of 4 consecutive frames.
We ﬁx all positive rewards to be 1 and all negative rewards to be -1, leaving 0 rewards unchanged.
Transitions associated with the loss of a life are considered terminal.
E. S OLVING (5) W HEN U SING THE (cid:96) 2 N ORM AND T OTAL S PACE C ONSTRAINT
Under the given setting, we may rewrite the original problem in the following form,
[Equation]
to
[Equation]
This problem can be directly solved with an application of Lagrange multiplier method, namely, let λ be the
Lagrange multiplier, then we solve the problem
[Equation]
whose solution can be written explicitly as α = − λ 2 [
[Equation]
ˆ B
[Equation]
[Equation]
ˆ B ( t ) ]
[Equation]
Combine this result with the constraint, we can get − λ 2 =
[Equation]
, which implies α
[Equation]