H IGHWAY R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Traditional Dynamic Programming (DP) approaches suffer from slow backward credit-assignment (CA): one time step per update.
A popular solution for multistep CA is to use multi-step Bellman operators.
Existing control methods, however, typically suffer from large variance of multi-step off-policy corrections or are biased, preventing convergence.
To overcome these problems, we introduce a novel multi-step Bellman Optimality operator, which quickly transports credit information from the future into the past through multiple “highways” induced by various policies.
Our operator is unbiased with respect to the optimal value function and converges faster than the traditional Bellman Optimality operator.
Its computational complexity is linear in lookahead depth. Moreover, it yields a family of novel multi-step off-policy algorithms that do not need importance sampling. We derive Highway Q-Learning, a convergent multi-step off-policy variant of Q-learning, which significantly outperforms DP on toy tasks.
We also propose a deep function approximation variant called Highway DQN.
We evaluate it on visual MinAtar Games, outperforming similar multi-step methods.
I NTRODUCTION
Recent advances in multi-step reinforcement learning (RL) have achieved remarkable empirical success (Horgan et al., 2018; Barth-Maron et al., 2018).
However, a major challenge of multi-step RL is to balance the trade-off between traditional “safe” one-time-step-per-trial credit assignment (CA) relying on knowledge stored in a learned Value Function (VF), and large CA jumps across many time steps.
A traditional way of addressing this issue is to impose a fixed prior distribution over the possible numbers of CA steps, e.g., TD( λ ) (Sutton & Barto, 2018), GAE( λ ) (Schulman et al., 2016).
This typically ignores the current state-specific quality of the current VF, which dynamically improves during learning.
Besides, the prior distribution usually has to be tuned case by case.
Multi-step RL should also work for off-policy learning, that is, learning from data obtained by other behavioral policies.
Most previous research on this has focused on Policy Iteration(PI)-based approaches (Sutton & Barto, 2018), which need to correct the discrepancy between target policy and behavior policy to evaluate the VF (Precup, 2000;
Harutyunyan et al., 2016; Munos et al., 2016; Sutton & Barto, 2018; Schulman et al., 2016).
Classic importance sampling(IS)-based methods is proven to be unbiased, but suffer from high variance due to the product of IS ratios (Cortes et al., 2010;
Metelli et al., 2018). Recently, several variance reduction methods have been proposed and shown to be effective in practice, such as Q( λ ) (Harutyunyan et al., 2016), Retrace ( λ ) (Munos et al., 2016) and C-trace (Rowland et al., 2020), and so on (Espeholt et al., 2018; Horgan et al., 2018; Asis et al., 2017).
In contrast to PI, Value Iteration (VI) methods propagate the values of the most promising actions backward one step at a time (Sutton & Barto, 2018; Szepesv´ari, 2010).
Such methods can safely use data from any behavioral policy.
However, step-by-step value propagation makes them somewhat ill-suited for general multi-step CA.
Here we provide a new tool for multi-step off-policy learning by extending VI approaches to the multi-step setting. The foundation of our method is a new Bellman operator, the Highway Operator , which connects current and future states through multiple “highways,” and focuses on the most promising one. Highways are constructed through various policies looking ahead for multiple steps.
Our operator has the following desirable properties: 1) It yields a new Bellman Optimality Equation that reflects the latent structure of multi-step CA, providing a novel sufficient condition for the optimal VF; 2) It effectively assigns future credit to past states across multiple time steps and has remarkable convergence properties; 3) It yields a family of novel multi-step off-policy algorithms that do not need importance sampling, safely using arbitrary off-policy data.
Experiments on toy tasks and visual MinAtar Games (Young & Tian, 2019) illustrate that our Highway RL algorithms outperform existing multi-step methods.
P RELIMINARIES
A Markov Decision Processes (MDP) (Puterman, 2014) is described by the tuple M = ( S , A , γ, T , µ 0 , R ) , where S is the state space; A is the action space; γ ∈ [0 , 1) is the discount factor.
We assume MDPs with countable S (discrete topology) and finite A .
T : S × A → ∆( S ) is the transition probability function; µ 0 denotes the initial state distribution; R : S ×
A → ∆( R ) denotes reward probability function.
We use the following symbols to denote related conditional probabilities: T ( s ′ | s, a ) , R ( ·| s, a ) , s, s ′ ∈ S , a ∈ A .
We also use r ( s, a ) ≜ E R ∼R ( ·| s,a ) [ R ] for convenience.
In order to make the space of value functions complete (assumption of Banach fixed point theorem), we assume bounded rewards, which with discounting produce bounded value functions.
We denote l ∞ ( X ) the space of bounded sequences with supremum norm ∥·∥ ∞ with support X assuming X is countable and has discrete topology.
Completeness of our value spaces then follows from completeness of l ∞ ( N ) 1 .
The goal is to find a policy π : S → ∆( A ) that yield maximal return. The return is defined as the accumulated discounted reward from time step t , i.e., G t = (cid:80) ∞ n [-=
-]=0 γ n r ( s t + n , a t + n ) .
The statevalue function (VF) of a policy π is defined as the expected return of being in state s and following policy π , V π ( s ) ≜
E [ G t | s t =
s ; π ] .
Let Π denote the space of all policies.
The optimal VF is V ∗ = max π ∈ Π V π .
It is also convenient to define the action-VF, Q π [Equation]
s, a ) ≜
E [ G t | s t = s, a t = a ; π ] and the optimal action-VF is denoted as Q ∗ = max π ∈ Π Q π .
The Bellman Expectation/Optimality Equation and the corresponding operators are as follows:
[Equation]
H IGHWAY R EINFORCEMENT L EARNING
Value Iteration (VI) looks ahead for one step to identify a promising value using only short-term information (see eq. 2).
How can we quickly exploit long-term information through larger lookaheads?
Our idea is to exploit the information conveyed by policies.
We connect current and future states through multiple “highways” induced by policies, allowing for the unimpeded flow of credit across various lookaheads, and then focus on the most promising highway.
Formally, we propose the novel Highway Bellman Optimality Operator G (cid:98) Π N (Highway Operator in short), defined by
[Equation]
where (cid:98) Π = { π 1 , · · · , π m · · · , π M | π m ∈ Π } is a set of behavioral policies , which are used to collect data; n is called the lookahead depth (also named bootstrapping step in RL literature), and N is the set of lookahead depths , which we assume always includes 0 ( 0 ∈ N ) unless explicitly stated otherwise; τ ns {+
+} = ( s 0 , a 0 , s 1 , a 1 , s 2 , a 2 , · · · , s n ) , and τ ns 0 ∼ π is the trajectory starting from s 0 by executing policy π for n steps.
Fig.
1 (Left) illustrates the backup diagram of this operator.
Our operator can be rewritten using the Bellman Operators: [Equation] As implied by eq.
(3) and eq.
(4), given some trial, we pick a policy and a possible lookahead (up to the trial end if N is sufficiently large) that maximize the cumulative reward during the lookahead interval plus a dynamic programming-based estimate of the future return. This provides a highway
𝑆 !
𝑆 " 𝑆 # 𝑆 $ 𝑆 % 𝑆 & 𝑆 ’ … … 𝑆 ( 𝑆 ) 𝑆 * … … - state reward (directed) transition (undirected) transition policy 𝜋 +,-. policy 𝜋 /.
-  max ! …
𝜋 ! 𝜋 " stateaction 𝑛 =𝑛 =𝑛 =max ! max " max " max " 𝜋 # 𝜋 # 𝜋 # max !
max " max " to quickly transport credit information from the future into the past.
That’s why our operator generally converges faster than the classical Bellman Optimality Operator.
See the detailed theoretical analysis in Section 5.
Fig.
1 (Right) illustrates this highway through an example: an N -horizon MDP problem and two policies.
A highway connects the start state s A to information derived from deeper lookaheads.
In this example, our operator converges to the optimal VF within 2 iterations, while B needs N .
The following result shows that our Highway operator is a contraction on the complete metric space l ∞ and admits V ∗ as a unique fixed point.
Theorem 1 (Properties of Highway Operator)
For any (cid:98) Π and N (s.t. 0 ∈ N ), we have 1) G (cid:98)
Π N is a contraction on complete metric space l ∞ ( S ) , i.e. for any V,
V ′ ∈ l ∞ ( S ) , we have∥G (cid:98)
Π N V − G (cid:98) Π N V ′
∥
≤ ∥B V − B V ′ ∥ ≤ γ ∥ V − V ′ ∥ ; 2) (Highway Bellman Optimality Equation, Highway Equation in short) V ∗ is the only fixed point G (cid:98) Π N , that is, for all V ∈ l ∞ ( S ) holds V = G (cid:98) Π N V if and only if V = V ∗ .
Formally, we have
[Equation]
For any V 0 ∈ l ∞ ( S ) and any sequence of policy sets ( (cid:98) Π k ) , k ∈ N , the sequence ( G (cid:98)
Π
k N ◦ G (cid:98) Π k − 1 N ◦ . . .
◦ G (cid:98) Π 1 N )[ V 0 ] , k ∈ N converges R-linearly to V ∗ with convergence rate γ .
All the proofs are provided in Appendix A. Note that 0 ∈ N is necessary for the guarantee of the fixed point property (Point 2), but not for the contraction property. This theorem implies that our Highway operator can provide a powerful extension to Bellman Optimality Equation, which can be potentially applied to various RL control methods. Table 1 summarizes the comparison to classical
Bellman Operators and some advanced operators.
More details on the comparison are in Section 5.
A LGORITHMS
Here we illustrate three applications of our Highway theory for model-based and model-free RL algorithms. Although we listed only three instances in this paper, note that our theory can be potentially applied to various RL methods which involve value estimation, such as actor-critic methods.
M ODEL - BASED R EINFORCEMENT L EARNING
Highway Value Iteration.
From the new operator, we can naturally derive a new Value Iteration algorithm, (Algorithm B.1).
Specifically, for a finite S , the update using the Highway Operator G (cid:98) Π N
[Table]
[Table]
(eq. 3) can be written as
[Equation]
(6) where v k is a |S| × 1 column vector of VF; r a and r π are |S| × 1 column vectors of rewards for action a and policy π respectively, where [ r a ] s = r
( s, a ) , [ r π ] s = (cid:80) a π ( a | s ) r ( s, a ) .
T a
and T π are |S| × |S| matrices of transition probabilities for action a and policy π respectively, where [ T a ] s,s ′ =
T ( s ′ | s, a ) , [ T π ] s,s ′
= (cid:80) a π ( a | s ) T ( s ′ | s, a ) .
The computational complexity of each iteration is O (cid:16)(cid:16) |A|
+ (cid:12)(cid:12)(cid:12)(cid:98) Π (cid:12)(cid:12)(cid:12) |N| (cid:17) |S| 2 (cid:17) .
Two strategies can be adopted to accelerate the update process.
First, the matrix in PART 1 and PART 2 in eq.
(6) can be computed and reused for each π and n , as they are fixed during the iteration process.
Second, PART 3 can be computed in parallel for each policy π ∈ (cid:98) Π and n ∈ N .
4. O FF -P OLICY L EARNING IN MODEL - FREE R EINFORCEMENT L EARNING
In model-free RL, it is convenient to use Q instead of V .
The corresponding operator 3 is defined as
[Equation]
where G n +1 Q ( τ n +1 s 0 ,a 0 ) is the n + 1 -step return; τ n +1 s 0 ,a 0 ≜ ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , · · · , s n +1 ) .
This operator is also unbiased w.r.t. Q ∗ , i.e., G (cid:98) Π N Q ∗ =
Q ∗ (similar to Theorem 1, see APPENDIX Theoremfor a formal statement).
This means that it can utilize any off-policy data collected by arbitrary policy, without additional corrections.
We propose two methods for the tabular VF and VF approximation, named Highway Q-Learning and Highway DQN respectively.
Highway Q-Learning.
Let D ( m ) s 0 ,a 0 = { τ n +1 s 0 ,a 0 | τ n +1 s 0 ,a 0 ∼ π m } denote the trajectory data collected by the policy π m .
The k -th VF Q k is updated in the following way:
[Equation]
(8) where M s 0 ,a 0 ⊆ (cid:110) m (cid:12)(cid:12) |D
( m ) s 0 ,a 0 | ̸ = 0 (cid:111) is a subset of indexes of the dataset that are not empty under ( s 0 , a 0 ) ; (cid:98) E D ( m ) s 0 ,a 0 [ · ] = 1 (cid:12)(cid:12)(cid:12) D ( m
[Equation]
s 0 ,a 0 (cid:12)(cid:12)(cid:12) (cid:80) τ n +1 s 0 ,a 0 ∈D ( m [Equation] s
[Equation]
0 [ · ] is the empirical averaged value.
Note that all we need to do is saving the trajectory data into the corresponding datasets D ( m ) s 0 ,a 0 , and then search over them, without having to know the form of π m or save the π m into the set of behavioral policies (cid:98) Π .
The algorithm, Highway Q-Learning , is presented in Appendix B Algorithm B.3.
Highway DQN.
For large-scale space or continuous MDPs, the VF is usually approximated.
Due to the estimation noise involved in function approximation (Van Hasselt et al., 2016), our method may lead to overestimation by the two maximization operations (over policies and lookahead depths).
However, as we show below, this issue can be easily solved through a minor modification.
We propose a new variant of eq.
(7), named Softmax Highway Operator , as follows
[Equation]
where the softmax operator smax α with the temperature parameter α is defined as
[Equation]
smax α reduces to max when α → ∞ .
We have the following theorem.
Theorem 2 For any α , any (cid:98) Π , and any N , we have (cid:101) G (cid:98) Π N Q ∗ = Q ∗ and ( ∀ Q ∈ l ∞ ( S × A )) : ∥ (cid:101) G (cid:98) Π N Q − Q ∗ ∥ ≤ γ ∥ Q − Q ∗ ∥ .
The operator in eq.
(9) is derived in the following way.
First, the Highway Q Operator G (cid:98) Π N in eq.
(7) can be rewritten as an equivalent form, G (cid:98)
[Equation]
N
[Equation]
n ∈{ 0 ,n ′ } ( B π ) n B Q .
Then, we replace the first two max operators max π max n ′ with the softmax operators smax απ smax αn ′ .
The above modification of Softmax Highway Operator (cid:101) G (cid:98) Π N is necessary to a) remain unbiased w.r.t.
Q ∗ (as shown in APPENDIX Theorem 2) 4 ; b) alleviate the overestimation issue and improve exploration with the softmax operation, which has been shown effective in recent RL literature (Fox et al., 2015; Haarnoja et al., 2017; Schulman et al., 2017; Song et al., 2019).
Based on the above theoretically justified operator, we propose the following objective function for updating the action VF Q θ ( s, a ) parametrized by the parameter θ :
[Equation]
where Q θ ′ is the target network parametrized by θ ′ , occasionally copied from θ ;
(cid:98) D
= { ( s 0 , a 0 ) } ) is the sampled batch data of state-action ( s 0 , a 0 ) pairs.
The computational complexity of the method is biased implied by the equation above is close to the one of existing eligibility trace-based methods (Schulman et al., 2016; Munos et al., 2016), which also need to compute n -step returns G nQ for each n .
The resulting algorithm is called Highway DQN , presented in APPENDIX Algorithm B.4.
In practice, our algorithm balances the trade-off between accuracy and sample efficiency by deciding the number of trials per policy, the size of the search space (of behavioral policies and lookahead depths), and the softmax temperature.
While more trials per policy may improve the estimation accuracy, they may also cost more samples and reduce sample efficiency.
On the other hand, while a larger search space may increase efficiency, it might incur overestimation issues when the estimate is biased, leading to high variance.
In summary, our Highway Q-Learning and Highway DQN can recycle any trajectory data collected by some arbitrary policy, and utilize the multiple-step trajectory data that do not require Importance
Sampling-based corrections (as stated in Theorem 1 and 2).
T HEORETICAL A NALYSIS
In this section, we study the theoretical properties of Highway operator 5 G (cid:98) Π N and show its superiority over classical Bellman operators, e.g., B and B π .
For convenience, our analysis is in the space of state VF, i.e. , the operators G (cid:98) Π N , B , B π , . . .
are assumed to be mappings on l ∞ ( S ) .
First, we compare our Highway Operator G (cid:98) Π N to the Bellman Optimality Operator B .
Theorem 3 (Comparison to Bellman Optimality Operator B ) For all V, V 0 ∈ l ∞ ( S ) holds:
1) (cid:13)(cid:13)(cid:13) G (cid:98) Π N V − V ∗ (cid:13)(cid:13)(cid:13) ≤ ∥B V − V ∗ ∥ ; 2) Assume V 0 ≤ V ∗ . For any s we have (cid:12)(cid:12)(cid:12) G (cid:98) Π N V {+
+} ( s ) − V ∗ ( s ) (cid:12)(cid:12)(cid:12) ≤ |B V {+
+}
( s ) − V ∗ ( s ) | , where the strict inequality holds as long as there exists π ′ ∈ (cid:98) Π such that arg max n ∈N (cid:16) B π ′ (cid:17) n B V 0 ( s ) > 0 .
The first point of the theorem implies that our operator converges at the same rate as the Bellman
Optimality Operator in the worst case.
The second point shows a state-wise convergence comparison under the case of V 0 ≤ V ∗ .
Our operator generally converges faster than the Bellman Optimality
Operator as long as one behavioral policy finds a better path by looking forward for n steps ( n > 0 ).
Note that the condition V 0 ≤ V ∗ can be easily satisfied by setting V 0 = min s ′ ,a ′ r ( s ′ , a ′ ) .
Moreover, as long as V 0 ≤ V ∗ , we have ( G (cid:98) Π N ) ◦ k V 0 ≤ V ∗ for any k (see Lemma 2 in Appendix).
Then, we compare our Highway operator to Bellman Expectation operator B π under the framework of Generalized Policy Iteration (GPI) (see APPENDIX Algorithm B.2) (Sutton & Barto, 2018).
GPI interacts between policy improvement and policy evaluation by applying B π for N ′ times.
We show that our operator provides an optimal solution for the evaluation-improvement trade-off to GPI, by selecting the best N ′ ∈ N to approximate the optimal VF V ∗ at k − 1 -th iteration if V k − 1 ≤ V ∗ .
Theorem 4 (Relation with Bellman Expectation Operator B π under GPI framework) Let π V denotes a policy which is greedy on VF V ∈ l ∞ ( S ) .
Then the operator related to Highway Value Iteration (with hyperparameters K = 1 , N , refer to B.1) acts always better (in sense of value) than the operator related to GPI iteration (with hyperparameters N ′ ∈ N , refer to Algorithm B.2) given they both act on the same VF V ≤ V ∗ , i.e., (cid:13)(cid:13)(cid:13) G [Equation]
Π ∪ [Equation] [Table] [Equation]
[Table]
[Equation] [Table]
Next, we show that by assuming that some of the behavioral policies act optimally within a few time steps, we can achieve better convergence rates.
Assumption A ( (cid:98) Π , n ) Given set of behavioral policies (cid:98) Π , a lookahead depth n .
Let S π ′ s,n denote the set of all possible visited states by executing policy π ′ for n steps from state s .
For each s ∈ S , there exists at least one policy π ′ ∈ (cid:98) Π such that π ′ ( s ′ )
=
π ∗ ( s ′ ) for any s ′ ∈ S π ′ s,n where π ∗ refers to an optimal policy.
Note the quantification order: ( ∀ s ∈ S , ∃ π ′ ∈ (cid:98) Π ,
∀ s ′ ∈ S
π ′ s,n ) : π ′ ( s ′ )
=
π ∗ ( s ′ ) .
Theorem 5 (Better Convergence Rates) Assume that N − 1 ∈ N and (cid:98) Π satisfies Assumption A ( (cid:98) Π , N − 1) , then ∥G (cid:98) Π N V − V ∗ ∥ ≤ γ N ∥
V − V ∗ ∥ for all V ∈ l ∞ ( S ) , V ≤ V ∗ , i.e., the sequence ( G (cid:98) Π N ) k V 0 , k ∈ N converges to V ∗ R-linearly with the convergence rate γ N for all
V 0 ∈ l ∞ ( S ) , V 0 ≤ V ∗ .
This theorem implies that when the set of behavioral policies (cid:98) Π satisfies Assumption A ( (cid:98) Π , N − 1) , our operator G (cid:98) Π N has a convergence rate of γ N .
Note that, in Assumption A ( (cid:98) Π , N − 1) , (cid:98) Π is not required to include an optimal policy π ∗ .
Instead, it only requires that, for each state, there exists one policy that behaves well within a period ( N − 1 consecutive steps), and the well-behaved policy could be varying towards different starting from each state.
Note that this condition is much weaker than having some optimal or near-optimal policies included in the behavioral policy set.
Specifically, Assumption A ( (cid:98) Π ′ , 1) can be satisfied by constructing a (cid:98) Π ′ =
{ π a | a ∈ A , ( ∀ s ∈ S ) :
π a ( a | s ) = 1 } , yielding a convergence rate of γ 2 .
Note that although the operator ( B ) ◦ N also leads to a convergence rate of γ N , our operator max π ∈ (cid:98) Π max n ∈N ( B π ) ◦ n B differs essentially from it. A major difference is that the Bellman Expectation Operator B π is applied for n times instead of applying the Bellman Optimality Operator B for N times.
In the model-free case, ( B π ) ◦ n B Q can utilize the n -step trajectory data generated by π with minor cost (just by accumulating rewards within n -step, i.e., (cid:80) n ′ γ n ′ r t + n ′ + γ n max Q ( s t + n , a t + n ) ). While ( B ) ◦ N can only utilize the 1 -step data and needs to update the VF N times (implying by eq. 2).
R ELATED W ORK
Multi-step RL methods has been studied in RL for a long history, including multi-step SARSA (Sutton & Barto, 2018), Tree Backup (Precup, 2000), Q( σ ) (Asis et al., 2017), and Monte Carlo methods (which can be regarded as ∞ -step learning).
λ -return assign exponentially decaying weights depends on the decay factor λ (Sutton & Barto, 2018; Schulman et al., 2016; White & White, 2016).
Sahil et al.
(2017) proposed a more general form called weighted returns, which assigns weights to all n -step returns.
Roughly, the lookahead depth, the decay factor, and the weights represent the prior knowledge or bias regarding appropriate CA intervals, usually tuned in a case-by-case manner.
Our method instead adaptively adjusts the lookahead depth in line with the quality of the data and the learned VF.
Another similar work related to CA is RUDDER (Arjona-Medina et al., 2019), which trains an LSTM to re-assign credits to previous actions.
Our methods derive a simple but sound principle for transporting the credit with minor cost.
Existing off-policy learning methods usually use additional corrections for off-policy data.
Classical importance sampling (IS)-based methods suffer from high variance due to the products of IS ratios (Cortes et al., 2010; Metelli et al., 2018). Several variance reduction techniques have been proposed.
Munos et al.
(2016)’s
Retrace( λ ) reduces the variance by clipping the IS ratios, and has achieved great success in practice.
Other work ignores IS: TB( λ ) (Precup, 2000) corrects the off-policy data by computing expectations w.r.t.
data and estimated VF, using the probabilities of the target policy.
Harutyunyan et al.
(2016)’s Q( λ ) corrects the off-policy data by adding an off-policy correction term.
Our method provides an alternative IS-free tool for off-policy learning.
Similar to Retrace( λ ), it can safely use arbitrary off-policy data.
Moreover, it is very efficient, offering a faster convergence rate under mild conditions.
More research is needed, however, to understand how the variance of our method compares to that of advanced IS-based variance reduction methods like Retrace( λ ).
Searching over various policies and (or) lookahead depths to improve the convergence of RL systems is an active field. Barreto et al. (2020) search over various policies while using only a fixed lookahead depth until the end of the trial.
He et al.
(2017) first propose to search over various lookahead depths along the trajectory data, but they do not search over various policies.
Moreover, they employ the lookahead returns to construct additional inequality bounds on the VF. Our work instead contributes [Figure] 100 200 300 400 500 delay of reward 10 0 1010 2 10 3 1010e p
i s o d e s RUDDERQ( ) Monte Carlo Highway Q learning (our) (b) Choice 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 delay of reward 1010 2 1010 4 10e p i s o d e s RUDDERQ( ) SARSA( ) Highway Q Learning (our) [Figure]
to a novel Bellman operator on updating the VF directly and provides a thorough theoretical analysis of the convergence properties. We also address that greedy operations (Barreto et al., 2020; He et al., 2017) may cause overestimation, by proposing a novel softmax operation to alleviate this issue in an unbiased fashion.
Compared to previous methods searching over the product of original policy space and action space (Efroni et al., 2018; 2019; 2020; Jiang et al., 2018; Tomar et al., 2020), ours has a smaller search space (i.e., a limited set of policies). In summary, this paper contributes to a theoretically-justified Bellman operator which searches both policies and lookahead depths, leading to more flexibility and scalability against different settings and applications.
Our method can also be viewed as combining the best of both worlds: (1) direct policy search based on policy gradients (Williams, 1992) or evolutionary computation (Rechenberg, 1971) (where lookahead equals trial length—no runtime-based identification of useful subprograms and subgoals), and (2) dynamic programming-based (Bellman, 1957) identification of useful subgoals during runtime.
This naturally and safely combines best subgoals/sub-policies (Schmidhuber, 1991) derived from data collected by arbitrary previously encountered policies.
It can be viewed as a soft hierarchical chunking method (Schmidhuber, 1992) for rapid CA, with a sound Bellman-based foundation.
E XPERIMENTS
We designed our experiments to evaluate the algorithms under different cases and investigate their properties.
Please refer to Appendix C for additional details of the experiment settings.
Model-based Toy Tasks.
We first evaluate on a model-based task: Multi-Room environments with different number of rooms.
The agent needs to go through many rooms and reach the goal to get a reward.
We compare our Highway Value Iteration to classical Value Iteration (VI) and Policy
Iteration (PI).
The algorithms are evaluated until convergence.
As shown in the three plots of Fig.(a), our Highway Value Iteration outperforms VI and PI in terms of number of iterations required, total number of samples (total number of queries to the MDP model), and computation time.
Model-free Toy Tasks.
To evaluate credit assignment efficiency, we evaluate the algorithms on two toy tasks involving delayed rewards (Arjona-Medina et al., 2019), in which a reward is only provided at the end of each trial.
We compare our method to classical eligibility trace methods including Q( λ ),
SARSA( λ ), Monte Carlo methods; and also the advanced credit assignment method RUDDER.
As shown in Fig.
2 (b) and (c), our method significantly outperforms all competitors on both tasks.
For example, on Trace Back, our method requires only 20 episodes to solve the task, while the second best algorithm RUDDER requires more than 1000.
Notably, the costs of Highway Q-Learning do not observably increase with the reward delays.
In contrast, other methods such as Q( λ ) require exponentially increasing numbers of trials.
MinAtar Games.
[-SpaceInvaders    5 Timestep (×10 6 ) 1e
 Q V a l u e s SpaceInvaders [Figure]-]We evaluate our algorithms on benchmark tasks from MinAtar (Young & Tian, 2019).
[-1e650100150 s c o r e s SpaceInvaders    5 Timestep ( [Figure] 1e
-]We compare to several advanced multi-step off-policy methods, including Multi-Step DQN (Horgan et al., 2018; Barth-Maron et al., 2018) and Retrace( λ ) Munos et al.
All multi-step methods are implemented on top of Maxmin DQN (Lan et al., 2020), which chooses the smallest Qvalue among multiple target Q networks.
All the compared methods adopt the same implementations to ensure that we measure differences between algorithms, not implementations.
[Figure]
s c
o r e s
[Figure] Timestep
[Figure] 6 ) 1e [Figure] s c
o r e s
[Figure]
1e
[Figure]
s c
o r e s
[Figure] [Figure] Fig. 3 shows the performance of the algorithms.
Highway DQN significantly outperforms all competitors in terms of both reward and sample efficiency on almost all the tasks.
Compared to the advanced Retrace( λ ), our method performs significantly better on 3 of 5 tasks while performing on par with it on the remaining 2 tasks.
Ablation Study.
We conducted the following ablation studies to investigate properties of Highway
DQN.
(1) Lookahead depth ( N = { 0 , 1 , · · · , N − 1 } for Highway DQN and N for Multi-step DQN).
As shown in Fig.
4a, compared to Multi-step DQN, our Highway DQN shows strong robustness against variations of the lookahead depth.
This is because Highway DQN can adaptively choose the lookahead depth.
(2) Replay buffer size.
As shown in Fig.
4b, when the memory size increases to 8 × 10 5 (orange line), our Highway DQN shows a performance improvement, while Multi-Step
DQN shows a degradation.
For the results with varying softmax Temperature α and number of target networks, please refer to APPENDIX Appendix C.4 for more details.
C ONCLUSIONS
We introduced a novel multi-step Bellman Optimality Equation for efficient multi-step credit assignment (CA) in reinforcement learning (RL).
We proved that its solution is the optimal value function and that the corresponding policy adjustments generally converge faster than the traditional
Bellman Optimality operator.
Our Highway RL methods combine the best of direct policy search (where CA is performed without trying to identify useful environmental states or subgoals during runtime), and standard RL, which finds useful states/subgoals through dynamic programming.
Highway RL quickly and safely extracts useful sub-policies derived from data obtained through previously tested policies.
The derived algorithms have several advantages over existing off-policy algorithms.
Their feasibility and effectiveness were experimentally illustrated on a series of standard benchmark datasets.
Future work will theoretically analyze the algorithm’s behavior in the model-free case.
E THICS S TATEMENT
This paper provides a new operator for off-policy learning-based methods.
The authors do not find any particular concerns w.r.t the potential ethical or societal consequence.
R EPRODUCIBILITY S TATEMENT
The code is available at https://anonymous.4open.science/r/
Highway-Reinforcement-Learning-4202 and more implementation details can be found at Appendix B and C.
A T HEOREM P ROOFS A. P ROOF OF PROPERTIES OF H IGHWAY O PERATOR G (cid:98) Π N
Theorem 1 (Properties of Highway Operator)
For any (cid:98) Π and N (s t. 0 ∈ N ), we have 1) G (cid:98)
Π N is a contraction on complete metric space l ∞ ( S ) , i e. for any V, V ′ ∈ l ∞ ( S ) , we have∥G (cid:98) Π N V − G (cid:98) Π N V ′ ∥
≤ ∥B V − B V ∥
≤ γ ∥ V
−
V ′ ∥ ; 2) (Highway Bellman Optimality Equation, Highway Equation in short) V ∗ is the only fixed point
G (cid:98) Π N , that is, for all V ∈ l ∞ ( S ) holds V = G (cid:98) Π N V if and only if V = V ∗ .
Formally, we have
G (cid:98) Π N V ∗
≜ max
[Equation]
For any V 0 ∈ l ∞ ( S ) and any sequence of policy sets ( (cid:98) Π k ) , k ∈ N , the sequence ( G (cid:98)
Π
k N ◦ G (cid:98) Π k − 1 N ◦ . . .
◦ G (cid:98) Π 1 N )[ V 0 ] , k ∈ N converges R-linearly to V ∗ with convergence rate γ .
Proof: 1)
The contraction property can be obtained:
[Equation]
2) By application of the Banach fixed point theorem to point 1., the operator G (cid:98) Π N must have a unique fixed point.
Therefore, it suffices to verify that V ∗ is a fixed point of G (cid:98) Π N .
Our operator can be rewritten as
[Equation]
Using the fact that B π V ∗ ≤ B V ∗ =
V ∗ and B π monotony we obtain ( B π ) n B V ∗ ≤ V ∗ for any π and n ≥ 0 (with the equality when n = 0 ), which implies
[Equation]
3) From 1) and 2) follows:
∥ ( G (cid:98) Π
[Equation]
N ◦ G
[Equation]
] −
[Equation]
By repeating the same argument we end up with ∥
[Equation]
−
[Equation]
∥ from which the statement follows.
□
Theorem 2 For any α , any (cid:98) Π , and any N , we have (cid:101) G (cid:98) Π N Q ∗ = Q ∗ and ( ∀ Q ∈ l ∞ ( S × A )) : ∥ (cid:101) G (cid:98) Π N Q − Q ∗ ∥ ≤ γ ∥ Q − Q ∗ ∥ .
Proof:
[Equation]
This operator can be represented as
[Equation]
First, following the proof in Theorem 1, we have ( B π ) n B Q ∗ ≤ Q ∗ , for any π and n ≥ 0 (with equality when n = 0 ).
Then, we have
[Equation]
Given an action VF Q [Equation] let us define two distribution over (cid:98) Π and N , denoted by P s,a (cid:98) Π and P s,a N such that [Equation]
It follows
[Equation]
□
Corollary 1 Assume Q 0 ∈ l ∞ ( S × A ) and Π k , k ∈ N be a sequence of sets of behavioral policies then the sequence ( (cid:101) G (cid:98)
Π k N ◦ (cid:101) G (cid:98)
Π k − 1 N ◦ . . .
◦ (cid:101) G (cid:98)
Π 1 N )[ Q 0 ] , k ∈ N converges R-linearly to Q ∗ with convergence rate γ .
Proof: The proof follows from just proved point Theorem 2 and proceeds analogically to proof of
Theorem 1 point 3).
□
Theorem 3 (Comparison to Bellman Optimality Operator B ) For all V, V 0 ∈ l ∞ ( S ) holds:
1) (cid:13)(cid:13)(cid:13) G (cid:98) Π N V − V ∗ (cid:13)(cid:13)(cid:13) ≤ ∥B V − V ∗ ∥ ; 2) Assume V 0 ≤ V ∗ . For any s we have (cid:12)(cid:12)(cid:12) G (cid:98) Π N V {+
+} ( s ) − V ∗ ( s ) (cid:12)(cid:12)(cid:12) ≤ |B V 0 ( s ) − V ∗ ( s ) | ,
where
the strict inequality holds as long as there exists π ′ ∈ (cid:98) Π such that arg max n ∈N (cid:16) B π ′ (cid:17) n B V 0 ( s ) > 0 .
Proof: 1) has been proved in Theorem 1.
2) Note that G (cid:98) Π N V 0 ≥ B V 0 .
Further, if V 0 ≤ V ∗ , then G (cid:98) Π N V 0 ≤ V ∗ and B V 0 ≤ V ∗ (as they are both monotonic). Puting all togather we obtain [Equation]
from which the desired inequality follows.
□
Theorem 4 (Relation with Bellman Expectation Operator B π under GPI framework) Let π V denotes a policy which is greedy on VF V ∈ l ∞ ( S ) .
Then the operator related to Highway Value
Iteration (with hyperparameters K = 1 , N , refer to B.1) acts always better (in sense of value) than the operator related to GPI iteration (with hyperparameters N ′ ∈ N , refer to Algorithm B.2) given they both act on the same VF V ≤ V ∗ , i.e.,
[Equation]
Proof: First, for all V ∈ l ∞ ( S ) , N ′ ∈ N , (cid:98) Π a set of behavioral policies, and any s ∈ S holds
[Equation]
Now using V ≤ V ∗ with Lemma 2 and further Eq.
(14) we get
[Equation]
(15) from which the conclusion follows.
□
The purpose of the theorem is to show that one iteration of HVI performs always better then one iteration of GPI when acting on the same VF (given conditions).
Straightforward extension of the result to HVI and GPI generated sequences of VF fails because of missing monotony of the operator related to GPI iteration.
Assumption A ( (cid:98) Π , n ) Given set of behavioral policies (cid:98) Π , a lookahead depth n .
Let S π ′ s,n denote the set of all possible visited states by executing policy π ′ for n steps from state s .
For each s ∈ S , there exists at least one policy π ′ ∈ (cid:98) Π such that π ′ ( s ′ )
=
π ∗ ( s ′ ) for any s ′ ∈ S π ′ s,n where π ∗ refers to an optimal policy.
Note the quantification order: ( ∀ s ∈ S , ∃ π ′ ∈ (cid:98) Π ,
∀ s ′ ∈ S
π ′ s,n ) : π ′ ( s ′ )
=
π ∗ ( s ′ ) .
Theorem 5 (Better Convergence Rates) Assume that N − 1 ∈ N and (cid:98) Π satisfies Assumption A ( (cid:98) Π , N − 1) , then ∥G (cid:98) Π N V − V ∗ ∥ ≤ γ N ∥
V − V ∗ ∥ for all V ∈ l ∞ ( S ) , V ≤ V ∗ , i.e., the sequence ( G (cid:98) Π N ) k V 0 , k ∈ N converges to V ∗ R-linearly with the convergence rate γ N for all
V 0 ∈ l ∞ ( S ) , V 0 ≤ V ∗ .
Proof: 1) First, if (cid:98) Π satisfies assumption Assumption A ( (cid:98) Π , N − 1) , that is, starting from any s , there exists at least one π ∈ (cid:98) Π executing the optimal actions for N − 1 steps, then we have
[Equation]
Second, if V ≤ V ∗ , then G (cid:98) Π N V ≤ V ∗ ( G (cid:98) Π N is monotonic) and (cid:0) B
π ∗ (cid:1) N − 1 B V ≤ V ∗ .
Finally, with the results above, we have
[Equation]
□
To utilize the better convergence rates obtained in theorem 5, we have to be a bit more careful than in theorem 1 point 3) and take care of the monotony requirements in theorem 5.
This is done in the following lemma.
Lemma 1 Assume a sequence of operators T k , k ∈ N on L ∞ ( S ) , where all T k satisfy ( ∀ V ∈
L ∞ ( S ) , V ≤ V ∗ ) :
∥ T k V − V ∗ ∥ ≤ γ ′
∥
V − V ∗ ∥ with common convergence rate γ ′ and common limit V ∗ .
Further assume all T k are monotonic and have fixed point V ∗ .
Then for any V 0 ∈ l ∞ ( S ) , V 0 ≤ V ∗ the sequence ( T k ◦ T k − 1 ◦ . . .
T 1 )
[ V 0 ] , k ∈ N converges R-linearly to V ∗ with convergence rate γ ′ .
Proof: Assuming V 0 ∈ l ∞ ( S ) , V 0 ≤ V ∗ and monotony and fixed point of operators T 1 , . .
, T k −implies:
V :=
( T k − 1 ◦ T k − 2 ◦ . . .
T 1 )
[ V 0 ] ≤ V ∗ .
Now we can apply the inequality with γ ′ to get ∥ ( T k ◦ T k − 1 ◦ . . .
T 1 )
[ V 0 ] − V ∗ ∥ ≤ γ ′
∥
( T k − 1 ◦ . . .
T 1 )
[ V 0 ] − V ∗ ∥ .
By repeating the same argument we end up with ∥ ( T k ◦ T k − 1 ◦ . . . T 1 )
[ V 0 ] − V ∗ ∥ ≤ γ ′ k
∥ V 0 − V ∗ ∥ from which the statement follows.
□
Let us denote (cid:98) Π k ( k ∈ N ) the sequence of behavioral policies generated by Algorithm B.1.
In Algorithm B.1 the set of behavioral policies (cid:98) Π k changes by adding a greedy policy every K iterations.
Regardless of the sets (cid:98) Π k , the algorithm converges to the optimal value function form theorempoint 3) .
This means that more and more close to optimal policies are added to (cid:98) Π k set with growing iteration number k .
When assumption Assumption A ( · , n ) gets satisfied for some n at iteration k 0 , then due to monotony of the sequence (cid:98) Π k , k ∈ N (i.e. the set (cid:98) Π k can just grow over time)
Assumption A ( · , n ) is satisfied for the whole suffix (cid:98) Π k , k ≥ k 0 .
We can then use lemma 1 on corresponding suffix of operator sequence G (cid:98)
Π k N , k ≥ k 0 with γ ′ =
γ n +1 to claim better convergence rate ( γ n +1 ) of the corresponding suffix of value functions V k , k ≥ k 0 .
When S is finite this gives us monotonic improvement of the convergence rate of the sequence V k , k ∈ N as Assumption A ( · , n ) gets eventually satisfied for bigger and bigger n . Theorem 6 (Highway Bellman Optimality Equation (Highway Equation) w.r.t. action VF Q ) For any (cid:98) Π and N , we have
G (cid:98) Π N is a contraction on complete metric space l ∞ ( S × A ) ,
[-Lemma 
-]i e. for any Q, Q ′ ∈ l ∞ ( S × A ) , we have ∥G (cid:98) Π N Q − G (cid:98) Π N Q ′ ∥ ≤ ∥B Q − B Q ′ ∥ ≤ γ ∥ Q − Q ′ ∥ ; 2. Q ∗ is the only fixed point G (cid:98) Π N .
That is,
[Equation]
and for all Q ∈ l ∞ ( S × A ) holds Q = G (cid:98) Π N Q if and only if Q = Q ∗ . Proof: Note that G (cid:98) Π N w.r.t action VF Q can also be represented by the Bellman Operators w.r.t Q , i. e [Equation]
Except for the change of value spaces ( l ∞ ( S × A ) instead of l ∞ ( S ) ) the proof is the same as for Theorem 1).
{+□ Lemma 
 If V 0 ≤+} V ∗ , then ( G (cid:98) Π
k N ◦ G (cid:98) Π k − 1 N ◦ . . . ◦ G (cid:98) Π 1 N ) V 0 ≤ V ∗ for any k and any sequence of policy sets (cid:98) Π 1 , (cid:98) Π 2 , . . . , (cid:98) Π k . Proof: As G (cid:98) Π N is monotonic (for any set of behavioral policies (cid:98) Π ), we have ( G (cid:98) Π 1 N ) V 0 ≤ ( G (cid:98) Π 1 N ) V ∗ =
V ∗ .
Further, applying G (cid:98) Π 2 N again to the both sides we obtain ( G (cid:98) Π 2 N ◦ G (cid:98) Π 1 N ) V 0 ≤ ( G (cid:98) Π 2 N ) V ∗ =
V ∗ .
Repeting the argument further k − 2 times we get the result.
□
A. P ROOF OF PROPERTIES OF M ULTI - STEP B ELLMAN O PTIMALITY OPERATOR
Recent works combined multi-step trajectory data and estimated the values of the most promising actions without any correction for the off-policy data (Horgan et al., 2018; Barth-Maron et al., 2018). The underlying operator is defined as
[Equation] where P (cid:98) Π is a distribution over (cid:98) Π and P (cid:98) Π ( π ) is the probability of selecting π . Here we use π ∼ P (cid:98) Π and τ Ns 0 ,a 0 ∼ π to formulate the procedure of prioritized experience replay (Schaul et al., 2015; Horgan et al., 2018), which samples the trajectory data collected by various behavioral policies according to a prior distribution.
[-Lemma 
-]Although these methods have shown promising results in practice, below we will show that this operator is generally biased w.r.t. the optimal action VF Q ∗ . In other words, the corresponding fixed point Q ∗B (cid:98) Π N is different from the optimal VF Q ∗ , and unbiased learning only happens when all behavioral policies are optimal.
Theorem 7 (Properties of the Multi-step Bellman Optimality Operator B (cid:98) Π N ) For any N ≥ 2 , 1) The operator B (cid:98) Π N is a contraction on complete metric space l ∞ ( S × A ) , i.e., for any two vectors Q,
[Equation]
2) Let Q ∗B (cid:98) Π N denote the fixed point of B (cid:98) Π N ,
i
e., B (cid:98) Π N Q ∗B (cid:98) Π N = Q ∗B (cid:98) Π N , we have Q ∗B (cid:98) Π N ≤ Q ∗ . 3) Q ∗B (cid:98) Π N = Q ∗ if and only if any π ∈ (cid:98) Π , P (cid:98) Π ( π ) > 0 satisfies π ( s ) =
π ∗ ( s ) for any s ∈ U := { s 1 ∈ S|∃ s 0 , a 0 , T ( s 1 | s 0 , a 0 ) > 0 } and π ∗ an optimal policy.
Before giving the proof of the above theorem, we’d like to mention a variant of our Highway Operator, named Expectation Highway Operator , defined as
[Equation]
Compared to the above Multi-Step Bellman Optimality Operator in eq. (17), our operator uses maximization over lookahead depths instead of a fixed lookahead depth. It’s interesting to note that this operator is unbiased w.r.t.
Q ∗ for any set of behavioral policies (cid:98) Π . Formally, we have the following theorem.
Theorem 8 For any (cid:98) Π , any P (cid:98) Π , and any N , 1) G N (cid:98) Π is a contraction on complete metric
space
l ∞ ( S × A ) , i
e
for
any two vectors Q, Q ′ ∈ l ∞ ( S × A ) , ( ∀ Q ∈ l ∞ ( S × A )) : ∥G N (cid:98) Π Q − G N (cid:98) Π Q ′ ∥ ≤ γ ∥ Q − Q ′ ∥ ; 2) G N (cid:98) Π Q ∗ = Q ∗ .
We now give the proofs of Theorem 7 and 8 respectively.
Proof of Theorem 7:
For simplicity, we will use P instead of P (cid:98) Π .
[Equation]
2) Applying Banach’s fixed point theorem to B (cid:98) Π N using the contraction result, we know that B (cid:98) Π N has only one fixed point. Following the proof in Theorem 1, we have ( B π ) N − 1 B Q ∗ ≤ Q ∗ for any π and N ≥ 2 .
This implies that
[Equation]
Using monotony of B (cid:98) Π N ( Q ≤ Q ′ implies B (cid:98) Π N Q ≤ B (cid:98) Π N Q ′ ), we get the monotonic sequence Q ∗ ≥
B (cid:98) Π N Q ∗ ≥ ( B (cid:98) Π N ) 2 Q ∗ ≥ · · · which converges to the fixed point Q ∗B (cid:98) Π N based on the contraction property and Banach fixed point theorem :
[Equation]
For the implication ” ⇐ = ”it suffices to show for all π ∈ (cid:98) Π , P ( π ) > 0 B π Q ∗ =
Q ∗ .
Since then we get
[Equation]
and the implication is proved.
Thus let us fix π ∈ (cid:98) Π , P ( π ) > 0 and s 0 ∈ S , a 0 ∈ A the following holds B π Q ∗ ( s 0 , a 0 ) =
E s 1 ∼T ( ·| s 0 ,a 0 ) E a 1 ∼ π ( ·| s 1 )
[ r ( s 0 , a 0 ) + γQ ∗ ( s 1 , a 1 )] .
Since in the first expectation we just care about s 1 for which T ( s 1 | s 0 , a 0 ) > [-
 but then-]0 , we can assume s 1 ∈ U .
As π on U can be replaced by the optimal policy π ∗ from the assumption, we get
[Equation]
The remaining implication ” =
⇒ ” will be proved by contradiction.
Assume that the conclusion does not hold, i.e. there exist π ∈ (cid:98) Π , P ( π ) > 0 and s 1 ∈ U such that P ( π ) > 0 and π ( ·| s 1 ) is not optimal.
Since s 1 ∈ U there exists s 0 ∈ S , a 0 ∈
A such that T ( s 1 | s 0 , a 0 ) >
First we aim to prove inequality
[Equation]
Since π ( ·| s 1 ) assigns positive probability to non-optimal action, it is easy to obtain (especially for finite A ) that
[Equation]
For other states different from s 1 we can still have equality but the countable sum leaves the inequality strict
[Equation]
Now since ( B π ) N − 2 Q ∗ ≤ Q ∗ we get
[Equation]
Using previous result we obtain
[Equation]
0 ) < Q ∗ ( s 0 , a 0 ) , and applying expectation with finite set (cid:98) Π using P ( π ) >
we get
B (cid:98)
[Equation]
N Q ∗ ( s 0 , a 0 ) < Q ∗ ( s 0 , a 0 ) , which can be combined with eq. (20) to show [Equation]
□ We now give proof of Theorem 8. Proof of Theorem 8: 1) We first prove the contraction property,
[Equation]
2) Similar to the proof in Theorem 1, ( B π ) n B Q ∗ ≤ Q ∗ for any π and n ≥ 0 (and the equality holds when n = 0 ), we have [Equation] A. D ISCUSSION ABOUT S OFTMAX H IGHWAY O PERATOR The Softmax Highway Operator is defined by
[Equation]
If we remove the max n ∈{ 0 ,n ′ } in the above operator, i.e.,
[Equation]
then the above operator is biased w.r.t.
Q ∗ .
This operator can be regarded as an extension to the multi-step Bellman Optimality Operator B (cid:98) Π n , average over various n with weights smax α n ′ ∈N ( · ) . Therefore, it has similar biased property to B (cid:98) Π n . However, simply adding max n ∈{ 0 ,n ′ } with minor computational cost, our Softmax Highway operator is unbiased while alleviating the overestimation issue and improving the exploration.
A. M ULTI -S TEP I MPORTANCE S AMPLING - BASED B ELLMAN E XPECTATION O PERATOR
In this section, we describe the classical off-policy learning method based on importance sampling (IS). IS-based off-policy methods evaluate the value function of a policy π ′ (called target policy ) using the data collected by a different policy π (called behavior policy ). The underlying operator, called Importance Sampling-based Bellman Expectation Operator (Sutton & Barto, 2018), is defined as follows,
[Equation] To utilize the multi-step data collected by different behavior policies, the above operator can be extended to a multi-step version with a set of behavioral policies (we call Multi-Step IS-based Bellman Expectation Operator ) (Sutton & Barto, 2018), which is defined as [Equation]
where N is the lookahead depth/bootstrapping step; τ Ns 0 = ( s 0 , a 0 , s 1 , a 1 , s 2 , a 2 , · · · , s N ) ; τ Ns 0 ∼ π is the trajectory starting from s 0 by executing policy π for N steps; and ζ 1: t ≜ (cid:81) tt ′ = 1 π ′ ( a t ′ | s t ′ ) π ( a t ′ | s t ′ ) is the product of IS ratios. The products of IS ratios could cause high variance.
B M ETHOD B. A LGORITHMS B.1. H IGHWAY V ALUE I TERATION
The Highway Value Iteration algorithm is presented as Algorithm B.1.
B.1. H IGHWAY Q-L EARNING
The Highway Q-Learning algorithm is presented as Algorithm B.3. [Figure]
B.1. H IGHWAY DQN
The Highway DQN algorithm is presented as Algorithm B.4.
C E XPERIMENTAL R ESULTS
The code of this paper is publicly-available at https://anonymous.4open.science/r/
Highway-Reinforcement-Learning-4202 .
C. E XPERIMENTS WITH M ODEL - BASED A LGORITHMS C.1. D ETAILS OF E NVIRONMENTS
Multi-Room is a grid world environment with multiple rooms connected by doors.
The agent’s goal is to reach a goal square in the opposite corner and get a reward ( r = 1000 ).
In addition, the agent will get a small reward r = 0 .
001 when it finds the exit door of the room.
We use the implementations based on the gym-minigrid (Chevalier-Boisvert et al., 2018). Algorithm
B
.1 Highway Value Iteration
Input: Initial set of behavioral policies (cid:98) Π 0 ; the set of lookahead depths N ; interval for adding new policy K .
Initialize:
Initial VF V 0 ∈ R |S| , ϵ . for k
= 1 , 2 , . .
do if ( k − 1) mod K ==
then
π k ( s ) =
arg max a [ r ( s, a ) + γ E s ′
[ V k − 1 ( s ′ )]] (cid:98)
Π k = (cid:98) Π k − 1 ∪ { π k } end
if
V k ← G (cid:98) Π k N V k −if ∥
V k − V k − 1 ∥ ∞ ≤ ϵ then break end if end for
Input: the lookahead depth N ′ .
Initialize:
Initial VF V 0 ∈ R |S| , ϵ . for k = 1 , 2 . . .
do π k ( s ) = arg max a [ r ( s, a ) + γ E s ′
[ V k − 1 ( s ′ )]]
V k ← ( B π k ) ◦ N ′ V k −if ∥ V k − V k − 1 ∥ ∞ ≤ ϵ then break end if end
1: Input: Set of lookahead depths N ; Number of behavior policies M (for computing target Q value); Epochs of running algorithm I run ; M ( computing searched I A
2: Initialize action 3: for m = 1 , · · · , I run do 4: Set π m to be ϵ -greedy policy with Q k 5: D ( m ) s,a ← ∅ , for all ( s, a ) ∈ S ×
A 6: for j = 1 , · · · , I rollout do 7: Collect a trajectory τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , · · · , s T ) with π m . 8: for t = 0 , 1 , · · · , T − 1 do 9: Add ( s t , a t , r t , s t +1 , a t +1 , r t +1 , · · · , s T ) to D ( m ) s t ,a t 10:
Add ( s t , a t ) to D 11: end for 12: end for 13: for j = 1 , · · · , I update do 14: Sample D . 15: Update the value function by the following rules
Q k +1 ( s 0 , a 0 ) =
max m ∈ M s 0 ,a 0 max n ∈N (cid:98) E D ( m ) s 0 ,a 0 (cid:2)
G n +1 Q k ( τ n +1 s 0 ,a 0 ) (cid:3) where (cid:98) E D ( m ) s 0 ,a 0 [ · ] = D ( m ) s,a (cid:88) r n +1 s 0 ,a 0 ∈D ( m ) s 0 ,a[ · ] M s 0 ,a 0 = { m 1 , · · · , m M } , m i ∼ Uniform (cid:16)(cid:110) m ′ (cid:12)(cid:12) |D
( m ′ )
s 0 ,a 0 | ̸ = 0 (cid:111)(cid:17) 16: k = k +17: end for 18: end for
C.1. D ETAILS OF A LGORITHMS
We compare our Highway Value Iteration to Policy Iteration and Value Iteration. For Policy Iteration, the lookahead step is set by N ′ = 10 (see Algorithm B.2). For our Highway Value
Iteration method, the interval of adding policy is set by K = 7 ; the set of lookahead depths N = { 0 , 1 , 2 , · · · , 9 } ; the size of behavioral policies is set by | (cid:98) Π | = 5 . The error bound ( ∥ V k − V k − 1 ∥ ∞ ≤ ϵ ) for all algorithms is set by ϵ = 10 − 10 . C. E XPERIMENTS OF M ODEL - FREE A LGORITHMS IN T OY T ASKS
Here we present the details of model-free-algorithms in artificial tasks.
We adopt the experimental setting of Arjona-Medina et al.
Input: Set of lookahead depths N ; Number
[Table]
M
[Table]
; Epochs
[Table]
I run ; Epochs
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Temperature
Initialize: State-action replay buffer D =
Initialize parameter θ of target Q function, θ ′ ← θ .
for m = 1 , · · · , I run do Set π m to be ϵ -greedy policy with Q θ
D ( m ) s,a ← ∅ , for all ( s, a ) ∈ S
×
A for j = 1 , · · · , I rollout do Collect a trajectory τ =
( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , · · · , s T ) with π m for t = 0 , 1 , · · · , T − 1 do
Add ( s t , a t , r t , s t +1 , a t +1 , r t +1 , s t +2 , a t +2 , · · · , s T ) to D ( m ) s t ,a t Add ( s t , a t ) to D end for end for for j = 1 , · · · , I update do
Sample a mini-batch (cid:98) D
= { ( s, a ) } from D .
Update θ by minimizing the following loss function:
[Equation]
(23) where (cid:98) E D ( m ) s 0 ,a 0 [ · ] =
D ( m ) s,a
[Equation]
end timesteps end for
C.2. D ETAILS OF E NVIRONMENTS
We evaluate the model-free algorithms on two toy tasks involving delayed rewards (Arjona-Medina et al., 2019), where a reward is only provided at the end of each trial and is associated with the previous actions.
For example, in task “Trace Back,” the final reward depends on the first two actions.
Each task is run with 100 random seeds.
In task “Choice,” the stochastic reward depends on the first action at the beginning; the final reward depends on the first two actions.
Please refer to
Arjona-Medina et al.
for more details.
C.2. A LGORITHMIC D ETAILS
The following methods are compared:
• RUDDER with reward redistribution for Q -value estimation, and RUDDER applied on top of Q -learning.
• Q -learning with eligibility traces according to Watkins ( Q ( λ ) ).
• SARSA with eligibility traces (SARSA( λ )).
• Monte Carlo.
For RUDDER, we use the default setting of Arjona-Medina et al.
For Q( λ ) and SARSA( λ ), the hyperparameter of eligibility traces is λ = 0 .
For Q( λ ), we use Watkins’ implementation.
The algorithms are evaluated until the task is solved.
For MC, Q-values are the exponential moving average of the episode return.
In all experiments, an ϵ -greedy policy with ϵ = 0 .
2 is adopted.
For our Highway Q-Learning, the set of lookahead depths N is set by { 0 , 1 , 2 , · · · , N − 1 } , where N = T − t is set to be the length until the end ( T is the length of trajectory and t the timestep of current state).
Epochs of rolling-out policy is set by I rollout = 1 .
Epochs of updating value function is set by I update = T .
C. E XPERIMENTS OF M ODEL - FREE A LGORITHMS IN M IN A TAR T ASKS C.3. E NVIRONMENTAL D ETAILS
This paper uses 5 games in MinAtar (Young & Tian, 2019), including Asterix, Breakout, Freeway, Seaquest, and Space Invaders.
The details of the environments can be found at https: //github.com/kenjyoung/MinAtar .
C.3. D ETAILS OF A LGORITHMS
We implement our and competing methods by extending the implementation of Maxmin DQN Lan (2019). The hyperparameters of training deep neural network are provided in Table 2. We reuse the hyper-parameters and settings of neural networks in the Maxmin DQN paper (Lan et al., 2020). For Maxmin DQN, the best number of target networks (a hyperparameter) was chosen from [2 , 3 , 4 , 5 , 6 , 7 , 8 , 9] and the best learning rates were chosen from [3 × 10 − 3 ; 3 × 10 − 4 ; 3 × 10 − 5 ] .
The set of lookahead depths of our Highway DQN is set by { 0 , 1 , · · · , N − 1 } , and the lookahead depth/bootstrapping step of Multi-Step DQN and Retrace( λ ) is set by N . For our Highway DQN, Multi-Step DQN and Multi-Step SARSA, the best hyperparameter N was chosen from [4 , 8 , 12] ; the best learning rate from [3 × 10 − 3 ; 3 × 10 − 4 ] , and the best number of target networks was chosen from [2 , 4 , 6] . For Retrace( λ ), λ is set by 1 according to the suggestion of Retrace( λ ) paper (Munos et al., 2016).
For our Highway DQN, the best hyperparameter of softmax temperature α was chosen from { 0 . 1 , 0 . 5 } . Epochs of rolling-out policy is set by I rollout = 1 . In practice, we adopt several measures to accelerate the computation of Highway DQN. We cache the Q values generated by the target network for data ( s, a ) such that they can be reused when the data is sampled again for training until the target network is updated.
With the cached Q values, all we need is a softmax over |N| × |K| numbers, which is typically fast on GPUs. Hyperparameter Value Optimizer
RMSprop Batch size Gradient Clipping
Gym:
MinAtari: Target network update frequency Gym:
MinAtari:
Hidden units of Q network
Gym: Fully-connected layer (64,64)
MinAtari:
Conv.
layer(out channels=16, kernel size=3, stride=1)
Conv.
layer(out channels=16, kernel size=3, stride=1)
Activation function
ReLU
Buffer size
{+Gym:
+}
{+MinAtari:
+}
Discount factor γ 0.
Exploration rate ϵ statr:1.0; end: 0.1; decay type: linear
[Figure]
C. A DDITIOANL RESULTS ON A BLATION S TUDY
We evaluate our Highway DQN with varying softmax temperature α and number of target networks.
As shown in Fig.
6a and Fig.
6b, appropriate softmax temperature α and number of target networks can reduce the overestimation of Q values and benefit the performance of the algorithm.