T WO -T IMESCALE N ETWORKS
FOR
N ONLINEAR
V ALUE F UNCTION A PPROXIMATION
A BSTRACT
A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control.
Many of the algorithms for learning values, however, are designed for linear function approximation—with a ﬁxed basis or ﬁxed representation.
Though there have been a few sound extensions to nonlinear function approximation, such as nonlinear gradient temporal difference learning, these methods have largely not been adopted, eschewed in favour of simpler but not sound methods like temporal difference learning and Q-learning.
In this work, we provide a two-timescale network (TTN) architecture that enables linear methods to be used to learn values, with a nonlinear representation learned at a slower timescale.
The approach facilitates the use of algorithms developed for the linear setting, such as data-efﬁcient least-squares methods, eligibility traces and the myriad of recently developed linear policy evaluation algorithms, to provide nonlinear value estimates.
We prove convergence for TTNs, with particular care given to ensure convergence of the fast linear component under potentially dependent features provided by the learned representation.
We empirically demonstrate the beneﬁts of TTNs, compared to other nonlinear value function approximation algorithms, both for policy evaluation and control.
I NTRODUCTION
Value function approximation—estimating the expected returns from states for a policy—is heavily reliant on the quality of the representation of state.
One strategy has been to design a basis—such as radial basis functions (Sutton and Barto, 1998) or a Fourier basis (Konidaris et al., 2011)—for use with a linear function approximator and temporal difference (TD) learning (Sutton, 1988).
For low-dimensional observation vectors, this approach has been effective, but can be onerous to extend to high-dimensional observations, potentially requiring signiﬁcant domain expertise.
Another strategy has been to learn the representation, such as with basis adaptation or neural networks.
Though there is still the need to specify the parametric form, learning these representations alleviates the burden of expert speciﬁcation.
Further, it is more feasible to scale to high-dimensional observations, such as images, with neural networks (Mnih et al., 2015; Silver et al., 2016).
Learning representations necessitates algorithms for nonlinear function approximation.
Despite the deﬁciencies in speciﬁcation for ﬁxed bases, linear function approximation for estimating value functions has several beneﬁts over nonlinear estimators.
They enable least-squares methods, which can be much more data-efﬁcient for policy evaluation (Bradtke and Barto, 1996; Szepesvari, 2010; van Seijen and Sutton, 2015), as well as robust to meta-parameters (Pan et al., 2017).
Linear algorithms can also make use of eligibility traces, which can signiﬁcantly speed learning (Sutton, 1988; Dann et al., 2014; White and White, 2016), but have not been able to be extended to nonlinear value function approximation.
Additionally, there have been a variety of algorithms derived for the linear setting, both for on-policy and off-policy learning (Sutton et al., 2009; Maei, 2011; van Seijen and Sutton, 2014; van Hasselt et al., 2014; Mahadevan et al., 2014; Sutton et al., 2016; Mahmood et al., 2017).
These linear methods have also been well-explored theoretically (Tsitsiklis and Van Roy, 1997; Maei, 2011; Mahmood and Sutton, 2015; Yu, 2015) and empirically (Dann et al., 2014; White and White, 2016), with some insights into improvements from gradient methods (Sutton et al., 2009), true-online traces (van Seijen and Sutton, 2014) and emphatic weightings (Sutton et al., 2016).
These algorithms are easy to implement, with relatively simple objectives.
Objectives for nonlinear value function approximation, on the other hand, can be quite complex (Maei et al., 2009), resulting in more complex algorithms (Menache et al., 2005; Di Castro and Mannor, 2010; Bhatnagar et al., 2013) or requiring a primal-dual formulation as has been done for control (Dai et al., 2017).
In this work, we pursue a simple strategy to take advantage of the beneﬁts of linear methods, while still learning the representation.
The main idea is to run two learning processes in parallel: the ﬁrst learns nonlinear features using a surrogate loss and the second estimates the value function as a linear function of those features.
We show that these Two-timescale Networks (TTNs) converge, because the features change on a sufﬁciently slow scale, so that they are effectively ﬁxed for the fast linear value function estimator.
Similar ideas have previously been explored for basis adaptation, but without this key aspect of
TTNs—namely the separation of the loss for the representation and value function.
This separation is critical because it enables simpler objectives—for which the gradient can be easily sampled—to drive the representation, but still enables use of the mean squared projected Bellman error (MSPBE)—on which all the above linear algorithms are based.
This separation avoids the complexity of the nonlinear
MSPBE, but maintains the useful properties of the (linear) MSPBE.
A variety of basis adaptation approaches have used a two-timescale approach, but with the same objective for the representation and the values (Menache et al., 2005; Di Castro and Mannor, 2010; Bhatnagar et al., 2013; J et al., 2016).
Yu and Bertsekas (2009) provided algorithms for basis adaptation using other losses, such as Bellman error using Monte carlo samples, taking derivatives through ﬁxed point solutions for the value function.
Levine et al.
(2017) periodically compute a closed form least-squares solution for the last layer of neural network, with a Bayesian update to prevent too much change.
Because these methods did not separate the value learn and basis adaptation, the resulting algorithms are more complex.
The strategy of using two different heads—one to drive the representation and one to learn the values—has yet to be systematically explored.
We show that TTNs are a promising direction for nonlinear function approximation, allowing us to leverage linear algorithms while retaining the ﬂexibility of nonlinear function approximators.
We ﬁrst discuss a variety of possible surrogate losses, and their potential for learning a useful representation.
We then show that TTNs converge, despite the fact that a linear algorithm is used with a changing representation.
This proof is similar to previous convergence proofs for policy evaluation, but with a relaxation on the requirement that features be independent, which is unlikely for learned features.
We then show empirically that TTNs are effective compared to other nonlinear value function approximations and that they can exploit several beneﬁts of linear value approximations algorithms.
In particular, for both low-dimensional and high-dimensional (image-based) observations, we show (a) the utility of least-squares (or batch) methods, (b) advantages from eligibility traces and (c) gains from being able to select amongst different linear policy evaluation algorithms.
We demonstrate that
TTNs can be effective for control with neural networks, enabling use of ﬁtted Q-iteration within
TTNs as an alternative to target networks.
B ACKGROUND
We assume the agents act in a ﬁnite Markov Decision Process (MDP), with notation from (White, 2017).
The dynamics of the MDP are deﬁned by the 3 -tuple ( S , A , P ) , where S is the set of states,
A the set of actions and P : S ×
A × S (cid:55)→ [0 , 1] the transition probability function.
The task in this environment is deﬁned by a reward function R : S ×
A × S (cid:55)→ R and a discount function γ : S ×
A × S (cid:55)→ [0 , 1] .
At each time step, the agent takes an action A t according to a policy π : S × A (cid:55)→ [0 , 1] and the environment returns reward R t +1 , next state S t +1 and discount γ t +1 .
The goal in policy evaluation is to compute the value function: the expected sum of discounted rewards from every state under a ﬁxed policy π .
The value function V π : S → R is deﬁned recursively from each state s ∈ S as
[Equation]
When using linear function approximation [Equation] this goal translates into ﬁnding parameters w ∈ R d to approximate the value function
[Equation]
where x : S → R d is a feature function.
[Equation]
More generally, a nonlinear function ˆ V ( s ) could be learned to estimate V π .
To formulate this learning problem, we need to consider the objective for learning the function ˆ V .
Let V π , ˆ V ∈ R |S| be the vectors of values for V π , ˆ V .
The recursive formula (1) deﬁnes a Bellman operator B π where the ﬁxed point satisﬁes
B π V π
= V π .
Consider a restricted value function class, such as the set of linear value functions ˆ V ∈ F =
{ Xw | w ∈ R d } where X ∈ R
|S|×
d is a matrix with the i -th row set to x ( s ) for i th state s ∈ S .
Then, it may no longer be possible to satisfy the recursion.
Instead, an alternative is to ﬁnd a projected ﬁxed point Π F B
π ˆ V
=
ˆ V where the projection operator Π F projects B π ˆ V to the space spanned by this linear basis:
[Equation]
(3) where d ∈ R |S| is a vector which weights each state in the weighted norm (cid:107) V (cid:107)
2 d = (cid:80) s ∈S d ( s ) V ( s ) 2 .
Many linear policy evaluation algorithms estimate this projected ﬁxed point, including TD (Sutton, 1988), least-squares TD (Bradtke and Barto, 1996) and gradient TD (Sutton et al., 2009).
The objective formulated for this projected ﬁxed-point, however, is more complex for nonlinear function approximation.
For linear function approximation, the projection operator simpliﬁes into a closed form solution involving only the features X .
Letting δ t =
R t +1 + γ ˆ V ( S t +1 )
− ˆ V ( S t ) , the resulting mean-squared projected Bellman error (MSPBE) can be written as
[Equation]
where E [ δ t x t ] = (cid:80) s ∈S d ( s ) E [ δ t | S t
= s ] x ( s ) .
For nonlinear function classes, the projection does not have a closed form solution and may be expensive to compute.
Further, the projection involves the value function parameters, so the projection changes as parameters change.
The nonlinear MSPBE and resulting algorithm are more complex (Maei et al., 2009), and have not seen widespread use.
Another option is simply to consider different objectives.
However, as we discuss below, other objectives for learning the value function either are similarly difﬁcult to optimize or provide poor value estimates.
In the next section, we discuss some of these alternatives and introduce Two-timescale
Networks as a different strategy to enable nonlinear value function approximation.
T WO - TIMESCALE N ETWORKS AND S URROGATE O BJECTIVES
We ﬁrst introduce Two-timescale Networks (TTNs), and then describe different surrogate objectives that can be used in TTNs.
We discuss why these surrogate objectives within TTNs are useful to drive the representation, but are not good replacements for the MSPBE for learning the value function.
TTNs use two concurrent optimization processes: one for the parameters of the network θ and one for the parameters of the value function w .
The value function is approximated as ˆ V ( s ) def = x θ ( s ) (cid:62) w where the features x θ : S → R d are a parametrized function and θ ∈ R m is adjusted to provide better features.
For a neural network, θ consists of all the parameters in the hidden layers, to produce the ﬁnal hidden layer x θ ( s ) .
The two optimization processes maintain different time scales, with the parameters θ for the representation changed as a slow process, and the parameters w for the value estimate changed as a fast process relative to θ .
The separation between these two processes could be problematic, since the target problem— estimating the value function—is not inﬂuencing the representation!
The slow process is driven by a completely separate objective than the fast process.
However, the key is to select this surrogate loss for the slow process so that it is related to the value estimation process, but still straightforward to compute the gradient of the loss.
We use ˆ V ( s ) as the output of the fast part, which corresponds to the value estimate used by the agent.
To distinguish, ˆ Y ( s ) denotes the output for the slow-part (depicted in Figure 1), which may or may not be an estimate of the value, as we discuss below.
Consider ﬁrst the mean-squared TD error (MSTDE), which corresponds to (cid:80) s ∈S d ( s ) E [ δ 2 t | S t
= s ] .
Notice that this does not correspond to the mean-squared Bellman error (MSBE), for which it is more difﬁcult to compute gradients (cid:107) B π ˆ V − ˆ V (cid:107)
d = (cid:80) s ∈S d ( s ) ( E [ δ t | S t
= s ]) 2 .
Using the MSTDE as a surrogate loss, with ˆ Y ( s ) =
x θ ( s ) (cid:62) ¯ w , the slow part of the network minimizes
L slow ( θ )= min ¯ w ∈ R d (cid:88)
[Equation]
This slow part has its own weights ¯ w associated with estimating the value function, but learned instead according to the MSTDE.
The advantage here is that stochastic gradient descent on the
MSTDE is straightforward, with gradient δ t ∇ { θ , ¯ w } [ γ t
+1 ˆ Y
( S t +1 )
− ˆ Y ( S t )] where ∇ { θ , ¯ w } ˆ Y ( S t ) is the gradient of the neural network, including the head of the slow part which uses weights ¯ w .
Using the MSTDE has been found to provide worse value estimates than the MSPBE—which we re-afﬁrm in our experiments.
It could, nonetheless, play a useful role as a surrogate loss, where it can inform the representation towards estimating values.
[Figure]
There are a variety of other surrogate losses that could be considered, related to the value function.
However, many of these losses are problematic to sample incrementally, without storing large amounts of data.
For example, the mean-squared return error (MSRE) could be used, which takes samples of return and minimizes mean-squared error to those sampled returns.
Obtaining such returns requires waiting many steps, and so delays updating the representation for the current state.
Another alternative is the MSBE.
The gradient of the nonlinear MSBE is not as complex as the gradient of the nonlinear MSPBE, because it does not involve the gradient of a projection.
However, it suffers from the double sampling problem: sampling the gradient requires two independent samples.
For these reasons, we explore the MSTDE as the simplest surrogate loss involving the value function.
Finally, surrogate losses could also be deﬁned that are not directly related to the value function.
Two natural choices are losses based on predicting the next state and reward.
The output of the slow part could correspond to a vector of values, such as Y t =
S t +1 ∈ R n or Y t =
(cid:104) S t +1 R t +1 (cid:105) .
The ability to predict the next state and reward is intuitively useful for enabling prediction of value, that also has some theoretical grounding.
Szepesvari (2010, Section 3.2.1) shows that the Bellman error is small, if the features can capture a horizon of immediate rewards and expected next states.
For linear encoders,
Song et al.
(2016) show that an optimal set of features enables predictions of next state and reward.
More generally, learning representations using auxiliary tasks or self-supervised tasks have had some successes in RL, such as using pixel control (Jaderberg et al., 2016) or classifying the temporal distance between frames (Aytar et al., 2018).
In computer vision, Gidaris et al.
(2018) showed that using rotated images as self-supervised tasks produced a useful representation for the main loss, without training the representation with the main loss.
Any of these self-supervised tasks could also be used for the surrogate objective, and motivate that separating out representation learning does not degrade performance.
For now, we restrict focus on simpler surrogate objectives, as the main purpose of this work is to demonstrate that the separation in TTNs is a sound approach for learning values.
C ONVERGENCE OF T WO -T IMESCALE N ETWORK A LGORITHM
Training TTNs is fully online, using a single transition from the environment at a time.
Projected stochastic gradient descent is used to reduce the surrogate loss, L slow ( θ ) and a linear policy evaluation algorithm, such as GTD2 or TD( λ ), is coupled to the network where the prediction vector w is callibrated proportional to −∇ w MSPBE θ ( w ) .
The full procedure is summarized in Algorithm 1, in
Appendix A.
Regarding the convergence of TTNs, a few remarks are in order:
The network needs to evolve sufﬁciently slowly relative to the linear prediction weights.
In our theoretical analysis, this is achieved by ensuring that the step sizes ξ t and α t of the network and the linear policy evaluation algorithm respectively
decay to zero at different rates
In particular , ξ t /α t → 0 as t → ∞ . With
this relative disparity in magnitudes, one can assume that the network is essentially quasi-static, while the faster
linear
component is equilibrated relative to the static features
The linear prediction algorithms need to converge for any set of features provided by the neural network, particularly linearly dependent features.
This induces a technical bottleneck since linear independence of the features are a necessary condition for the convergence of the prediction methods GTD and GTD2.
We overcome this by following a differential inclusion based analysis for GTD2.
Finally, we need to guarantee the stability of the iterates (both feature vector θ t and the prediction vector w t ) and this is ensured by projecting the iterates to respective compact, convex sets.
The analysis for the convergence of the neural network is general, enabling any network architectures that are twice continuously differentiable.
We prove that the TTNs converge asymptotically to the stable equilibria of a projected ODE which completely captures the mean dynamics of the algorithm.
We now state our main result (for notations and technical details, please refer Appendix B).
The results are provided for cases when TD( λ ) or GTD2 is used as the linear prediction method.
However, note that similar results can be obtained for other linear prediction methods.
Theorem 1.
Let ¯ θ =
( θ, ¯ w ) (cid:62) and Θ ⊂ R m + d be a compact, convex subset with smooth boundary.
Let the projection operator Γ Θ be Frechet differentiable and (cid:98) Γ Θ¯ θ ( − 12 ∇ L slow )(¯ θ ) be Lipschitz continuous.
Also, let Assumptions 1-3 hold.
Let K be the set of asymptotically stable equilibria of the following ODE contained inside Θ :
[Equation]
Then the stochastic sequence { ¯ θ t } t ∈ N generated by the TTN converges almost surely to K (sample path dependent).
Further,
TD( λ ) Convergence: Under the additional Assumption 4-TD( λ ), we obtain the following result:
For any λ ∈ [0 , 1] , the stochastic sequence { w t } t ∈ N generated by the TD( λ ) algorithm (Algorithm 2) within the TTN setting converges almost surely to the limit w ∗ , where w ∗ satisﬁes
[Equation]
with ¯ θ ∗ ∈ K (sample path dependent).
E XPERIMENTS
We investigate the performance of TTNs versus a variety of other nonlinear policy evaluation algorithms, as well as the impact of choices within TTNs.
We particularly aim to answer (a) is it beneﬁcial to optimize the MSPBE to obtain value estimates, rather than using value estimates from surrogate losses like the MSTDE; (b) do TTNs provide gains over other nonlinear policy evaluation algorithms; and (c) can TTNs beneﬁt from the variety of options in linear algorithms, including leastsquares approaches, eligibility traces and different policy evaluation algorithms.
More speculatively, we also investigate if TTNs can provide a competitive alternative to deep Q-learning in control.
Experiments were performed on-policy in ﬁve environments.
We use three classic continuous-state domains: Puddle World, a continuous-state grid world with high-magnitude negative rewards for walking through a puddle; Acrobot, where a robot has to swing itself up; and Cartpole, which involves balancing a pole.
We also use two game domains: Catcher, which involves catching falling apples; and Puck World, in which the agent has to chase a puck (Tasﬁ, 2016).
Catcher includes both a variant with 4-dimensional observations—position and velocity of the paddle, and (x,y) of the apple— and one with image-based observations—with two consecutive 64-by-64 grayscale images as input.
This domain enables us to analyze the beneﬁt of the algorithms, on the same domain, both with low-dimensional and high-dimensional observations.
We describe the policies evaluated for these domains in Appendix D.
We include a subset of results in the main body, with additional results in the appendix.
Results in Cartpole are similar to Acrobot; Cartpole results are only in the appendix.
The value estimates are evaluated using root-mean-squared value error (RMSVE), where value error is ( V π ( s ) − ˆ V ( s )) 2 .
The optimal values for a set of 500 states are obtained using extensive rollouts from each state and the RMSVE is computed across these 500 states.
For the algorithms, we use the following settings, unless speciﬁed otherwise.
For the slow part (features), we minimize the mean-squared TD error (MSTDE) using the AMSGrad optimizer (Reddi et al., 2018) with β 1 =and β 2 = 0 . 99 .
The network weights use Xavier initialization (Glorot and Bengio, 2010); the weights for the fast part were initialized to 0.
In Puddle World, the neural network consists of a single hidden layer of 128 units with ReLU activations.
In the other environments, we use 256 units instead.
To choose hyperparameters, we ﬁrst did a preliminary sweep on a broad range and then chose a smaller range where the algorithms usually made progress, summarized in Appendix D. Results are reported for hyperparameters in the reﬁned range, chosen based on RMSVE over the latter half of a run with shaded regions corresponding to one standard error.
[Figure]
TTN vs. competitors.
We compare to the following algorithms: nonlinear TD, nonlinear GTD (Maei et al., 2009), Adaptive Bases (ABBE and ABTD) (Di Castro and Mannor, 2010), nonlinear
TD + LSTD regularization (inspired by Levine et al.
We describe these algorithms in more depth in Appendix D. All of the algorithms involve more complex updates compared to TTNs, except for nonlinear TD, which corresponds to a semi-gradient TD update with nonlinear function approximation.
For TTNs, we use LSTD for the linear, fast part.
In Figure 2, TTN is able to perform as well or better than the competitor algorithms.
Especially in
Puddle World, its error is signiﬁcantly lower than the second best algorithm.
Interestingly, Nonlinear
GTD also performs well across domains, suggesting an advantage for theoretically-sound algorithms.
The utility of optimizing the MSPBE.
First, we show that the TTN beneﬁts from having a second head learning at a faster timescale.
To do so, we compare the prediction errors of using TTN, with the fast process optimizing the MSPBE (using LSTD) and the slow one optimizing the MSTDE, and one trained end-to-end using the MSTDE with AMSGrad.
As a baseline, we include TTN with a ﬁxed representation (a randomly initialized neural network) to highlight that the slow process is indeed improving the representation.
We also include results for optimizing the MSTDE with the ﬁxed representation.
[Figure]
In Figure 3, we see that optimizing the MSPBE indeed gives better results than optimizing the
MSTDE.
Additionally, we can conclude that using the MSTDE, despite being a poor objective to learn the value function, can still be effective for driving feature-learning since it outperforms the ﬁxed representation.
Linear algorithms and eligibility traces.
TTNs give us the ﬂexibility to choose any linear policy evaluation algorithm for the fast part.
We compare several choices: TD, least-squares
TD (LSTD) (Bradtke and Barto, 1996), forgetful
LSTD (FLSTD) (van Seijen and Sutton, 2015), emphatic TD (Sutton et al., 2016), gradient TD (the
TDC variant)
(Sutton et al., 2009) and their true-online versions (van Seijen and Sutton, 2014; van
Hasselt et al., 2014) to learn the value function.
GTD and ETD are newer temporal difference methods which have better convergence properties and can offer increased stability.
The true-online variants modify the update rules to improve the behavior of the algorithms when learning online and seem to outperform their counterparts empirically (van Seijen and Sutton, 2014).
Least-squares methods summarize past interaction, but are often avoided due to quadratic computation in the number of features.
For TTNs, however, there is no computational disadvantage to using LSTD methods, for two reasons.
It is common to choose deep but skinny architectures (Mnih et al., 2015; Hessel et al., 2017).
Furthermore, if the last layer is fully connected, then we already need to store O ( d 2 ) weights and use
O ( d 2 ) time to compute a forward pass—the same as LSTD.
We include FLSTD, which progressively forgets older interaction, as this could be advantageous when the feature representation changes over time.
For TTN, incremental versions of the least-squares algorithms are used to maintain estimates of the required quantities online (see appendix D).
All of these linear algorithms can use eligibility traces to increase their sample efﬁciency by propagating TD errors back in time.
The trace parameter λ can also provide a bias-variance tradeoff for the value estimates (Sutton, 1988; Dann et al., 2014).
For nonlinear function approximation, eligibility traces can no longer be derived for TD.
Though invalid, we can naively extend them to this case by keeping one trace per weight, giving us nonlinear TD( λ ).
[Figure]
The results overall indicate that TTNs can beneﬁt from the ability to use different linear policy evaluation algorithms and traces, in particular from the use of least-squares methods as shown in
Figure 4 for Puddle World and Catcher.
The dominance of LSTD versus the other linear algorithms is consistent, including in terms of parameter sensitivity, persists for the other three domains.
We additionally investigated sensitivity to λ , and found that most of the TTN variants beneﬁt from a nonzero λ value and, in many cases, the best setting is high, near 1.
One exception is the least-squares methods, where LSTD performs similarly for most values of λ .
Nonlinear TD( λ ), on the other hand, performs markedly worse as λ increases.
This is unsurprising considering the naive addition of eligibility traces is unsound.
We include these sensitivity plots in the appendix, in Figure ?? .
Surrogate loss functions.
For all the previous experiments, we optimized the MSTDE for the slow part of the network, but as discussed in Section 3, other objectives can be used.
We compare a variety of objectives, by choosing different Y t , including Y t =
R t +1 (Reward); Y t =
S t +1 (Next State); and
Y t =
R t +1 + ˆ Y
( S t +1 ) .
(Semi-gradient MSTDE).
In Puck World, in Figure 5 a), we can see that every auxiliary loss performed well.
This does not appear to be universally true, as in Acrobot we found that the MSTDE was a less effective surrogate loss, leading to slower learning (see Figure 5 b).
Alternate losses such as the semi-gradient MSTDE and next state predictions were more successful in that domain.
These results suggest that there is no universally superior surrogate loss and that choosing the appropriate one can yield beneﬁts in certain domains.
[Figure]
Control
Although the focus of this work is policy evaluation, we also provide some preliminary results for the control setting.
For control, we include some standard additions to competitor learning algorithms to enable learning with neural networks.
The DQN algorithm (Mnih et al., 2015) utilizes two main tricks to stabilize training: experience replay—storing past transitions and replaying them multiple times—and a target network—which keeps the value function in the Q-learning targets ﬁxed, updating the target network infrequently (e.g., every k = 10 , 000 steps).
We use an alternative strategy to target networks for TTN.
The use of a target network is motivated by ﬁtted Q-iteration (FQI) (Ernst et al., 2005), which updates towards ﬁxed Q-values with one sweep through a batch of data.
TTNs provide a straightforward mechanism to instead directly use FQI, where we can solve for the weights on the entire replay buffer, taking advantage of the closed form solution for linear regression towards the Q-values from the last update.
Batch FQI requires storing all data, whereas we instead have a sliding window of experience.
We therefore additionally incorporate a regularization term, which prevents the weights from changing too signiﬁcantly between updates, similarly to Levine et al.
Each FQI iteration requires solving a least squares problem on [Figure] the entire buffer, an operation costing O ( nd 2 ) computation where d is the number of features in the last layer of the network and n is the size of the buffer; we update the network every k steps, which reduces the per-step computation to O ( nd 2 /k ) .
The slow part drives feature-learning by minimizing the semi-gradient MSTDE for state-action values.
As another competitor, we include
LS-DQN (Levine et al., 2017), a DQN variant which also utilizes adjustments to the ﬁnal layer’s weights towards the FQI solution, similar to TTN-FQI.
The experimental details differ for control.
On nonimage Catcher, we do a sweep over α slow and λ reg , the regularization parameter, for TTN and sweep over the learning rate and the number of steps over which (cid:15) is annealed for DQN.
On image Catcher, runs require signiﬁcantly more computation so we only tune hyperparameters by hand.
The FQI update in TTNs was done every 1000 (10000) steps for non-image (image) Catcher.
We run each algorithm 10 times (5 times) for 200 thousand steps ( million steps) on the non-image (image) Catcher.
We see that TTN is able to perform well on both versions of Catcher in Figure 6, particularly learning more quickly than the DQN variants.
This difference is especially pronounced in the image version of catcher, where TTN is also able to achieve much higher average returns than DQN.
Both algorithms seem to suffer from catastrophic forgetting later during training as the performance dips down after an initial rise, although TTN still stabilizes on a better policy.
Overall, these results suggest that
TTNs are a promising direction for improving sample efﬁciency in control, whilst still maintaining stability when training neural networks.
D ISCUSSION AND C ONCLUSION
In this work, we proposed Two-timescale Networks as a new strategy for policy evaluation with nonlinear function approximation.
As opposed to many other algorithms derived for nonlinear value function approximation, TTNs are intentionally designed to be simple to promote ease-of-use.
The algorithm combines a slow learning process for adapting features and a fast process for learning a linear value function, both of which are straightforward to train.
By leveraging these two timescales, we are able to prove convergence guarantees for a broad class of choices for both the fast and slow learning components.
We highlighted several cases where the decoupled architecture in TTNs can improve learning, particularly enabling the use of linear methods—which facilitates use of least-squares methods and eligibility traces.
This work has only begun the investigation into which combinations for surrogate losses and linear value function approximation algorithms are most effective.
We provided some evidence that, when using stochastic approximation algorithms rather than least-squares algorithms, the addition of traces can have a signiﬁcant effect within TTNs.
This contrasts nonlinear TD, where traces were not effective.
The ability to use traces is potentially one of the most exciting outcomes for TTNs, since traces have been so effective for linear methods.
More generally, TTNs provide the opportunity to investigate the utility of the many linear value function algorithms, in more complex domains with learned representations.
For example, emphatic algorithms have improved asymptotic properties (Sutton et al., 2016), but to the best of our knowledge, have not been used with neural networks.
Another promising direction for TTNs is for off-policy learning, where many value functions are learned in parallel.
Off-policy learning can suffer from variance due to large magnitude corrections (importance sampling ratios).
With a large collection of value functions, it is more likely that some of them will cause large updates, potentially destabilizing learning in the network if trained in an end-to-end fashion.
TTNs would not suffer from this problem, because a different objective can be used to drive learning in the network.
We provide some preliminary experiments in the appendix supporting this hypothesis (Appendix C.7)
B C ONVERGENCE PROOF OF TWO - TIMESCALE NETWORKS B. D EFINITIONS & N OTATIONS
- Let R + denote the set of non-negative real numbers, N = { 0 , 1 , 2 , . . . } and (cid:107) · (cid:107) denote the
Euclidean norm or any equivalent norm.
- A map f : R d → R d is Lipschitz continuous if (cid:107) f
( x ) − f ( y ) (cid:107) ≤ L ( (cid:107) x − y (cid:107) ) , for some L ∈ (0 , ∞ ) , ∀ x , y ∈ R d . - A set-valued map h : R d → { subsets of R d } is called a Marchaud map, if it satisﬁes the following conditions:
For each x ∈ R d , h ( x ) is convex and compact.
For each x ∈ R d , ∃ K ∈ (0 , ∞ ) such that sup y ∈ h ( x ) (cid:107) y (cid:107) ≤ K (1 + (cid:107) x (cid:107) ) .
h is upper-semicontinuous, i.e. , if { x n } n ∈ N → x and { y n } n ∈ N → y , where x n ∈ R d , y n ∈ h ( x n ) , ∀ n ∈ N ,
then y ∈ h ( x ) .
For x 1 , x 2 ∈ R d and D ∈ R k × k a diagonal matrix, we deﬁne the inner-product < x 1 , x 2 > D (cid:44) x (cid:62) 1 D x 2 .
We also deﬁne the semi-norm (cid:107) x (cid:107) D (cid:44) <
x , x > 12 D .
If all the diagonal elements of D are strictly positive, then (cid:107) · (cid:107) D is a norm.
- For any set X , let ˚ X denote the interior of X and ∂X denote the boundary of X . -
For brevity, let ¯ θ =
( θ, ¯ w ) (cid:62) and Φ ¯ θ be the feature matrix corresponding to the feature parameter ¯ θ ,
i
e.
[Equation]
where x θ ( s ) (cid:62) is the row-vector corresponding to state s .
Further, deﬁne the |S| × |S| -matrix P π as follows:
[Equation]
- Also, recall that L slow ( θ
= MSTDE
( θ
[Equation]
- A function Γ : U ⊆
R d 1 → X ⊆ R
d 2 is Frechet differentiable at x ∈ U if there exists a bounded linear operator (cid:98) Γ x : R d 1 → R d 2 such that the limit
[Equation]
exists and is equal to (cid:98) Γ x ( y ) .
We say Γ is Frechet differentiable if Frechet derivative of Γ exists at every point in its domain.
B. A SSUMPTIONS
Assumption 1: The pre-determined, deterministic, step-size sequence { ξ t } t ∈ N satisﬁes ξ t > 0 , ∀ t ∈ N , (cid:88) t ∈ N ξ t = ∞ , (cid:88) t ∈ N ξ 2 t < ∞ .
Assumption 2: The Markov chain induced by the given policy π is ergodic, i.e. , aperiodic and irreducible.
Assumption 2 implies that the underlying Markov chain is asymptotically stationary and henceforth it guarantees the existence of a unique steady-state distribution d π over the state space S (Levin and
Peres, 2017), i.e. , lim t →∞ P ( S t = s )
=
d π ( s ) , ∀ s ∈ S .
Assumption 3:
Given a realization of the transition dynamics of the MDP in the form of a sample trajectory
O π = { S 0 , A 0 , R 1 , S 1 , A 1 , R 2 , S 2 , . . .
} , where the initial state S 0 ∈ S is chosen arbitrarily, while the action A (cid:51)
A t ∼ π ( S t , · )
, the transitioned state S (cid:51) S t +1 ∼ P ( S t , A t , · ) and the reward R (cid:51) R t +
= R ( S t , A t , S t +1 ) .
B. C ONVERGENCE A NALYSIS
[Figure]
To analyze the long-run behaviour of our algorithm, we employ the ODE based analysis (Borkar, 2008; Kushner and Yin, 2003; Ljung, 1977) of the stochastic recursive algorithms.
Here, we consider a deterministic ordinary differential equation (ODE) whose asymptotic ﬂow is equivalent to the long-run behaviour of the stochastic recursion.
Then we analyze the qualitative behaviour of the solutions of the ODE to determine the asymptotically stable sets.
The ODE-based analysis is elegant and conclusive and it further guarantees that the limit points of the stochastic recursion will almost surely belong to the compact connected internally chain transitive invariant set of the equivalent
ODE.
Since the algorithm follows a multi-timescale stochastic approximation framework, we will also resort to the more generalized multi-timescale differential inclusion based analysis proposed in (Borkar, 1997; Ramaswamy and Bhatnagar, 2016).
Note that there exists only a unilateral coupling between the neural network (where the feature vectors ¯ θ t are calibrated by following a stochastic gradient descent w.r.t.
L slow ) and the various policy evaluation algorithms (see Figure 7).
This literally implies that the policy evaluation algorithms depend on the feature vectors ¯ θ t but not vice-versa.
Therefore, one can independently analyze the asymptotic behaviour of the feature vectors { ¯ θ t } t ∈ N .
Also, as a technical requirement, note that since one cannot guarantee the stability (almost sure boundedness) of the iterates { ¯ θ t } t ∈ N (which is a necessary condition required for the ODE based analysis.
Please refer Chapter 2 of Borkar (2008)), we consider the following projected stochastic recursion:
[Equation]
where Γ Θ ( · ) is the projection onto a pre-determined compact and convex subset Θ ⊂ R m + d , i.e. , Γ Θ ( x ) = x , for x ∈ ˚Θ , while for x / ∈ ˚Θ , it is the nearest point in Θ w.r.t.
the Euclidean distance (or equivalent metric).
Deﬁne the ﬁltration {F t } t ∈ N , a family of increasing natural σ -ﬁelds, where F t (cid:44) σ (cid:0) { ¯ θ i , S i , R i ; 0 ≤ i ≤ t } (cid:1) .
The following lemma characterizes the limiting behaviour of the iterates { ¯ θ t } t ∈ N :
Lemma 1.
Let Assumptions 1-3 hold.
Let Θ ⊂ R m + d be a compact, convex subset with smooth boundary.
Let Γ Θ be Frechet differentiable.
Further, let (cid:98) Γ Θ¯ θ ( − 12 ∇ L slow )(¯ θ ) be Lipschitz continuous.
Let K be the set of asymptotically stable equilibria of the following ODE contained inside Θ :
[Equation]
Then the stochastic sequence { ¯ θ t } t ∈ N generated by the TTN converges almost surely to K .
Proof.
We employ here the ODE based analysis as proposed in (Borkar, 2008; Kushner and Clark, 2012).
Firstly, we recall here the stochastic recursion which updates ¯ θ t :
[Equation]
S [Equation] where Γ Θ is the projection onto a pre-determined compact and convex subset Θ ⊂ R m + d .
Here, δ t (cid:44) R t +1 + γ t
+1 ˆ Y ¯ θ t ( S t +1 )
− ˆ Y ¯ θ t ( S t ) is the temporal difference.
Also, ∇ ¯ θ t
ˆ Y ¯ θ ∈ R ( m + d ) ×|S| is the
Jacobian of ˆ Y ¯ θ at ¯ θ = ¯ θ t and ∇ ¯ θ t ˆ Y ¯ θ ( s ) is the column corresponding to state s .
Now the above equation can be rewritten as
[Equation]
where h 1 (¯ θ )
[Equation]
the noise term
[Equation]
and the bias (cid:96)
[Equation]
Further
[Equation]
where (cid:98) Γ Θ is the Frechet derivative (deﬁned in Eq.
Note that Γ Θ is single-valued since Θ is convex and also the above limit exists since the boundary ∂ Θ is assumed smooth.
Further, for x ∈ ˚ Θ , we have
[Equation]
sufﬁciently
[Equation]
i.e. , (cid:98) Γ Θ x ( · ) is an identity map for x ∈ ˚Θ .
A few observations are in order:
C1: (cid:98) Γ Θ¯ θ ( h 1 (¯ θ )) is a Lipschitz continuous function in ¯ θ
This follows from the hypothesis of the
Lemma.
C2: (cid:98) Γ Θ¯ θ t (cid:0) M 1 t +1 (cid:1) is a truncated martingale difference noise.
Indeed, it is easy to verify that the noise sequence { M 1 t +1 } t ∈ N is a martingale-difference noise sequence w
r
t to the ﬁltration {F t +1 } t ∈ N , i.e. , M 1 t +1 is F t +1 -measurable and integrable, ∀ t ∈ N and E (cid:2)
M 1 t +1 |F t (cid:3) =
a.s., ∀ t ∈ N .
Also, since Γ Θ ( · ) is a continuous linear operator, we have (cid:98) Γ Θ ( M 1 t +1 ) to be F t +1 -measurable and integrable, ∀ t ∈ N likewise.
C3:
[Equation]
(cid:1)
[Equation]
0 as t → ∞ a.s.
Indeed,
[Equation]
By taking t → ∞ , C3 follows directly from the ergodicity (Levin and Peres, 2017)
(Assumption 2) and ﬁniteness of the underlying Markov chain.
C4: o ( ξ t ) →
0 as t → ∞ (follows from Assumption 1).
C5: Iterates { ¯ θ t } t ∈ N are stable (forcefully), i.e. bounded almost surely, since θ t ∈ Θ , ∀ t ∈ N (ensured by the projection operator Γ Θ ) and Θ is compact ( i.e. , closed and bounded).
C6:
∃ K 0 ∈ (0 , ∞ ) ,
such that
[Equation]
This follows directly from the ﬁniteness of the Markov chain and from the assumption that the boundary ∂ Θ is smooth.
Now, by appealing to Theorem 2, Chapter 2 of (Borkar, 2008)), we conclude that the stochastic recursion (10) asymptotically tracks the following ODE
[Equation]
In other words, the stochastic recursion (10) converges to the asymptotically stable equilibria of the
ODE (15) contained inside Θ .
Remark 1.
It is indeed non-trivial to determine the constraint set Θ without prior adequate knowledge about the limit set of the ODE (15).
A pragmatic approach to overcome this concern is to initiate the stochastic recursion with an arbitrary convex, compact set Θ with a smooth boundary and gradually spread to the whole of R m + d
(Chen, 2006).
Remark 2.
It is also important to characterize the hypothesis of the above lemma ( i.e. , (cid:98) Γ Θ¯ θ ( − 1 2 ∇ L slow )(¯ θ ) is Lipschitz continuous) with respect to the features
ˆ Y ¯ θ .
To achieve that one has to consider the non-projected form of the ODE (15).
Apparently, when one considers the spreading approach proposed in the above remark, then it is essentially encouraged to consider the non-projected form since the limiting ﬂow of the ODE arising from the projected stochastic recursion is more likely to lie inside the compact, convex set as Θ becomes larger.
Thereupon, it is easy to observe that the condition
ˆ Y ¯ θ is twice continuously-differentiable is sufﬁcient to ensure the Lipschitz continuity of (cid:98) Γ Θ¯ θ ( − 12 ∇ L slow )(¯ θ ) .
Additionally, in that case K = { ¯ θ |∇ ¯ θ L slow (¯ θ ) = 0 } which is the set of local extrema of J .
B. TD( λ ) A LGORITHM
One can directly apply the TD( λ ) with linear function approximation algorithm to estimate the value function with respect to the features provided by the neural network.
The TD( λ ) algorithm is provided in Algorithm 2.
Here e t , w t ∈ R d .
Further, δ t (cid:44) R t +1 + γ t +1 w (cid:62) t
x θ t ( S t +1 ) − w (cid:62) t x θ t ( S t ) is the temporal difference.
Parameters: α t > 0 , λ ∈ [0 , 1] ;
Initialization: w 0 = 0 , e 0 = 0 ;
For each transition ( S t , R t +1 , S t +1 ) in O π , do:
[Equation]
[Equation] [Equation]
Assumption 4-TD( λ ): The pre-determined, deterministic, step-size sequence { α t } t ∈ N satisﬁes:
[Equation]
N
[Equation]
Note that the step-size schedules { α t } t ∈ N and { ξ t } t ∈ N satisfy ξ t α t → 0 , which implies that { ξ t } converges to 0 relatively faster than { α t } .
This disparity in terms of the learning rates induces an asynchronous convergence behaviour asymptotically (Borkar, 1997), with feature parameter sequence { ¯ θ t } converging slower relative to the TD( λ ) sequence { w t } .
The rationale being that the increment term of the underlying stochastic gradient descent of the neural network is smaller compared to that of the TD( λ ) recursion (17), since the neural network SGD is weighted by the step-size schedule { ξ t } t ∈ N which is smaller than { α t } t ∈ N for all but ﬁnitely many t .
This unique pseudo heterogeneity induces multiple perspectives, i.e. , when viewed from the faster timescale recursion (recursion controlled by α t ), the slower timescale recursion (recursion controlled by ξ t ) seems quasi-static (‘almost a constant’), while viewed from the slower timescale, the faster timescale recursion seems equilibrated.
Further, it is analytically admissible (Borkar, 1997) to consider the slow timescale stochastic recursion ( i.e. , the neural network SGD) to be quasi-stationary ( i.e. , ¯ θ t ≡ ¯ θ , ∀ t ∈ N ), while analyzing the asymptotic behaviour of the relatively faster timescale stochastic recursion (17).
Thereupon, we obtain the following directly from Theorem 1 of (Tsitsiklis and Van Roy, 1997).
Lemma 2.
Assume ¯ θ t ≡ ¯ θ , ∀ t ∈ N .
Let Assumptions 1-3 and 4-TD( λ ) hold.
Then for any λ ∈ [0 , 1] , the stochastic sequence { w t } t ∈ N generated by the TD( λ ) algorithm (Algorithm 2) within the TTN setting converges almost surely to the limit w ∗ , where w ∗ satisﬁes
[Equation]
(18) with T ( λ ) J ( s ) (cid:44) (1 − λ ) (cid:80) ∞ i =
λ i E (cid:104)(cid:80) ij =
γ [ j ] R j +1 + γ
[ i +1]
J ( S i +1 ) (cid:12)(cid:12) S
= s (cid:105) and γ [ j ]
=
Π ji =
γ
i (with γ 0 = 1 ).
Also, Π ¯ θ is deﬁned according to Eq.
(3) with F = { Φ ¯
θ
w
|
w ∈ R d } .
For other single-timescale prediction methods like ETD and LSPE, similar results follow.
Regarding the least squares method LSTD, which offers the signiﬁcant advantage of non-dependency on stepsizes (albeit computationally expensive) couples smoothly within the TTN setting without any additional consideration.
B. GTD2 A LGORITHM
However, one cannot directly apply the original GTD2 and TDC algorithms to the TTN setting, since a necessary condition required for the convergence of these algorithms is the non-singularity of the feature speciﬁc matrices E (cid:104)
x θ t ( S t ) x θ t ( S t ) (cid:62) (cid:105) and E (cid:2) ( x θ t ( S t ) − γ t +1 x θ t ( S t +1 ))
x θ t ( S t ) (cid:62) (cid:3) .
Please refer Theorem 1 and Theorem 2 of (Sutton et al., 2009).
Without the non-singularity assumption, it is indeed hard to guarantee the almost sure boundedness of the GTD2/TDC iterates.
In the TTN setting that we consider in this paper, one cannot explicitly assure this condition, since the features are apparently administered by a neural network and it is not directly intuitive on how to control the neural network to generate a collection of features with the desired non-singularity characteristic.
Henceforth, one has to consider the projected versions of these algorithms.
We consider here the projected GTD2 algorithm provided in Algorithm 3.
Parameters:
[Equation]
Initialization:
[Equation]
For each transition ( S t , R t +1 , S t +1 ) in O π do:
[Equation]
Here u t , w t ∈ R d .
Further, δ u t +1 (cid:44) R t +1 + γ t +1 u (cid:62) x θ t ( S t +1 ) − u (cid:62) x θ t ( S t ) is the temporal difference.
Here, Γ W ( · ) is the projection operator onto a pre-determined convex, compact subset W ⊂ R d with a smooth boundary ∂W .
Therefore, Γ W maps vectors in R d to the nearest vectors in W w.r.t.
the
Euclidean distance (or equivalent metric).
Convexity and compactness ensure that the projection is unique and belongs to W .
Similarly, U is a pre-determined convex, compact subset of R d with a smooth boundary ∂U .
Projection is required since the stability of the iterates { w t } t ∈ N and { u t } t ∈ N are hard to guarantee otherwise.
Assumption 4-GTD2: The pre-determined, deterministic, step-size sequences { α t } t ∈ N and { β t } t ∈ N satisfy
[Equation]
Deﬁne the ﬁltration {F t } t ∈ N , a family of increasing natural σ -ﬁelds, where F t (cid:44) σ (cid:0) { w i , u i , ¯ θ i , S i , R i ; 0 ≤ i ≤ t } (cid:1) .
Similar to the TD( λ ) case, here also we follow the quasi-stationary argument.
Henceforth, we analyze the asymptotic behaviour of GTD2 algorithm under the assumption that feature vector ¯ θ t is quasi-static, i.e. ¯ θ t ≡ ¯ θ =
( θ, ¯ w ) (cid:62) .
Lemma 3.
Assume ¯ θ t ≡ ¯ θ =
( θ, ¯ w ) (cid:62) , ∀ t ∈ N .
Let Assumptions 1-3 and 4-GTD2 hold.
Then
[Equation]
where A ∗ is the set of asymptotically stable equilibria of the following ODE:
[Equation]
and A u is the asymptotically stable equilibria of the following ODE:
[Equation]
Γ W w ( t ) (cid:0)(cid:0) Φ (cid:62) ¯ θ D d π δ u − Φ (cid:62) ¯ θ
D d π Φ ¯ θ (cid:1) w ( t ) (cid:1) , w (0) ∈ ˚ W and t ∈ R
+ , with δ u deﬁned in Eq.
Proof.
The two equations in the modiﬁed GTD2 algorithm constitute a multi-timescale stochastic approximation recursion, where there exists a bilateral coupling between the stochastic recursions (19) and (20).
Since the step-size sequences { α t } t ∈ N and { β t } t ∈ N satisfy β t α t → 0 , we have β t →faster than α t → 0 .
This disparity in terms of the learning rates induces a pseudo heterogeneous rate of convergence (or timescales) between the individual stochastic recursions which results in a pseudo asynchronous convergence behaviour when considered over a ﬁnite time window.
Also note that the coherent long-run behaviour of the multi-timescale stochastic recursion will asymptotically follow this short-term behaviour with the window size extending to inﬁnity(Borkar, 1997; 2008).
This pseudo behaviour induces multiple viewpoints, i.e. , when observed from the faster timescale recursion (recursion controlled by α t ), the slower timescale recursion (recursion controlled by β t ) appears quasi-static (‘almost a constant’), while observed from the slower timescale, the faster timescale recursion seems equilibrated.
Further, it is analytically admissible (Borkar, 1997) to consider the slow timescale stochastic recursion (20) to be quasi-stationary ( i.e. , u t ≡ u , ∀ t ∈ N ), while analyzing the limiting behaviour of the relatively faster timescale stochastic recursion 19.
Analysis of the faster time-scale recursion: The faster time-scale stochastic recursion of the GTD algorithm is the following:
[Equation]
Under the previously mentioned quasi-stationary premise that u t ≡ u and ¯ θ t ≡ ¯ θ = ( θ, ¯ w ) (cid:62) , ∀ t ∈ N , thereupon, we analyze the long-term behaviour of the following recursion:
[Equation]
(24) where x t = x θ ( S t ) and δ u t +1 (cid:44)
[Equation]
The above equation can be rearranged as the following:
[Equation]
where the noise
[Equation]
h
[Equation]
and the bias (cid:96)
[Equation]
Similar to Equation (12), we can rewrite the above recursion as follows:
[Equation] [Equation]
where (cid:98) Γ W w t ( · ) is the Frechet derivative (deﬁned in Equation (8)) of the projection operator Γ W .
A few observations are in order:
D
: The iterates { w t } t ∈ N are stable, i.e. , sup t ∈ N (cid:107) w t (cid:107) < ∞ a.s.
This immediately follows since W is bounded.
D2: { (cid:98) Γ W w t (cid:0) M 2 t +1 (cid:1) } t ∈ N is a martingale-difference noise sequence with respect to the ﬁltration {F t +1 } t ∈ N .
This follows directly since { M 2 t +1 } t ∈ N is a martingale-difference noise sequence with respect to the same ﬁltration.
D3: { (cid:98) Γ W w t (cid:0) M 2 t +1 (cid:1) } t ∈ N are square-integrable and ∃ K 2 ∈ (0 , ∞ )
such that
[Equation] [Equation]
This follows directly from the ﬁniteness of the underlying Markov chain and from the assumption that the boundary ∂W is smooth. D4: (cid:98) Γ W w ( h 2 ( w )) is Lipschitz continuous with respect to w .
Proof similar to C1.
D5: (cid:98) Γ W w t (cid:0) (cid:96) 2 t (cid:1) → 0 as t → ∞ a.s. Proof similar to C3.
Now by appealing to Theorem 2, Chapter 2 of (Borkar, 2008) along with the above observations, we conclude that the stochastic recursion 23 asymptotically tracks the following ODE almost surely:
[Equation]
Therefore, w t converges asymptotically to the stable equilibria of the above ODE contained inside
W almost surely.
Qualitative analysis of the solutions of ODE (27) :
A trivial qualitative analysis of the long-run behaviour of the ﬂow induced by the above ODE attests that the stable limit set is indeed the solutions of the following linear system inside W (This follows since (cid:98) Γ W w ( y )
=
y for w ∈ ˚ W and also because (cid:98) Γ W w ( · ) does not contribute any additional limit points on the boundary other than the roots of h 2 since ∂W is smooth).
[Equation]
Note that E (cid:2) x
t x (cid:62) t (cid:3) =
Φ (cid:62) ¯ θ
D d π Φ ¯ θ .
Claim 1: The above linear system of equations is consistent, i.e. , E (cid:2) δ u t +1 x t (cid:3) ∈ R (Φ (cid:62) ¯ θ D d π Φ ¯ θ ) , i.e. , the range-space of Φ (cid:62) ¯ θ
D d π Φ ¯ θ : To see that, note that the above system can indeed be viewed as the least squares solution to the Φ ¯ θ w = δ u with respect to the weighted-norm (cid:107) · (cid:107)
D d π , where
[Equation]
where ¯ R is the expected reward.
(Note that E (cid:2) δ u t +1 x t (cid:3) =
Φ (cid:62) ¯ θ D d π δ u ).
The least-squares solution w 0 ∈ R d (which certainly exists but may not be unique) satisﬁes
[Equation]
Now choose w = D − 1 d π Φ (cid:62) ¯ θ D d π ( δ u − Φ ¯ θ w 0 ) .
Then
[Equation]
[ End of proof of Claim 1 ]
Since Φ (cid:62) ¯ θ D d π Φ ¯ θ may be singular ( i.e. , Φ (cid:62) ¯ θ D d π Φ ¯ θ is not invertible), the above least squares solution may not be unique and hence the collection of asymptotically stable equilibria of the ﬂow induced by the ODE (27) may not be singleton for every u .
Let’s denote the asymptotically stable equilibria of the ﬂow induced by the said ODE to be the set A u , where ∅ (cid:54) =
A u ⊆ W .
Analysis of the slower time-scale recursion: The slower time-scale stochastic recursion of the
GTD2 algorithm is the following:
[Equation]
Note that since ξ t β t → 0 , the stochastic recursion (20) is managed on a faster timescale relative to the the neural network stochastic recursion (10) and henceforth, we continue to maintain here the quasi-stationary condition ¯ θ t ≡ ¯ θ =
( θ, ¯ w ) (cid:62) .
Now the above equation can be rearranged as follows:
[Equation]
(31) where ∆ w t t +1 (cid:44) ( x t − γ t +1 x t +1 ) (cid:0) w t (cid:62) x t (cid:1) =
(cid:0) ( x t − γ t
+1 x t +1 ) x (cid:62) t (cid:1) w t , the noise term M 3 t +1 (cid:44)
∆ w t t +1 − E (cid:2)
∆ w t t +1 |F t (cid:3) and the bias (cid:96) 3 t
[Equation]
F t
[Equation]
− E (cid:2) ∆ w t t +1 (cid:3) .
Similar to Equation (12), we can rewrite the above recursion as follows:
[Equation]
(32) where (cid:98) Γ U u t ( · ) is the Frechet derivative (deﬁned in Equation (8)) of the projection operator Γ U .
Now the above equation can be interpreted in terms of stochastic recursive inclusion as follows:
[Equation] [Equation]
with (cid:98) Γ U u t (cid:0) E (cid:2) ∆ w t t +1 (cid:3)(cid:1) ∈ h 3 ( u t ) , where the set-valued map h 3 : R d → { subsets of R d } is deﬁned as
[Equation]
, where w ∈ A u (cid:111) .
Indeed h 3 ( u ) = (cid:110)(cid:98)
Γ U u ( B w ) , where B = E (cid:2)(cid:0) ( x t − γ t
+1 x t +1 ) x (cid:62) t (cid:1)(cid:3) and w ∈
A u (cid:111) .
It is easy to verify that B =
Φ (cid:62) ¯ θ D d π ( I − γ t +1 P π )
Φ ¯ θ .
Here, one cannot directly apply the multi-timescale stochastic approximation results from (Borkar, 1997) since the said paper assumes that the limit point from the slower timescale recursion is unique (Please see Chapter 6 of (Borkar, 2008)).
But in our setting, the slower timescale recursion (23) has several limit points (note that the stable limit set A u is not singleton).
This is where our analysis differs from that of the seminal paper on GTD2 algorithm, where it is assumed that both the matrices E (cid:2) x t x (cid:62) t (cid:3) and E (cid:2) ( x t − γ t +1 x t +1 ) x (cid:62) t (cid:3) are certainly non-singular.
However, in our TTN setting, one cannot guarantee this condition, since the features are apparently provided by a neural network and it is hard to fabricate the neural network to generate a collection of features with the desired non-singularity properties.
In order to analyze the limiting behaviour of the GTD2 algorithm under the relaxed singularity setting, henceforth one has to view the stochastic recursion (30) as a stochastic recursion inclusion (Benaïm et al., 2005) and apply the recent results from (Ramaswamy and Bhatnagar, 2016) which analyzes the asymptotic behaviour of general multi-timescale stochastic recursive inclusions.
A few observations are in order:
E1:
For each u ∈ U , h 3 ( u ) is a singleton: This follows from the deﬁnition of h 3 and Claimabove, where we established that each w ∈
A u is the least squares solution to the linear system of equations Φ ¯ θ w = δ u .
Therefore, it further implies that that h 3 is a Marchaud map as well.
E2: sup t ∈ N ( (cid:107) w t (cid:107) + (cid:107) u t (cid:107) ) <
∞ a.s.
This follows since W and U are bounded sets.
E3: { (cid:98) Γ U u t (cid:0) M 3 t +1 (cid:1) } t ∈ N is a martingale-difference noise sequence with respect to the ﬁltration {F t +1 } t ∈ N .
This follows directly since { M 3 t +1 } t ∈ N is a martingale-difference noise sequence with respect to the same ﬁltration.
E4: { (cid:98) Γ U u t (cid:0) M 3 t +1 (cid:1) } t ∈ N are square-integrable and ∃ K 3 ∈
(0 , ∞ )
, such that
[Equation]
This follows directly from the ﬁniteness of the underlying Markov chain and from the assumption that the boundary ∂U is smooth.
E5: (cid:98) Γ U u t (cid:0) (cid:96) 3 t (cid:1) → 0 as t → ∞ a
s. Proof similar to C3.
This implies that the bias is asymptotically irrelevant.
E6:
For each u ∈ U , the set
A u is a globally attracting set of the ODE (27) and is also Lyapunov stable.
Further, there exists K 4 ∈ (0 , ∞ ) such that sup w ∈A u (cid:107) w (cid:107) ≤ K 4 (1 + (cid:107) u
(cid:107) ) .
This follows since A u ⊆ W and W is bounded.
E7:
The set-valued map q : U → { subsets of R d } given by q ( u ) =
A u is upper-semicontinuous: Consider the convergent sequences { u n } n ∈ N → u and { w n } n ∈ N → w with u n ∈ U and w n ∈ q ( u n ) =
A u .
Note that w ∈ W , u ∈ U since W
and U are compact.
Also
[Equation]
(from Claim 1).
Now taking limits on both sides we get
[Equation]
This implies that w ∈ A u = q ( u ) .
The claim thus follows.
Thus we have established all the necessary conditions demanded by Theorem 3.10 of (Ramaswamy and Bhatnagar, 2016) to characterize the limiting behaviour of the stochastic recursive inclusion (33).
Now by appealing to the said theorem, we obtain the following result on the asymptotic behaviour of the GTD2 algorithm:
[Equation]
where A ∗ is the set of asymptotically stable equilibria of the following ODE:
[Equation]
One can obtain similar results for projected TDC.
We now state our main result:
Theorem 2.
Let Θ ⊂ R m + d be a compact, convex subset with smooth boundary.
Let Γ Θ be Frechet differentiable.
Further, let (cid:98) Γ Θ¯ θ ( − 12 ∇ L slow )(¯ θ ) be Lipschitz continuous.
Also, let Assumptions 1hold.
Let K be the set of asymptotically stable equilibria of the following ODE contained inside Θ :
[Equation]
Then the stochastic sequence { ¯ θ t } t ∈ N generated by the TTN converges almost surely to K (sample path dependent).
Further,
TD( λ ) Convergence: Under the additional Assumption 4-TD( λ ), we obtain the following result:
For any λ ∈ [0 , 1] , the stochastic sequence { w t } t ∈ N generated by the TD( λ ) algorithm (Algorithm 2) within the TTN setting converges almost surely to the limit w ∗ , where w ∗ satisﬁes
[Equation]
38) with T ( λ ) deﬁned in Lemma 2 and ¯ θ ∗ ∈ K (sample path dependent).
GTD2 Convergence: Let W, U ⊂ R d be compact, convex subsets with smooth boundaries.
Let Assumption 4-GTD2 hold.
Let Γ W and Γ U be Frechet differentiable.
Then the stochastic sequences { w t } t ∈ N and { u t } t ∈ N generated by the GTD2 algorithm (Algorithm 3) within the TTN setting satisfy
[Equation]
where A ∗ is the set of asymptotically stable equilibria of the following ODE:
[Equation]
and A u is the asymptotically stable equilibria of the following ODE:
[Equation]
with ¯ θ ∗ ∈ K (sample path dependent) and δ u deﬁned in Eq.
C A DDITIONAL E XPERIMENTS
[Figure] [Figure]
Nonlinear TD
TOTD
TD ETD
TOETD
LSTD
FLSTD (a)
[Figure]
Error
Step size (10^x)
TD
ETD
TDC TOTD
TOETD
LSTD (b)
TD
TOETD
LSTD
TDC
[Figure]
(c)
[Figure]
(b)
[Figure] [Figure]
TDC
Nonlinear TD TOTD
TD
ETD
TOETD LSTD
FLSTD (a)
Puddleworld
[Figure]
Error
[Figure]
size (10^x)
TD ETD
TDC
TOTD TOETD
LSTD (
[Figure]
TD
TOETD LSTD
TDC
Puddleworld
[Figure] [Figure]
[Figure]
C. I MAGE C ATCHER
We also ran policy evaluation experiments on image-based catcher with 2 stacked 64x64 frames as input.
The policy evaluated was the same as was used in the non-image setting.
Similar to the non-imaged based catcher experiments, we have similar plots
[Figure]
[Figure] [Figure]
C. C ARTPOLE
In the classic Cartpole environment
[Figure]
the agent has to balance a pole on a cart.
The state is given by vector of 4 numbers (cart position, cart velocity, pole angle, pole velocity).
The two available actions are applying a force towards the left or the right.
Rewards are +1 at every timestep and an episode terminates once the pole dips below a certain angle or the cart moves too far from the center.
We use the OpenAI gym implementation (Brockman et al., 2016).
The policy to be evaluated consists of applying force in the direction the pole is moving with probability 0.9 (stabilizing the pole) or applying force in the direction of the cart’s velocity with probability 0.1.
We inject some stochasticity so that the resulting policy does not perform overly well, which would lead to an uninteresting value function
[Figure] [Figure]
(c)
[Figure]
C. A CROBOT
In the classic Acrobot domain, the agent consisting of two links has to swing up past a certain height.
The agent observes a 4-dimensional state consisting of the angles and the angular velocities of each link.
The avaiable actions are three possible levels of torque to be applied to the joint.
The evaluated policy is obtained by training an agent with true-online Sarsa on a tile coding representation and then ﬁxing its learned epsilon-greedy policy.
[Figure]
[Figure] [Figure]
C.
P UCK W ORLD
In Puck World (Tasﬁ, 2016), the agent has to move in a two-dimensional box towards a good puck while staying away from a bad puck.
The 8-dimensional state consists of (player x location, player y location, player x velocity, player y velocity, good puck x location, good puck y location, bad puck x location, bad puck y location).
Each action increases the agent’s velocity in one of the four cardinal directions apart from a “None” action which does nothing.
The reward is the negative distance to the good puck plus a penalty of − 10 + x if the agent is within a certain radius of the bad puck, where x ∈ [ − 2 , 0] depends on the distance to the bad puck (the reward is slightly modiﬁed from the original game to make the value function more interesting).
The policy moves the agent towards the good puck, while having a soft cap on the agent’s velocity.
In more detail, to choose one action, it is deﬁned by the following procedure: First, we choose some eligible actions.
The None action is always eligible.
The actions which move the agent towards the good puck are also eligible.
For example, if the good puck is Northeast of the agent, the North and
East actions are eligible.
If the agent’s velocity in a certain direction is above 30, then the action for that direction is no longer eligible.
Finally, the agent picks uniformly at random from all eligible actions.
[Figure] [Figure]
[Figure]
C. O FF - POLICY C ATCHER
We run a preliminary experiment to check if TTN can have an advantage in the off-policy setting.
The target policy is the same as the one used for other Catcher experiments (described in Appendix D).
The behaviour policy is slightly different.
If the apple is within 20 units (the target policy is 25 units), then the agent takes the action in the direction of the apple with probability 0.7 and one of the other two actions with probability 0.15 each.
If the apple is not within range, then the agent takes the
None action 10% of the time and one of the other two with equal probability.
This combination of behaviour and target policies results in importance sampling ratios in the range of 0 to 8.7, moderately large values.
We try TTN with three off-policy algorithms (TD, TDC and LSTD) and compare to off-policy
Nonlinear TD.
For TTN, the features are learnt optimizing the MSTDE on the behaviour policy while the values are learned off-policy.
The main difference between TTN and Nonlinear TD is the fact that
Nonlinear TD does off-policy updates to the entire network while TTN only changes the linear part with off-policy updates.
[Figure]
From ﬁgure C.7, we see that TTN can outperform nonlinear TD in terms of average error and also has reduced variance (smaller error bars).
This seems to suggest that the TTN architecture can grant additional stability in the off-policy setting.
D E XPERIMENTAL DETAILS
LSTD
This version of LSTD was used in the policy evaluation experiments.
Algorithm
Inputs: η inv , λ, w 1 (initial weight vector)
Initialization: A 1 = diag ( η inv ) ∈ R
d × d , b 1 = 0 ∈ R d , z 1 = 0 ∈ R d , d is the number of features
For each transition ( S t , R t +1 , S t +1 ) do: ∆ t = ( x ( S t
[Equation]
The policy evaluated in Puddle World randomly took the north and east actions with equal probability, with episodes initialized in the southwest corner.
The policy evaluated in Catcher increased the velocity in the direction of the apple if the agent was within 25 units or else chose the “None” action with a 20% chance of a random choice.
Competitor details Nonlinear TD uses the semi-gradient TD update with nonlinear function approximation.
This is known not to be theoretically-sound and there exist counterexamples where the weights and predictions diverge (Tsitsiklis and Van Roy, 1997).
Nevertheless, since this is the simplest algorithm, we use this as a baseline.
Nonlinear GTD, conversely, has proven convergence results as it is an extension of the gradient TD methods to nonlinear function approximation.
Castro et al.
proposed three variants of adaptive bases algorithms, each optimizing a different objective.
We include two of them: ABTD, which is based on the plain TD update and ABBE, which is derived from the MSTDE.
We omit ABPBE, which optimizes the MSPBE, since the algorithm is computationally inefﬁcient, requiring O ( d 2 m ) memory, where d is the number of features in the last layer and m is the number of parameters for the bases (ie.
weights in the neural network).
Also, the derivation is similar in spirit to that of nonlinear GTD, so we would expect both to perform similarly.
Levine et al.
’s algorithm was proposed for the control setting, combining DQN with periodic LSTD resolving for the last layer’s weights.
We adapt their idea for policy evaluation by adding regularization to the last layer’s weights to bias them towards the LSTD solution and using semi-gradient TD on this objective.
Then, we can then train in a fully online manner, which makes the algorithm comparable to the other competitors.
This algorithm is labeled as Nonlinear TD-Reg.
Hyperparameters Here we present the reﬁned hyperparameter ranges that were swept over for the experimental runs.
First, we outline the hyperparameters of all the algorithms and, afterwards, we give the values tested for each hyperparameter.
There is one range for Puddle World and another for Catcher (both versions) since different ranges worked well for each domain.
For the two-timescale networks, for each environment, the learning rate for the slow part was set to a ﬁxed value which was sufﬁciently low for the fast part to do well but was not tuned extensively.
This was done for all the experiments except for those on surrogate losses, where we swept over a range of values (since different losses could have different optimization properties).
Two-timescale Networks:
For all algorithms, the learning rate for the slow part α slow .
TD,
ETD
TOTD
TOETD learning rate α , trace parameter λ TDC : primary weights learning rate α , secondary weights learning rate β , trace parameter λ LSTD: initializer for the A − 1 matrix η inv , trace parameter λ FLSTD: initializer for the A matrix η , learning rate α , forgetting rate ψ , trace parameter λ Castro MSTDE, Castro TD: layer learning
rate
α , other layers learning rate β
Nonlinear
GTD
primary weights learning rate α , secondary weights learning rate
β
Nonlinear
TD
learning rate α , trace parameter λ
Nonlinear
TD - Reg : learning rate α , learning rate towards LSTD solution (regularization) β , initializer for the A matrix η
For all algorithms that used eligibility traces, the range swept over was λ ∈ { 0 , 0 . 3 , 0 . 5 , 0 .
The other hyperparameters are shown individually below.
Two-timescale Networks:
[Equation]
TD
[Equation]
ETD, TOTD, TOETD:
[Equation]
[Equation] [Equation]
[Equation] [Equation]
TDC: α ∈ 10 c , c ∈ {− 4 , − 2 .
75 , ..., −
β ∈ 10 c , c ∈ {− 3 , − 2 }
LSTD: η inv ∈ 10 c , c ∈ {− 2 , − 1 .
FLSTD:
η ∈ 10 c , c ∈ {− 3 , − 2 , − 1 , 0 } α ∈ 10 c , c ∈ {− 4 , − 3 } ψ ∈ 10 c , c ∈ {− 5 , − 4 , − 3 }
Castro MSTDE, Castro TD: α ∈ 10 c , c ∈ {− 6 , − 5 .
75 , ..., − 4 } β ∈ 10 c , c ∈ {− 6 , − 5 .
75 , ..., − 4 }
Nonlinear GTD: α ∈ 10 c , c ∈ {− 3 , − 2 .
75 , ..., − 0 .
25 } β ∈ 10 c , c ∈ {− 5 , − 4 .
5 , ..., − 0 .
Nonlinear TD: α ∈ 10 c , c ∈ {− 4 , − 3 .
75 , ..., − 1 . 25 }
Nonlinear TD - Reg: α ∈ 10 c , c ∈ {− 6 , − 5 .
5 , ..., − 3 } β ∈ 10 c , c ∈ {− 3 .
5 , − 3 , ..., − 0 .
5 } η ∈ 10 c , c ∈ {− 2 , − 1 , 0 , 1 }
Control experiments For both control experiments, we modiﬁed the Catcher environment slightly from its default settings.
The agent is given only 1 life and an episode terminates after it fails to catch a single apple.
The reward for catching an apple is +1 and is -1 at the end of an episode.
The discount factor is set to 0.99.
For image-based Catcher, we stack two consecutive frames which we treat as the state.
This is done so that the agent can perceive movements in the paddle, thus making the state Markov.
Both DQN and TTN use an (cid:15) -greedy policy.
For DQN, (cid:15) is annealed from 1.0 to 0.01 (0.1) for nonimage (image)
Catcher over a certain number of steps.
For TTN, (cid:15) is ﬁxed to a constant value, 0.01 (0.1) for nonimage (image) Catcher.
For both algorithms, we use a replay buffer of size 50000 (200000) for nonimage (image) Catcher.
The buffer is initialized with 5000 (50000) transitions from a random policy.
The minibatch size for
DQN and feature learning was 32.
TTN uses the AMSGrad optimizer with β 1 = 0 and β 2 = 0 .and
DQN uses the ADAM optimizer with default settings, β 1 = 0 .
9 , β 2 = 0 .
For image catcher, due to the long training times, the hyperparameters were manually tuned.
For nonimage catcher, we concentrated our hyperparameter tuning efforts on the most important ones and use the following strategy.
We ﬁrst used a preliminary search to ﬁnd a promising range of values, followed by a grid search.
For TTN with FQI, we focused tuning on the step sizes for the features and the regularization factor.
For DQN, we focused on adjusting the step size and the number of steps over which (cid:15) , the probability of picking a random action, was annealed.
The other hyperparameters were left to reasonable values.
The ﬁnal hyperparameters for nonimage catcher:
[Equation]
towards previous weights) DQN — α = 10 − 3 .
75 , decaying (cid:15) over 20 thousand steps, update the target network every 1000 steps.
For LS-DQN, do a FQI upate every 50,000 steps with a regularization weight of 1.
The ﬁnal hyperparameters for image catcher:
TTN — α slow = 10 − 5 , λ reg = 10 − 3 (regularization towards previous weights), solve for new weights using FQI every 10,000 steps.
DQN — α = 10 − 3 .
75 , decaying (cid:15) over 1 million steps, update the target network every 10,000 steps.
For LS-DQN, do a FQI update every 500,000 steps with a regularization weight of 1.
Network architectures
For TTN, this describes the neural network which learns the features.
The layer marked as “features" is used to predict state-action values for the fast, linear part.
Image catcher - TTN
To predict the next state, we add an action embedding as done in (Oh et al., 2015).
Input 2x64x64 grayscale image (stacked frames)
Conv 32 ﬁlters, 7x7 kernel, stride 4, ReLU
Conv 64 ﬁlters, 5x5 kernel, stride 2, ReLU
Conv 64 ﬁlters, 3x3 kernel, stride 1, ReLU
Dense1 (features) 256 units, ReLU
From Dense1, Dense
(reward prediction) 3 units (one per action), Linear
From Dense1, add action embedding 256 units, Linear
Reshape
Into 8x8 image
Deconv 16 ﬁlters, 5x5 kernel, stride 2, ReLU
Deconv 16 ﬁlters, 5x5 kernel, stride 2, ReLU Deconv 16 ﬁlters, 5x5 kernel, stride 2, ReLU
Conv4 (state prediction) 1 ﬁlter, 3x3 kernel, stride 1, Linear
Image catcher - DQN
Input 2x64x64 grayscale image (stacked frames)
Conv 32 ﬁlters, 7x7 kernel, stride 4, ReLU
Conv 64 ﬁlters, 5x5 kernel, stride 2, ReLU
Conv 64 ﬁlters, 3x3 kernel, stride 1, ReLU
Dense 256 units, ReLU
Dense2 (Q-values) 3 units, Linear
Nonimage catcher - TTN
Input 4-dimensional state
Dense 128 units, ReLU Dense 128 units, ReLU
Dense3 (features) 128 units, ReLU
Dense4 (state and reward prediction) 15 units, Linear
Nonimage catcher - DQN
Input 4-dimensional state
Dense 128 units, ReLU
Dense 128 units, ReLU Dense 128 units, ReLU
Dense
[Table]