O
FF - POLICY M ULTI - STEP Q- LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control.
Deep Q-learning, however, still suffers from poor data-efﬁciency which is limiting with regard to realworld applications.
We follow the idea of multi-step TD-learning to enhance data-efﬁciency while remaining off-policy by proposing two novel TemporalDifference formulations: (1) Truncated Q-functions which represent the return for the ﬁrst n steps of a target-policy rollout w.r.t.
the full action-value and (2) Shifted
Q-functions, acting as the farsighted return after this truncated rollout.
We prove that the combination of these short- and long-term predictions is a representation of the full return, leading to the Composite Q-learning algorithm.
We show the efﬁcacy of Composite Q-learning in the tabular case and compare our approach in the function-approximation setting with TD3, Model-based Value Expansion and TD3( ∆ ), which we introduce as an off-policy variant of TD( ∆ ).
We show on three simulated robot tasks that Composite TD3 outperforms TD3 as well as state-of-the-art off-policy multi-step approaches in terms of data-efﬁciency.
I NTRODUCTION
In recent years, Q-learning (Watkins and Dayan, 1992) has achieved major successes in a broad range of areas by employing deep neural networks (Mnih et al., 2015; Silver et al., 2018; Lillicrap et al., 2016), including environments of higher complexity (Riedmiller et al., 2018) and even in ﬁrst real world applications (Haarnoja et al., 2019).
Due to its off-policy update, Q-learning can leverage transitions collected by any policy which makes it more data-efﬁcient compared to onpolicy methods.
Deep Q-learning, however, still has a very high demand for data samples which is limiting with regard to robot applications.
One reason for the low data-efﬁciency is the long temporal horizon the reward signal has to propagate through.
Data-efﬁciency of on-policy Temporal-Difference methods can be enhanced by the use of n -step returns, where a Monte Carlo rollout of length n is combined with a bootstrap of the value function.
To employ n -step returns in an off-policy setting, subtrajectories of the exploratory policy have to be stored.
These stored multi-step returns, however, will differ from the true value of the target-policy.
In order to beneﬁt from n -step data, the replay buffer has to be restricted in size or n has to be set to a small value to keep the samples close to the target-policy (Barth-Maron et al., 2018;
Hessel et al., 2018).
To avoid these problems, a dynamics model can be used for imaginary rollouts, the so-called Model-based Value Expansion (MVE) (Feinberg et al., 2018).
Alternatively, the full return can be composed of value functions with increasing discount, an approach called TD( ∆ ) (Romoff et al., 2019).
In this work, we deﬁne a model-free Temporal-Difference formulation which follows the idea of multi-step learning while remaining off-policy.
Our contributions are threefold.
First, we introduce the Composite Q-learning algorithm.
For its formulation, we deﬁne Truncated Q-functions , representing the return for the ﬁrst n steps of a target-policy rollout w.r.t.
the full action-value.
In addition, we introduce Shifted Q-functions which represent the farsighted return after this truncated rollout.
Both are then combined in a mutual recursive deﬁnition of the Q-function for the ﬁnal algorithm.
Second, we evaluate MVE within TD3, leading to MVE-TD3 .
And third, we introduce TD3 (∆) , an extension of TD( ∆ ) to deep Q-learning.
We discuss related work in Section 2, describe the theoretical background in Section 3 and deﬁne
Composite Q-learning, MVE-TD3 and TD3( ∆ ) in Section 4.
By breaking down the long-term return into a composition of several short-term predictions, our method increases data-efﬁcency which we show in the tabular case and for three simulated robot tasks in Section 5.
We then conclude in
Section 6.
R ELATED W ORK
In order to correct for the deviation from the current target-policy, several methods suggest the use of Importance Sampling (Precup et al., 2000; 2001; Munos et al., 2016).
Approaches based on
Importance Sampling, however, can come with a vast increase in variance or can be of high cost and are not easily applicable to deterministic policies.
One way to remain off-policy in multi-step Q-learning is to get the Monte Carlo rollout on the basis of the current target-policy applied to a learned dynamics model (Feinberg et al., 2018;
Buckman et al., 2018).
Due to accumulating errors of single-step models, this can lead to severe stability issues in the Q-update, which is also the conclusion of Feinberg et al.
The authors suggest to average all intermediate i -step returns to smoothen out the model error, the so-called TD- k trick.
In contrast to our work, Feinberg et al.
assume to have access to the true reward function.
In order to balance the length of the model-assisted rollout, Buckman et al.
couple it with an uncertainty estimate from value function and model ensembles.
We alleviate the problem of accumulating error by estimating a multi-step dynamics model implicitly via consecutive bootstrapping.
Most related to our approach is TD( ∆ ) (Romoff et al., 2019).
Romoff et al.
formalize a Bellmanoperator over the differences between value functions of increasing discount values.
Their approach is on-policy and can therefore beneﬁt directly from n -step returns.
We extend TD( ∆ ) to the offpolicy case below.
Q-learning can also be extended to a SARSA-like tree-backup called Q ( σ ) (Asis et al., 2018; Hernandez-Garcia and Sutton, 2018).
However, it is an open question how to adjust this idea to continuous action-spaces and deterministic policies.
In the Hybrid Reward Architecture (HRA), van Seijen et al.
(2017) suggest a decomposition of the reward and the estimation of value functions for each part of this decomposition which are then combined as an approximation of the full return.
HRA addresses the problem of complex rewards and is thus complementary to our work focusing on long time scales.
Concurrently to our work, Asis et al.
(2019) introduced Fixed
Horizon TD-learning , formalizing action-value functions for different horizons in time.
In contrast to our work, however, Asis et al. deﬁne a consecutive bootstrapping formulation w.r.t.
the target policies of the different truncated horizons and not the full return.
A ﬁrst general description of using intermediate predictions in Temporal Differences to restrict predictions to a ﬁxed temporal horizon has been introduced in (Sutton, 1988).
B ACKGROUND
We consider tasks modelled as Markov decision processes (MDP), where an agent executes action a t ∈ A in some state s t ∈ S following its stochastic policy π .
According to the dynamics model
M of the environment, the agent transitions into some state s t +1 ∈ S and receives scalar reward r t .
The agent aims at maximizing the expected long-term return:
[Equation]
where T is the (possibly inﬁnite) temporal horizon of the MDP and γ ∈ [0 , 1] the discount factor.
It therefore tries to ﬁnd π ∗ , s.t.
R π ∗ ≥ R π for all π .
If the model of the environment is unknown, model-free methods based on the Bellman Optimality Equation over the so-called action-value can be used:
[Equation]
In the following, we abbreviate E a j>t ∼ π,s j>t ∼M [ ·| s t , a t ] by E t,π, M [ · ] .
One popular represen- tative of continuous model-free reinforcement learning is the Deep Deterministic Policy Gradient algorithm (DDPG) (Lillicrap et al., 2016).
In DDPG, actor µ is a deterministic mapping from states to actions, µ : S (cid:55)→ A , representing the actions that maximize the critic Q µ , i.e.
Q Tr 1 + Q Tr 2 ··· Q Tr n r t r t
Q Sh 1 Q
Sh 2 ··· Q Sh n + Q r t γ γ γ γ γ γ γ
Intermediate Predictions Composite Q (a) s t a t 2fullyconnectedlayers(500,leakyReLU) fullyconnected (500,leakyReLU)
Q Tr 1 Q Tr 2 ...Q Tr n fullyconnected (500,leakyReLU)
Q Sh 1 Q
Sh 2 ...Q Sh n
Q (b) µ ( s t ) =
arg max a Q µ ( s t , a ) .
Q and µ are estimated by function approximators Q ( · , ·| θ Q ) and µ ( ·| θ µ ) , parameterized by θ Q and θ µ .
The critic is optimized on the mean squared error between predictions Q ( s j , a j | θ Q ) and targets y j
[Equation] [Equation]
s
[Equation]
µ
[Equation]
) |
[Equation]
Q
[Equation]
) , where Q (cid:48) and µ (cid:48) are target networks, parameterized by θ Q (cid:48) and θ µ (cid:48) .
The parameters of µ are optimized following the deterministic policy gradient theorem (Silver et al., 2014):
[Equation]
and the parameters of the target networks are updated according to:
[Equation]
← (cid:91) (1 − τ ) θ Q (cid:48)
+ τθ Q and θ µ
[Table]
← (cid:91) (
[Table]
τ )
[Table]
µ (cid:48)
+ τθ µ , (4) with τ ∈ [0 , 1] .
TD3 (Fujimoto et al., 2018) adds three adjustments to vanilla DDPG.
First, the minimum prediction of two distinct critics is taken for target calculation to alleviate overestimation bias, an approach belonging to the family of Double Q-learning algorithms (van Hasselt et al., 2016).
Second, Gaussian smoothing is applied to the target-policy, addressing the variance in updates.
Third, actor and target networks are updated every d -th gradient step of the critic, to account for the problem of moving targets.
O FF - POLICY M ULTI - STEP Q- LEARNING
In this section, we introduce the Composite Q-learning algorithm, an off-policy multi-step reinforcement learning method to enhance data-efﬁciency, along with MVE-TD3 and TD3( ∆ ) as baselines.
4. C OMPOSITE Q- LEARNING
The main motivation behind this work is the assumption that learning values on short time scales can be achieved faster than for the full temporal horizon of a task, which can be prohibitively long.
As Murphy (2005) shows for the ﬁxed-batch ﬁtted Q-iteration case, the number of samples needed to achieve a certain generalization error is exponential in the horizon of the MDP.
As discussed in Jin et al.
, Q-learning with UCB-exploration has total regret polynomial in the horizon of the MDP and exponential with (cid:15) -greedy exploration (Kearns and Singh, 2002).
We thus argue that truncated horizons for a ﬁxed MDP translate to lower sample complexity of value-estimation.
Building upon this idea, we estimate the return of n -step rollouts of the target-policy via Truncated
Q-functions which we then combine to the full return with model-free Shifted Q-functions, an approach we call Composite Q-learning , while remaining purely off-policy .
Since these quantities cannot be estimated directly from single-step transitions, we introduce a consecutive bootstrapping scheme based on intermediate predictions.
For an overview, see Figure 1a.
The full algorithm is in the appendix.
Code based on the implementation of TD3 1 can be found in the supplementary.
4.1. T RUNCATED Q- FUNCTIONS
In order to formalize the off-policy estimation of n -step returns, assume that n (cid:28) ( T − 1) and that ( T − 1 − t ) mod n = 0 for task horizon T .
We make use of the following observation:
[Equation]
That is, we can deﬁne the action-value as the combination of partial sums of length n .
We can then deﬁne the Truncated Q-function as Q π n ( s t , a t ) =
E t,π, M
[ (cid:80) t + n − 1 j = t γ j − t r j ] , which we plug into
Equation (5):
[Equation] [Equation]
Theorem 1.
Let Q π 1 ( s t , a t ) =
r t be the one-step Truncated Q-function and Q πi> 1 ( s t , a t ) =
r t + γ E t,π, M [ Q πi − 1 ( s t +1 , a t +1 )] the i-step Truncated Q-function.
Then Q πi ( s t , a t ) represents the truncated return Q πi ( s t , a t )
=
E t,π, M
[ (cid:80) t
+ i − 1 j = t γ j − t r j ] .
Following Theorem 1 (the proof can be found in the supplementary), we approximate Q
πn ( s t , a t ) off-policy via consecutive bootstrapping.
Let Q Tr ( · , ·| θ Q Tr ) denote a function approximator with parameters θ Q Tr and n outputs, subsequently called heads , estimating Q πi .
Each output Q Tr i bootstraps from the prediction of the preceding head, with the ﬁrst approximating the immediate reward function.
The targets are therefore given by:
[Equation]
(7) where µ (cid:48) corresponds to the actor maximizing the full Q-value as deﬁned in Section 3.
That is,
Q Tr represents evaluations of µ at different stages of truncation and y Tr j,i<n serve as intermediate predictions to get y Tr j,n .
We then only use Q Tr n , which implements the full n -step return, as the ﬁrst part of the composition of the Q-target.
Please note that in order to estimate Equation (6), the dynamics model would be needed to get s t + c · n of a rollout starting in s t .
In the following, we describe an approach to achieve an estimation of Equation (6) model-free.
4.1. S HIFTED Q- FUNCTIONS
To get an estimation for the remainder of the rollout Q πn : ∞ =
E t,π, M [ γ n Q ( s t + n , a t + n )] after n steps, we use a consecutive bootstrapping formulation of the Q-prediction.
Theorem 2.
Let Q π 1: ∞ ( s t , a t ) =
E t,π, M [ γQ π ( s t +1 , a t +1 )] be the one-step Shifted Q-function and
Q πi> 1: ∞ ( s t , a t ) =
E t,π, M [ γQ πi − 1: ∞ ( s t +1 , a t +1 )] the i-step Shifted Q-function.
Then Q πi : ∞ ( s t , a t ) represents the shifted return Q πi : ∞ ( s t , a t ) =
E t,π, M [ γ i Q π ( s t
+ i , a t + i )] .
Again, the proof of Theorem 2 can be found in the supplementary.
Let Q Sh ( · , ·| θ Q Sh ) denote the function approximator estimating the Shifted Q-function Q πn : ∞ , parameterized by θ Q Sh .
We can shift the Q-prediction by bootstrapping without taking the immediate reward into account, so as to skip the ﬁrst n rewards of a target-policy rollout.
The Shifted Q-targets for heads Q
Sh i therefore become:
[Equation]
4.1. C OMPOSITION
Following the deﬁnitions of Truncated and Shifted Q-functions, we can compose the full return.
Theorem 3.
Let Q πn ( s t , a t ) =
E t,π, M
[ (cid:80) t + n − 1 j = t γ j − t r j ] be the truncated return and
Q πn : ∞ ( s t , a t ) =
E t,π, M [ γ n Q ( s t + n , a t + n )] the shifted return.
Then Q π ( s t , a t ) =
Q πn ( s t , a t )
+
Q πn : ∞ ( s t , a t ) represents the full return, i.e. Q π ( s t , a t ) =
E t,π, M
[ (cid:80) ∞
j = t γ j − t r j ] .
The incorporation of truncated returns breaks down the time scale of the long-term prediction by the Shifted Q-function.
For details, see the proof of Theorem 3 in the supplementary.
We can thus deﬁne the Composite Q-target as:
[Equation]
approximated by Q ( · , ·| θ Q ) with parameters θ Q .
Since we have true reward r j , we include it in the target.
Please note, that shifting the action-value in time imposes a bottleneck, since it relies on the estimation of the full action-value.
We therefore jointly estimate Q Tr , Q Sh and Q with function approximator Q C ( · , ·| θ Q C ) .
Let SE C denote the squared error between targets y C j and predictions
Q C ( s j , a j | θ Q C ) .
See Figure 1b for the detailed architecture of the Composite Q-network.
Each pair Q
Tr i + Q
Sh i | 1 ≤ i ≤ n is a complete approximation of the true Q-value.
The circular dependency can lead to stability issues, due to the ampliﬁcation of propagated errors.
We add a regularization term to the loss penalizing the deviation between the prediction of Q and the n different Q-pairs to keep estimates in a narrow range.
It is implemented as the weighted mean squared error, i.e. the loss function becomes:
[Equation]
for batch of size m and with β being the regularization weight.
Actor µ is then updated on Q .
4. TD3( ∆ ) AND MVE-TD
Most related to Composite Q-learning are TD( ∆ ) (Romoff et al., 2019) and MVE-DDPG (Feinberg et al., 2018).
In this section, we describe how to extend TD( ∆ ) to an off-policy setting and how to combine MVE and TD3.
TD3( ∆ )
Another way to divide the value function into multiple time scales is TD( ∆ ) (Romoff et al., 2019).
To this point, it has only been applied in an on-policy setting.
In favor of comparability, we extend TD( ∆ ) to Q-learning, yielding TD3( ∆ ).
The main idea of TD( ∆ ) is the combination of different value functions corresponding to increasing discount values.
Let γ ∆ denote a ﬁxed ordered sequence of increasing discount values, i.e. γ ∆ =
( γ 1 , γ 2 , . . . , γ k ) (cid:62) | γ i> 1 >γ i − 1 .
We can then deﬁne delta functions W i as:
[Equation]
Let Q ∆ ( · , ·| θ Q ∆ ) denote the function approximator estimating Q γ 1 ≤ i ≤ k .
Based on the derivations in (Romoff et al., 2019), the targets for Q-learning can be formalized as:
[Equation]
which can then be used in any Q-learning algorithm.
The authors suggest the use of n -step targets within TD( ∆ ) which is not easily applicable in an off-policy setting.
In our experiments, we therefore compare our approach to single-step TD3 (∆) .
The algorithm can be found in the appendix.
MVE-TD3 We also apply Model-based Value Expansion within TD3, subsequently called MVETD3.
We add Gaussian policy smoothing to the rollout of the model.
In contrast to Feinberg et al.
(2018), however, we do not assume to have knowledge about the reward function.
Our model therefore approximates both, dynamics and reward.
E XPERIMENTAL R ESULTS
We evaluate Composite Q-learning in both, the tabular setting and the actor-critic method TD3.
5. T ABULAR C OMPOSITE Q- LEARNING
To analyze the effect of incorporating short-term prediction Q Tr n in the Q-update
[Figure]
we apply Composite Q-learning in the tabular case to the MDP of horizon K given in Figure 2a.
We compare it to vanilla Q-learning, as well as multi-step Q-learning based on subtrajectories of the exploratory policy and imaginary rollouts of the target-policy with the true model of the MDP.
[Figure]
Results for K = 20 are depicted in Figure 2b.
All approaches update the Q-function with a learning rate of 10 − 3 on the same ﬁxed batch of 10 3 episodes with a percentage of 10% non-optimal transitions.
For the multi-step approaches, we set rollout length n = 4 .
Since there is no generalization among states in the tabular setting, we update the Shifted Q-function with a learning rate of 10 −and the Truncated Q-functions with a learning rate of 10 − 3 .
An evaluation of different learning rates for the Shifted Q-functions is depicted in Figure 3.
We compare Composite Q-learning and vanilla Q-learning to Shifted Q-learning, which corresponds to Q-learning with a one-step shifted target (without approximate n -step returns from a Truncated Q-function).
The results show that shifting the value in time alone is slowing down convergence.
Precisely, Shifted Q-learning with a learning rate of 1 . 0 is equivalent to vanilla Q-learning, the same holds for Composite Q-learning with a learning rate of 10 − 3 for the Shifted Q-function.
The counterpart with different learning rates for the Truncated Q-functions, while keeping the learning rates for the full Q-estimate and Shifted Q-functions ﬁxed, can be seen in Figure 4. While there is improvement in convergence using a larger learning rate for the Truncated Q-functions, the results show higher variance and less beneﬁt than the Shifted Q-functions in Figure 3.
The results suggest that the interplay between short- and long-term predictions yields the most beneﬁt if the learning rate for the Shifted Q-functions can be set to a higher value. We believe this to be due to two possible reasons.
The higher learning rates for the Truncated Q-functions might lead to overﬁtting to the shorter horizons (given that it represents an easier learning problem compared to the full return), whereas the Shifted Q-functions allow for
[Figure]
higher learning rates, since they only have to consider the variability of the distribution over next states leading to decreased variance in their targets.
In the experiments in Section 5.2, we show that this can also be achieved by generalization as in the given architecture in Figure 1b.
The erroneous updates of on-policy multi-step Q-learning lead to convergence to a wrong action-value which is underlining the importance of truly off-policy learning.
However, if the true model can be used for the rollout, this can be really effective.
The difference in convergence speed between Q-learning and Composite Q-learning is highly signiﬁcant ( p -value of 4 · 10 − 8 according to a t -test) and grows with increasing horizon, as shown in Table 1.
This is in line with the ﬁndings of Jin et al.
(2018) who establish a connection between the horizon of an MDP and the sample complexity of value estimation.
[Table]
5. C OMPOSITE Q- LEARNING WITH F UNCTION A PPROXIMATION
We evaluate TD3, Composite TD3, MVE-TD3 and TD3( ∆ ) on three robot simulation tasks of
OpenAI Gym (Brockman et al., 2016)
based on MuJoCo (Todorov et al., 2012): Walker2d-v2,
Ant-v2 and Hopper-v2.
A visualization of the environments is depicted in Figure 5.
Parameter Setting Our main focus is on the analysis of the structure of Q-functions and not to achieve maximum possible performance.
We therefore keep the main parameters the same across all approaches (the underlying algorithm was TD3 in all cases), since this would lead to another source of potential differences otherwise.
Learning rate ( 10 − 3 ), target update ( 5 · 10 − 3 ) and actor setting (two hidden layers with 400 and 300 neurons and ReLU activation) are the same as in the default setting of TD3.
We use Gaussian exploration noise with σ = 0 .
The critic in Composite
Q-learning consists of four layers with 500 neurons and leaky ReLU activation, the critic in all other 5 Updates × 10−−−− Q - v a l u e f o
r ( s 0 , a ) Composite Q-learning (10 − 3 ) Composite Q-learning (10 − 2 ) Composite Q-learning (10 − 1 ) Composite Q-learning (10 0 )
Q-learning     5 Updates × 10−−−−
Q - v a l u e f
o
r
( s 0 , a )     5 Updates × 10−−−−
Q - v a l u e f
o
r
( s 0 , a )     5 Updates × 10−−−−
Q - v a l u e f
o
r
( s 0 , a ) approaches and the model in MVE-TD3, with a learning rate of 10 − 3 , has two layers.
However, we also evaluate TD3 and TD3( ∆ ) with the same critic setting as Composite TD3, see Figure 8 in the supplementary.
This corresponds to the most stable setting we could ﬁnd for TD3 with the given subset of MuJoCo tasks.
For Composite TD3, we set n = 50 and β = 10 − 4 for Walker2d-v2 and
Hopper-v2 and β = 5 · 10 − 5 for Ant-v2.
For TD3( ∆ ), we use the γ -schedule as suggested by the authors, i.e. γ 1 = 0 and γ i> 1 = γ i − 1 + , with an upper limit of 0.99 (Romoff et al., 2019).
For
MVE-TD3, we use a rollout length of 3, as described in Feinberg et al.
Data Efﬁciency We choose the area under the learning curve as performance measure, which is a common way to evaluate data-efﬁciency and learning stability (see e.g. Hessel et al.
Since vanilla TD3 can have severe stability issues for a small number of runs, we compare median and interquartile ranges (IQR).
As listed in Table 2, Composite TD3 outperforms TD3, as well as state-of-the-art multi-step approaches, in terms of learning speed throughout training.
After the considered time frame of 4 · 10 5 transitions, Composite TD3 has a 19%, 11% and 19% larger area under the median learning curve, in comparison to vanilla TD3.
As depicted in Figure 6, MVE-TD is highly sensitive w.r.t.
accumulating errors in reward and state prediction, even when applying the TD- k trick.
However, if dynamics and reward are estimated with high accuracy, model-based rollouts can be very effective, as the results show for Ant-v2.
Based on our experiments, TD3( ∆ ) seems to be less sensitive than MVE-TD3, albeit also less efﬁcient than Composite TD3.
The maximum return found by all approaches within 4 · 10 5 transitions can be seen in Table 3.
[Figure] [Table]
TD-error Analysis To test our intuition stated in the motivation, we refer to the lower row of
Figure 6 which shows the TD-errors for the different stages of truncation [Table] as well as the TD-errors for the complete Q-estimate.
All Truncated Q-estimates have lower TD-error throughout learning and it can be seen that the TD-error is consistently higher for longer horizons.
This reafﬁrms our [Figure] [Table] hypothesis that learning is easier on shorter horizons.
The Composite Q-target therefore reﬂects the true action-value faster which is beneﬁcial for Q-learning [Figure] states.
Sensitivity to Hyperparameters Lastly, we evaluate the inﬂuence of n and β , exemplary for the
Walker2d-v2 environment.
The results can be seen in Figure 7.
Shorter truncation horizons can lead to faster convergence, but the variance increases.
If n is rather large, however, it can slow learning down.
The same holds if the regularization weight β is set to a high value.
On the other hand, regularization is needed to keep the predictions in a narrow range, since it can lead to stability issues, otherwise.
C ONCLUSION
We introduced Composite Q-learning, an off-policy learning method that divides the long-term value into smaller time scales.
It combines Truncated Q-functions acting on a short horizon with Shifted
Q-functions for the remainder of the rollout.
We analyzed the efﬁcacy of Composite Q-learning in the tabular case and showed that the beneﬁt of short-term predictions increases with growing task horizon.
We further evaluated MVE-TD3 and introduced TD3( ∆ ), an off-policy variant of TD( ∆ ).
We showed on three simulated robot tasks that Composite TD3 outperforms vanilla TD3 by 19%, 11% and 19% in terms of area under the median learning curve.
The given results provide evidence that Composite TD3 enhances data-efﬁciency compared to other approaches in off-policy multi-step learning.
Going forward, the uncertainty estimate based on the variance of the Composite Q-network could be of beneﬁt in both, update calculation and exploration.
We further leave the application of the truncated formulation of other quantities such as state change or auxiliary costs as future work.
A A LGORITHM FOR C OMPOSITE TD
A detailed description of Composite DDPG is given in Algorithm 1, where the adjustments of TD are omitted for simplicity.
In order to transform Algorithm 1 to its TD3-equivalent, Gaussian policy smoothing has to be added to all targets in Line 8, as well as taking the minimum prediction of two distinct critics for each target.
Furthermore, actor and target networks have to be updated with delay
[Equation]
1 initialize critic Q C , actor µ and targets Q C(cid:48) , µ (cid:48) 2 initialize replay buffer R 3 for episode = 1 ..
E do get initial state s for t =
T do apply action a t = µ ( s t | θ µ ) + ξ , where ξ ∼ N (0 , σ ) observe s t +1 and r t and save transition ( s t , a t , s t +1 , r t ) in R calculate targets: y Tr
[Equation] [Equation]
= r j
[Equation]
) | θ Q Sh (cid:48) n ))
y C j = (cid:104)
y Qj , y Tr j, 1 , y Tr j,i> 1 , y Sh j, 1 , y
Sh j,i> 1 (cid:105) update Q C m according
Equation (10)
B A LGORITHM FOR TD3( ∆ )
A detailed description of DDPG( ∆ ) is given in Algorithm 2.
The authors suggest the use of n -step samples which is not easily applicable in an off-policy setting.
In our experiments, we therefore compare our approach to single-step TD3 (∆) .
To transform DDPG( ∆ ) to TD3( ∆ ), the adjustments as described in Appendix A have to be applied analogously.
Algorithm 2:
) 1 initialize critic Q ∆ , actor µ and targets Q ∆ (cid:48) , µ
2 initialize replay buffer R 3 set discount values
∆
( γ γ γ
) (cid:124) targets: y
[Equation]
µ µ
C T RUNCATED Q- FUNCTIONS
Theorem 1.
Let Q π 1 ( s t , a t ) =
r t be the one-step Truncated Q-function and Q πi> 1 ( s t , a t ) =
r t + γ E t,π, M [ Q πi − 1 ( s t +1 , a t +1 )] the i-step Truncated Q-function.
Then Q πi ( s t , a t ) represents the truncated return Q πi ( s t , a t )
=
E t,π, M
[ (cid:80) t
+ i − 1 j = t γ j − t r j ] .
Proof.
Proof by induction.
Q π 1 ( s t , a t ) =
r t by deﬁnition.
The theorem follows from induction step:
[Equation]

[Equation]
(cid:88)
j =( t
[Equation]
1) γ j
[Equation]
( t
[Equation] [Equation]

 =
[Equation]
t
[Equation]
 
[Equation]
j
[Equation]
[Equation] [Equation]

 =
r
[Equation]
M  
[Equation]
(cid:88)
j =( t +1)
γ j − t r
[Equation]

 =
E t
[Equation]
π
[Equation]
M   t
[Equation]
(cid:88)
j = t γ j − t r j   .
D S HIFTED Q- FUNCTIONS
Theorem 2.
Let Q π 1: ∞ ( s t , a t ) =
E t,π, M [ γQ π ( s t +1 , a t +1 )] be the one-step Shifted Q-function and
Q πi> 1: ∞ ( s t , a t ) =
E t,π, M [ γQ πi − 1: ∞ ( s t +1 , a t +1 )] the i-step Shifted Q-function.
Then Q πi : ∞ ( s t , a t ) represents the shifted return Q πi : ∞ ( s t , a t ) =
E t,π, M [ γ i Q π ( s t
+ i , a t + i )] .
Proof.
Proof by induction.
Q π 1: ∞ ( s t , a t ) =
E t,π, M [ γQ π ( s t +1 , a t +1 )] by deﬁnition.
The theorem follows from induction step:
[Equation]
E R ESOLVING THE M UTUAL R ECURSION
Theorem 3.
Let Q πn ( s t , a t ) =
E t,π, M
[ (cid:80) t + n − 1 j = t γ j − t r j ] be the truncated return and
Q πn : ∞ ( s t , a t ) =
E t,π, M [ γ n Q ( s t + n , a t + n )] the shifted return.
Then Q π ( s t , a t ) =
Q πn ( s t , a t )
+
Q πn : ∞ ( s t , a t ) represents the full return, i.e. Q π ( s t , a t ) =
E t,π, M
[ (cid:80) ∞
j = t γ j − t r j ] .
Proof.
[Equation]
Repeating this process then gives Q π ( s t , a t ) =
E t,π, M
[ (cid:80) ∞
j = t γ j − t r j ] .
Due to the composite structure, the Shifted Q-function represents the long-term sum of partial returns provided by the Truncated Q-function, as opposed to single reward values.
F
[Equation]
) WITH L ARGER N ETWORKS
To further analyze the impact of the Composite Q-learning structure
[Figure]
we evaluate TD3 and TD3( ∆ ) with the same number of parameters for the critic, as depicted in Figure 8.
The approaches do not seem to make use of the additional capacity and reveal a worse performance in the given time frame.
[Figure] [Figure]
G S HALLOW A RCHITECTURE
In order to analyze importance of the multi-layered structure reﬂecting the temporal dependencies between short- and long-term predictions
[Figure]
we compare the architecture used in our experiments to a shallow architecture which includes all predictions as outputs of the last hidden layer for the
Walker2d-v2 environment in Figure 9.
Based on the results in Appendix F
[Figure]
we reduced the amount of hidden layers to two.
[Figure]
H I NDIVIDUAL C OMPARISONS
For clarity
[Figure]
we provide individual comparisons between Composite TD3 and all baselines.
[Figure]
[Figure] [Figure]