The Emphatic Approach to Average-Reward Policy Evaluation
Abstract
Off-policy policy evaluation has been a longstanding problem in reinforcement learning.
This paper looks at this problem under the average-reward formulation with function approximation.
Differential temporal-difference (TD) learning has been proposed recently and has shown great potential compared to previous averagereward learning algorithms.
In the tabular setting, off-policy differential TD is guaranteed to converge.
However, the convergence guarantee cannot be carried through the function approximation setting.
To address the instability of offpolicy differential TD, we investigate the emphatic approach proposed for the discounted formulation.
Specifically, we introduce average emphatic trace for average-reward off-policy learning.
We further show that without any variance reduction techniques, the new trace suffers from slow learning due to high variance of importance sampling ratios.
Finally, we show that differential emphatic TD( β ), extended from the discounted setting, can save us from the high variance while introducing bias.
Experimental results on a counterexample show that differential emphatic TD( β ) performs better than an existing competitive off-policy algorithm.
Introduction
Off-policy learning is an important topic in reinforcement learning.
Particularly, off-policy predictions are an essential component in model learning, options learning (Sutton, Precup, & Singh, 1999) and life-long learning (Sutton, Bowling, & Pilarski, 2022; White, Modayil, & Sutton, 2012).
It has been extensively investigated in the discounted setting.
Here, the goal is to estimate the value function of a target policy with off-policy data collected by a different behavior policy .
In the online setting, Temporal-Difference (TD) learning is the most celebrated approach.
It is known that off-policy TD( λ ), which corrects the probability of action selection with importance sampling ratios , has guaranteed convergence in the tabular setting.
However, moving on to the function approximation setting, TD( λ ) is proven to diverge in some cases, even with linear function approximation.
This well-known issue is called the deadly traid , which characterizes the instability of TD algorithms with function approximation and off-policy learning.
Existing efforts in addressing the deadly triad can be roughly divided into value-based and densityratio-based methods.
Unlike value-based methods which don’t maintain an estimation of the densityratio, most of density-ratio-based methods (Liu et al., 2018; R. Zhang et al., 2020; S. Zhang, Liu, & Whiteson, 2020; Mousavi et al., 2020) apply to both the discounted setting and the averagereward setting.
However, it is shown that a value-based method consistently performs better than a competitive density-ratio-based method in the average-reward setting (S. Zhang, Wan, et al., 2021).
In this paper, we will mainly focus on value-based methods.
Existing value-based methods for the discounted setting can be categorized into three classes.
Gradient TD (GTD) and emphatic TD (ETD) algorithms are two classes that are designed with a convergence guarantee.
Another group of algorithms aims for faster learning instead of guaranteed convergence.
Examples are Vtrace( λ ) (Espeholt et al., 2018) and ABTD( ζ ) (Mahmood, Yu, & Sutton, 2017).
Compared to the discounted setting, the average-reward setting is more challenging and has been less studied.
The goal of average-reward policy evaluation is to estimate the reward rate and the differential value function .
In this setting, there are two important variants of TD learning algorithms: average-cost TD (Tsitsiklis & Van Roy, 1999) and differential TD (Wan, Naik, & Sutton, 2021).
It is previously known that off-policy differential TD will converge in the tabular setting (Wan et al., 2021), but the convergence guarantee doesn’t carry over to the function approximation setting (S. Zhang,
Wan, et al., 2021).
For off-policy average-cost TD, it is not clear yet whether it can diverge or not in the tabular setting, but this paper will show its divergence in the function approximation setting.
In addressing the instability, to the best of our knowledge, gradient TD is the first and only method extended to the average-reward setting among the three classes of off-policy remedies mentioned above for the discounted setting.
In this paper, we will take the next step to investigate the emphatic approach in the average-reward setting.
Our contributions are as follows:
Firstly, we illustrate the advantage of differential TD algorithms over average-cost TD algorithms in the problem of average-reward off-policy policy evaluation with function approximation.
Secondly, we complete the spectrum of the emphatic algorithms for the continuing setting with the proposed average emphatic trace.
However, we show that this theoretically sound trace suffers from high variance.
Finally, to avoid the high variance of the new trace, we extend generalized ETD (ETD( β ); Hallak et al., 2016) to the average-reward setting.
We show that differential ETD( β ) achieves the best performance on a counterexample in terms of the asymptotic error and step-size sensitivity compared to existing value-based algorithms, including a competitive off-policy algorithm.
We also include some rudimentary results of the nonlinear variants of these algorithms on MuJoCo tasks.
Background Problem formulation
We consider an infinite horizon Markov Decision Process (MDP), which is defined as a tuple ⟨S , A , P, r ⟩ where S is the finite state space, A is the finite action space, P ( s [-
-]′ | s, a ) is the transition function, and r is the deterministic reward function.
The policy of the agent is defined as π ( a | s ) :
S × A → [0 , 1] .
We denote the size of the state space and the action space by S = |S| and A = |A| .
Assuming the target policy π induces a unichain, we consider the average-reward formulation where the agent’s performance is evaluated with the reward rate defined as
[Equation]
where S 0 can be sampled from an arbitrary state distribution under our assumption.
It is also useful to define a differential value function over states v π ( s ) :
S → R :
[Equation]
Particularly, we consider the setting of online off-policy policy evaluation with linear function approximation where the agent needs to estimate both the reward rate and the differential value function of a target policy π while interacting with the environment with a behavior policy µ .
We assume the MDP is parameterized by the feature function ϕ : S → R d or equivalently the feature matrix Φ ∈ R S × d where d is the dimension of the feature.
At each time step t , instead of observing
S t , the agent observes the feature of the state ϕ t .
=
ϕ ( S t ) .
Then the agent selects an action based on the behavior policy µ and observes the next state feature ϕ t +1 and reward R t +1 .
With linear function approximation, the agent approximates the reward rate with parameter ¯ R and the differential value function with ˆ v ( s ) =
ϕ ( s ) ⊤ θ or in matrix form, ˆ v = Φ θ , where θ ∈ R d is a parameter vector.
We assume the Markov chains induced by the behavior policy µ and the target policy π are irreducible.
This assumption ensures the unique existence of the stationary distribution of the behavior policy d µ and that of the target policy d π .
Moreover, we define D v .
=
diag ( v ) for some vector v .
Specifically, we use D π for D d π and D µ for D d µ .
We use ∥ · ∥ v to denote the vector norm induced by D v for some vector v , i.e., ∥ x ∥ v = (cid:112) x ⊤ D v x .
We also define e = [1 , 1 , · · · , 1] ⊤ ∈ R S .
Off-policy average-cost TD and off-policy differential TD
In this section, we present two instances from the two major temporal-difference (TD) learning methods for average-reward reinforcement learning, average-cost TD (Tsitsiklis & Van Roy, 1999) and differential TD (Wan et al., 2021).
For simplicity, we consider their off-policy one-step versions: off-policy average-cost TD(0) and off-policy differential TD(0).
Most of the arguments should be easily extendable to their multi-step versions.
For the rest of the paper, we will drop the multi-step notation, referring to them as off-policy average-cost TD and off-policy differential TD.
The same goes for other algorithms in the remaining of the paper.
The off-policy average-cost TD makes the following update:
[Equation]
where ρ t .
=
π ( A t | S t ) µ ( A t | S t ) is the importance sampling ratio, α > 0 is the primary step-size parameter, and η > 0 is a second step-size parameter to control how fast the reward rate is updated compared to the differential value function.
Let u t
=
[ ¯ R t , θ ⊤ t ] ⊤ , then we can rewrite Update (1) as follows:
[Equation]
Similar to previous work (Sutton, Mahmood, & White, 2016), we only analyze the stability of the update.
If the update is stable, with similar techniques from earlier work by Precup, Sutton, and
Dasgupta (2001), we can prove that the update converges with probability one and bounded error.
To analyze the stability of the stochastic Update (2), we will look at the corresponding expected update ¯ u t
{++
 =+} ¯ u t
+ α ( b ac − A ac ¯ u t ) where A ac is defined as follows (Tsitsiklis & Van Roy, 1999):
[Equation]
On the other hand, off-policy differential TD performs the below update:
[Equation]
Follow the same definitions and arguments for off-policy average-cost TD, we have the corresponding expected update ¯ u t +
=
¯ u t + α ( b diff −
A diff ¯ u t ) for off-policy differential TD where A diff can be expressed as follows (S. Zhang, Wan, et al., 2021):
[Equation]
Instability of average-reward TD algorithms
Following Sutton et al.
, we analyze the A matrix in the expected update ¯ u t +1 =
¯ u t + α ( b −
A ¯ u t ) of different algorithms.
We call the matrix D µ ( I − P π ) the big key matrix and the matrix Φ ⊤ D µ ( I − P π ) Φ the key matrix .
If the eigenvalues of an A matrix all have positive real parts, we say the A matrix is stable.
If the A matrix of the expected update of an algorithm is stable, we say the algorithm is stable.
The stability of an algorithm is the necessary condition for the convergence of the algorithm (Sutton et al., 2016).
In average-reward on-policy setting, the big key matrix is positive semi-definite.
To ensure the stability of the A matrix, it is usually assumed that e is not in the column space of Φ (Tsitsiklis & Van Roy, 1999; S. Zhang, Wan, et al., 2021), which guarantees the positive definiteness of the key matrix.
Further, since µ = π and d ⊤ µ ( I − P π ) Φ = d ⊤ π ( I − P π ) Φ = 0 , the
A matrices for on-policy average-cost TD and on-policy differential TD are identical and have the same eigenvalues all of whose real parts are positive under such an assumption.
Thus, the on-policy stability for both algorithms is ensured.
In the off-policy case, the big key matrix may not be positive semi-definite.
Thus, the key matrix may not be positive definite.
In fact, we lose the stability guarantee of both algorithms in the off-policy setting.
Next, we will present two corresponding counterexamples of off-policy average-cost TD and off-policy differential TD.
Instability of off-policy average-cost TD
We first look at off-policy average-cost TD.
The two-state MDP with c = 1 in Figure 1 is a counterexample in which off-policy average-cost TD is unstable.
In this example, the stationary distributions for both target policy and behavior policy are very straightforward: d π = [0 . 4 , 0 . ⊤ and d µ = [0 . 6 , 0 . 4] ⊤ .
The transition probability matrix induced by the target policy is P π = (cid:20) 0 .
4 0 .(cid:21) .
The big key matrix is
[Equation]
[Equation]
− 0 .
[Equation]
[Equation]
(cid:21) =
(cid:20) 0 .− 0 .− 0 .0 .(cid:21) .
Further, the key matrix of this counterexample is
[Equation]
Now, the eigenvalues of A ac consist of η and the eigenvalues of the key matrix, in this case, − 0 . 04 .
Then, by definition, off-policy average-cost TD is unstable.
However, off-policy differential TD is stable in this counterexample with a wide range of η .
For example, when η = 1 , we have Φ ⊤ d µ = 1 . 4 , d ⊤ µ ( I − P π ) Φ = − 0 .
2 , and A diff =
(cid:20)− 0 .
2 1 .− 0 .(cid:21) .
The eigenvalues of A diff are 1225 + i √ 625 and 1225 − i √ 625 which have positive real parts.
Thus, off-policy differential TD is stable in this counterexample.
Instability of off-policy differential TD
To settle the instability of off-policy differential TD, we need to modify the counterexample for off-policy average-cost TD.
Notice that for off-policy average-cost TD, changing the value of c or η would only change the magnitude but not the sign of the eigenvalues of the A ac matrix.
Thus, average-cost TD will always be unstable regardless of any values of c and η .
However, this is not the case for off-policy differential TD.
When we change the value of c or η , the value and the sign of the eigenvalues of the A diff matrix will change.
Setting out to choose a c such that differential TD is unstable regardless of the value of η , we found that our purpose is fulfilled by choosing c = √ 175 .
In this case, the A diff matrix is
[Equation]
whose trace is η − 7 and whose determinant is 7(7 − η ) .
Thus, the eigenvalues of the A diff matrix, λand λ 2 , satisfy λ 1 + λ 2 = η − 7 and λ 1 λ 2 =
− η ) .
When η ̸ = 7 , at least one of the eigenvalues would be negative.
When η = 7 , both eigenvalues are zero.
Thus, off-policy differential TD is always unstable in this counterexample.
Comparing average-cost TD and differential TD
It is known that off-policy differential TD is guaranteed to converge while off-policy average-cost
TD has no guarantee at present in the tabular setting (Wan et al., 2021).
Here in the function approximation setting, we have shown that both off-policy average-cost TD and off-policy differential c 2 c a 1 r =ar =a 2 , r =a
[Figure] r = [Figure] TD can be unstable.
Also, the degrees of their instability differ: For the MDP in Figure 1, off-policy average-cost TD is always unstable regardless of the choices of c and η , while off-policy differential
TD can be stable for a large set of c and η .
This demonstrates a clear advantage of off-policy differential TD over off-policy average-cost TD when the key matrix has at least one eigenvalue whose real part is negative.
For the case where the key matrix has a full set of eigenvalues whose real parts are positive, off-policy average-cost TD is guaranteed to be stable.
On the other hand, we don’t have a general result for off-policy differential TD yet.
We conjecture that off-policy differential TD will also be stable.
To shed some light on this question, we prove that when the key matrix is positive definite, off-policy differential TD is stable.
The result is presented with the following proposition, whose proof can be found in the Appendix:
Proposition 1.
If ∆ .
= min
∥ θ ∥ 2 =1 θ ⊤ Φ ⊤ D µ ( I − P π ) Φ θ > 0 , then with sufficiently large η , matrix
A diff is positive definite. Having settled the potential benefit of differential TD in the off-policy setting, we will use it as the base average-reward algorithm in the remaining of the paper. Next, we will discuss how we can alleviate the instability issue by reweighting updates.  Average-reward emphatic TD learning
Addressing the instability of TD by reweighting updates
In the discounted reinforcement learning literature, there are two distinct approaches that try to address the instability by reweighting the updates using importance sampling ratios.
The first approach (Precup et al., 2001) uses a full importance sampling ratios product to completely correct the state distribution to the on-policy distribution, which can be directly adopted to the average-reward setting.
Adopting it to off-policy differential TD, the update with full importance-sampling (IS) correction is as follows:
[Equation]
We refer to temporal difference algorithms that use H t for reweighting as full IS TD and the above algorithm instance as differential full IS TD.
By fully correcting the state distribution, the expected update returns to the on-policy expected update where both average-cost TD and differential TD converge.
Reweighting the updates with the contemplation of how likely it will arrive at the current state started from the very first time step by following the target policy instead of the behavior policy, full IS TD is often thought of as the alternative life view.
However, this approach only works in theory but never becomes a practical algorithm due to the high-variance nature of the importance sampling ratios product.
Another relatively more practical approach is Emphatic TD (ETD; Sutton et al., 2016), which adopts the excursion view instead of the alternative life view.
In the excursion view, the updates are reweighted towards a state distribution between the stationary distribution of the behavior policy d µ and that of the target policy d π .
At every time step, a new excursion is started from the current state.
The excursion will last many steps until termination and be reweighted by importance sampling ratios.
With a fixed discount factor as soft termination, we can think that the excursions would never terminate, but old excursions would be decayed every time step.
Correcting the state distribution of updates closer to d π , the emphatic approach has proven to be effective in several environments with linear function approximation (Ghiassian & Sutton, 2021a, 2021b) and been successfully extended to deep reinforcement learning (Jiang et al., 2021, 2022).
However, moving on the discussion to the average-reward setting, ETD from the excursion view is undefined since there is no discounting or termination here, and the expectation of the original followon trace without decaying would go to infinity.
In the following section, we will introduce how to define the corresponding emphatic trace for the average-reward setting from the excursion view.
Average emphatic TD
This section introduces a novel way of correcting the state distribution.
Instead of performing an unnormalized weighted average over different excursions started from previous states like ETD, we use a normalized uniform average over excursions:
[Equation]
We refer to temporal difference algorithms that use this F t for reweighting as average emphatic TD (AETD) and the above algorithm instance as differential AETD.
Next, we will analyze the property of the expected update of differential AETD.
Following the derivation of ETD by Sutton et al.
(2016) and the above procedure for off-policy averagecost TD, the A ae matrix of expected update of differential AETD, ¯ u t
+
=
¯ u t + α ( b ae − A
ae ¯ u t ) , is
[Equation]
where f ( s ) =
d µ ( s ) lim t →∞ E µ [ F t | S t = s ] .
Plug in F t defined in Update (5) [Equation] we have
[Equation]
In vector form, we have f ⊤ = f ⊤ P π .
Since the expectations of importance sampling ratios are one and F 0 = 1 , the expectation of F t will remain one.
This implies that (cid:80) s f ( s ) = 1 .
Together with f ⊤ = f ⊤ P π , we can infer that f = d π .
Thus, AETD will correct the state distribution back to the on-policy distribution, which is the same as full IS TD.
However, compared with full IS TD, the variance of the trace is greatly tamed down by the averaging.
Note that the above analysis of f applies to all AETD algorithms.
Here, specifically, differential AETD has the same stability as on-policy differential TD, which is the best we can hope for theoretically.
We present the stability of differential AETD with the following proposition, which is proven above:
Proposition 2.
The A ae matrix is the same as on-policy differential TD:
[Equation]
Thus, differential AETD and its expected update are stable.
For the convergence analysis, we leave it for future work.
AETD as an extension of generalized ETD
In this section, we adopt generalized ETD (ETD( β ); Hallak et al., 2016) to the average-reward setting and describe an alternative way to look at AETD.
ETD( β ) is the generalization of ETD with the decaying factor of the trace, β , as a free parameter.
Taking values from (0 , 1) , β controls a bias-variance trade-off.
It is reported that ETD( β ) with an intermediate value of β achieves better performance than ETD (Ghiassian & Sutton, 2021a, 2021b).
Adopting it to the average-reward setting, differential ETD( β ) performs the following update:
[Equation]
Similar to the discounted episodic setting (Hallak et al., 2016), when β = 0 , differential ETD( β ) will degenerate to off-policy differential TD.
However, when β = 1 , the original ETD( β ) is problematic because the expectation of F t would go to infinity in the continuing case.
From this point of view,
AETD could be viewed as the proper extension of ETD( β ) when β = 1 since it treats every time step equally and maintains a bounded expectation.
Then, conversely, as a method in the spectrum of
ETD( β ) algorithms, AETD can also be used in the discounted setting.
It is reasonable to conjecture that AETD’s advantage of having less bias will come into play when the MDP has terminations, in which the variance issue is more amicable.
Coming back to the average-reward setting, for 0 < β < 1 , the stability of differential ETD( β ) is not clear yet.
We conjecture that for every β , there exists an average-reward MDP, in which differential
ETD( β ) is unstable.
However, we will show that differential ETD( β ) does help in alleviating the instability of off-policy differential TD in Section 5.
From a practical perspective, it would be interesting to provide the conditions under which differential ETD( β ) converges.
We leave this for future work.
Experiments
This section presents the experimental results of the emphatic algorithms, differential AETD (DiffAETD) and differential ETD( β ) (Diff-ETD( β )).
We compare them with the following baselines: offpolicy differential TD (Diff-TD; Wan et al., 2021) and differential GTD 1 (Diff-GTD 1 ; S. Zhang, Wan, et al., 2021).
For differential ETD( β ), the decaying parameter β is chosen from { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .
Note that when β = 0 , differential ETD( β ) is equivalent to differential TD; when β = 1 , the sensible version of emphatic algorithms is differential AETD.
For the evaluation metric, we use the absolute reward rate error (ARRE) | r ( π ) − ¯ R t | to evaluate the accuracy of the reward rate estimation.
When possible, we use Tsisiklis and Van Roy’s variant (Tsitsiklis & Van Roy, 1999) of root-mean-squared value error (RMSVE) inf c ∈ R ∥ ˆ v t − ( v π + c e ) ∥
d µ for differential value function estimation following Wan et al.
Results on the counterexample
We first use the two-state MDP in Figure 1 to demonstrate the effectiveness of the emphatic algorithms.
We use constant step sizes α = 2 x for all algorithms where x ∈ {− 18 , − 17 , · · · , − 1 , 0 } . We use a
[Figure] fixed constant second step size η = 1 for all compared algorithms.
For differential GTD 1 , we do not use ridge regularization on θ since our examples have a unique solution.
We run each algorithm for 20 , 000 steps.
The results are averaged over 30 independent runs and reported with the best performing hyperparameters [Figure] with which the area under the learning curve of ARRE is the smallest.
Here, we report the results on the counterexample with c = √ 175 in Figure 2.
Additional results with other value of c can be found in the Appendix.
The first two columns plot ARRE and RMSVE, respectively.
The third column plot the step-size sensitivity (recall that α = 2 x ).
From the top row of the figure, we can see that differential ETD( β ) is the best-performing algorithm compared to baselines in this environment.
It achieves both the lowest ARRE and RMSVE.
Also, it has a stable convergence and a nice U-shaped sensitivity curve.
For both off-policy differential TD and differential GTD 1 , the bounded ARREs with small step sizes in Figure 2(c) may have hidden the divergence of the algorithms because of insufficient training steps.
In Section 3, we have proven that off-policy differential TD is unstable in this counterexample, which means its expected update diverges.
Nevertheless, off-policy differential TD itself converges but with higher errors and within a very narrow range of step sizes.
Interestingly, divergence occurs with step sizes smaller than the converging step sizes.
Finally, differential GTD 1 doesn’t converge with the chosen step sizes, which contradict its convergence guarantee.
We suspect the reason is that differential GTD 1 is highly sensitive to the magnitude of the features (see the convergence of differential GTD 1 in Figure B.1(c) in the Appendix).
The converging step sizes may be out of the scope we are searching for.
The bottom row of Figure 2 shows how different emphatic algorithms perform.
We can easily notice that differential AETD can barely learn in this environment.
It suffers not only from the high variance of importance sampling ratios but also from the large features in this counterexample.
On the other hand, differential ETD( β ) is almost immune to the large features (see also Figure B.1(d) in the Appendix).
It achieves the lowest ARRE with β = 0 .
2 within the training steps.
However, from Figure 2(d), it is reasonable to conjecture that a high value of β would achieve the lowest ARRE if more training steps are given.
This can also be supported by Figure 2(e), which shows that higher values of β have slightly lower RMSVEs.
From Figure 2(f), we can see that as β becomes larger, the width of the U-shaped curve becomes smaller in accordance with the observations from (Ghiassian & Sutton, 2021a, 2021b), ETD( β ) suffers from high variance more with higher value of β .
In this section, we present some rudimentary experiment results on four MuJoCo tasks. Following S. Zhang, Wan, et al. (2021), we first train a deterministic policy π 0 with TD3 (Fujimoto, Hoof, &
10 3 Steps 7 .7 .7 .7 .8 .|
¯ˆ r − ¯ r π |
HalfCheetah ( σ = 0 . 9 )
Diﬀ-TD Diﬀ-GTD
Diﬀ-AETD Diﬀ-ETD( β ) 10 3 Steps 4 .4 .4 .4 .4 .
Walker2d ( σ = 0 . 9 ) 10 3 Steps 2 .3 .3 .4 .
Hopper ( σ = 0 . 9 ) 10 3 Steps 0 .0 .0 .0 .0 .
Swimmer ( σ = 0 . 9 )
Meger, 2018).
To avoid ill-posed importance sampling ratios, we use a stochastic target policy π by introducing Gaussian noise with zero mean and variance 0 .
1 to π 0 .
The behavior policy is composed similarly with Gaussian noise with zero mean and variance σ which takes value from { 0 . 5 , 0 . 7 , 0 . 9 } .
For all the algorithms, the learning rate is tuned from { 1 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 } .
To stabilize training, we clip all the importance sampling ratios to a maximum value of 1 .
The results are averaged over 30 independent runs and reported with the best performing hyperparameters, with which the ARRE is the smallest at the end of the training following S. Zhang, Wan, et al.
For other implementation details, please refer to the Appendix.
Here, we report the results with σ = 0 .
9 in Figure 3.
The results with other values of σ are deferred to the Appendix.
From the figure, we can see that differential ETD( β ) performs similarly to off-policy differential TD with slightly lower errors.
On the other hand, differential AETD is very unstable and performs inconsistently across different tasks compared to other algorithms.
Specifically, it performs the best in Swimmer, but its learning plateaus very quickly in the other three tasks.
Finally, differential GTD 1 exhibits learning difficulty in all the tasks but Swimmer, in which it performs slightly better than differential ETD( β ) and off-policy differential TD.
Conclusions and future work
In this paper, we have discussed some of our observations on the problem of average-reward off-policy policy evaluation with linear function approximation.
Specifically, we find that differential TD is relatively more robust than average-cost TD in the off-policy setting.
However, off-policy differential
TD without any treatments still suffers from instability.
We illustrate the instability of off-policy average-cost TD and off-policy differential TD with corresponding counterexamples.
To address the instability, we investigate the emphatic approach for off-policy learning, which is well understood in the discounted setting.
Our contributions to this subject are twofold.
Firstly, we complete the spectrum of the emphatic algorithms for the continuing setting with the average emphatic trace.
This completion is of two senses.
In a way, average emphatic TD can be considered the extension of ETD to the average-reward setting.
Alternatively, it can also be regarded as the proper form of ETD( β ) for β = 1 and applicable to the discounted setting.
Secondly, we validate the effectiveness of the emphatic algorithms in the average-reward scenario through an empirical study on a two-state MDP family, including the counterexamples.
We demonstrate the bias-variance trade-off controlled by the decaying parameter β , mirroring the same observation in the discounted setting.
More importantly, we show that differential ETD( β ) is the best performing algorithm in the two-state MDPs we tested in terms of the asymptotic error and step-size sensitivity.
Additionally, we extend the differential emphatic algorithms to nonlinear function approximation.
Rudimentary experiment results show batch differential TD is not sensitive to reweighting updates by expected emphasis on MuJoCo robotic simulation tasks.
Further investigation is needed for the nonlinear function approximation setting.
Despite the potential of the emphatic algorithms for average-reward reinforcement learning, it is not clear yet under what conditions, with what values of β , differential ETD( β ) is guaranteed to converge.
Further, if it converges, we would also like to know the bias of the point of convergence.
We hope to answer these critical theoretical questions in the future.
A Proof of Proposition
The proof is inspired by the proof of Lemma 2 in the work of S. Zhang, Zhang, and Maguluri (2021) which provides finite sample analysis for on-policy average-cost TD.
Recall that u
= [ ¯ R, θ ⊤ ] ⊤ and
[Equation]
we can rewrite the minimization problem min ∥ u
[Equation]
Now for any ¯ R ∈ R and θ ∈ R d
[Equation]
we have
[Equation]
The last step is due to we assume a finite state space which implies there exist B ≥ 0 such that max s ∈S
∥ ϕ ( s ) ∥ 2 ≤ B .
Next, by the definition of ∆ , for any θ ∈ R d , we have
[Equation]
Then we have min
[Equation]
x (1 − x ) .
When η ≥ ∆ + 3 B (cid:113)(cid:0) 3 B ∆ (cid:1) 2 − 1 , we have
[Equation]
Consequently, min ∥ u ∥ 2 =1 u ⊤ Au ≥ ∆2 > 0 .
B Additional experiment results and details Results on the two-state MDP with small features
To draw a complete picture of how differential emphatic algorithms perform, we also report results on the two-state MDP with c = 1 .
To make the task more challenging, we increase the gap
[Figure]
between the behavior and target policies.
In the experiment, we set π ( a 1 | c ) =
π ( a 2 | 2 c ) = 0 .
2 and µ ( a 1 | c ) = µ ( a 2 | 2 c ) = 0 . 8 .
The results are shown in Figure B.1.
From the top row of the figure, we can see that similar performance is achieved by the two baseline methods and differential ETD( β ), among which differential
GTD 1 has a slightly higher ARRE.
Suffering from high variance, differential AETD learns much more slowly than these methods.
Nevertheless, its performance in this small-feature setting is much better than in the previous large-feature environment.
In fact, it is much better than differential full
IS TD, which is not shown in the results but barely learns.
As for step-size sensitivity, from Figure
B.1(c), we can see that off-policy differential TD is the least sensitive and differential GTD 1 is the most sensitive.
Although differential ETD( β ) with the selected value of β is more sensitive to step sizes compared to off-policy differential TD, we can see that from Figure B.1(f), with β = 0 .
2 , differential ETD( β ) actually achieves almost the same performance as off-policy differential TD.
Implementation details for experiments on MuJoCo tasks
There are two challenges when applying differential AETD and differential ETD( β ) in complex tasks like MuJoCo tasks that require nonlinear function approximation.
Firstly, although having a lower variance than full IS TD, AETD and ETD( β ) still suffer from the high variance issue.
This is exacerbated when neural networks are applied.
Secondly, calculating the emphatic traces requires trajectory-based data instead of transition-based data, which is more challenging to maintain when a data buffer is needed for training neural networks.
To address these two issues, we adopt the strategy to learn an expected emphasis (S. Zhang, Liu, Yao, & Whiteson, 2020; Jiang et al., 2022) in the nonlinear setting.
Specifically, we estimate the following expected emphasis:
[Equation]
where F t is the emphasis defined in update (5) or update (6).
We use single-step bootstrap target to learn the expected emphasis.
Specifically, for every transition pair ( S t , A t , R t +1 , S t +1 ) , ˜ f
( S t +1 ) is updated towards ρ t ˜ f
( S t ) for AETD and βρ t ˜ f
( S t )
+ 1 for ETD( β ).
Following S. Zhang, Wan, et al.
(2021), we use batch updates to stabilize the training of neural networks.
To this end, we collect data with the behavior policy for 10 6 steps.
The implementations of off-policy differential TD and differential GTD 1 are extended from S. Zhang, Wan, et al..
For differential AETD and differential ETD( β ), we use neural networks to parameterize v and ˜ f .
Similar to off-policy differential TD, we use a target network for ˜ f
which is updated every 100 steps.
106 .7 .7 .8 .|
¯ˆ r − ¯ r π |
HalfCheetah ( σ = 0 . 5 )
Diﬀ-TD Diﬀ-GTD
{+Diﬀ-AETD Diﬀ-ETD( β ) 10 3 2 .2 .3 .3 .
+}
Walker2d ( σ = 0 .
Hopper ( σ = 0 . 5 ) 10 3 0 .0 .0 .0 .0 .
Swimmer ( σ = 0 . 5 ) 10 3 Steps 7 .7 .7 .8 .|
¯ˆ r − ¯ r π |
HalfCheetah ( σ = 0 .
7 ) 10 3 Steps 3 .4 .4 .4 .
Walker2d ( σ = 0 .
{+7 ) 10 3 Steps 2 .2 .3 .
+}
Hopper ( σ = 0 . 7 ) 10 3 Steps 0 .0 .0 .0 .0 .
Swimmer ( σ = 0 . 7 )
More results on MuJoCo tasks
The performance of different algorithms in MuJoCo tasks with σ = 0 .
5 and σ = 0 .
7 is shown in
Figure B.2.
The conclusion of these results are similar to that of Figure