Continual Learning In Environments With Polynomial Mixing Times
Abstract
The mixing time of the Markov chain induced by a policy limits performance inreal-world continual learning scenarios.
Yet, the effect of mixing times on learningin continual reinforcement learning (RL) remains underexplored.
In this paper,we characterize problems that are of long-term interest to the development ofcontinual RL, which we call scalable MDPs, through the lens of mixing times.
Inparticular, we theoretically establish that scalable MDPs have mixing times thatscale polynomially with the size of the problem.
We go on to demonstrate thatpolynomial mixing times present signiﬁcant difﬁculties for existing approaches,which suffer from myopic bias and stale bootstrapped estimates.
To validate ourtheory, we study the empirical scaling behavior of mixing times with respect to thenumber of tasks and task duration for high performing policies deployed acrossmultiple Atari games.
Our analysis demonstrates both that polynomial mixingtimes do emerge in practice and how their existence may lead to unstable learningbehavior like catastrophic forgetting in continual learning settings.
Introduction
Continual reinforcement learning (RL) [1] is an aspirational ﬁeld of research confronting the difﬁculties of long-term, real-world applications by studying problems of increasing scale, diversity,and non-stationarity.
The practical requirement for researchers to work on problems of reasonablecomplexity in the short-term presents a meta-challenge: choosing the right small-scale problemsso that the approaches we develop scale up to the use cases of the future.
Here, we address thismeta-challenge by formalizing RL problems that vary in size and by analyzing the scaling behaviorof popular RL algorithms.
In particular, we analyze on the often-ignored mixing time that expressesthe amount of time until the agent-environment dynamics converge to some stationary behaviour.
Towards this end, we speciﬁcally make the following contributions in this work:
Scalable MDPs: We propose the formalism of scalable MDPs in Deﬁnition 1 to characterizean abstract class of MDPs where the MDPs within the class are differentiated based on a
changing scale parameter.
Understanding the inﬂuence of this scale parameter on learning
can promote better understanding of the meta-challenge of extrapolating our results on small
scale problems to large scale problems of the same class.
Polynomial Mixing Times: Theorem 1 establishes the key result of this paper that as any
scalable MDP is scaled, its mixing time must grow polynomially as a function of the growing
state space.
This has major implications for regret analysis in these MDPs.
Myopic Bias
During Scaling: We demonstrate in Corollaries 2, 3, and 4 that Theorem
1 implies traditional approaches to RL cannot efﬁciently scale to problems of large size
without experiencing myopic bias in optimization that slows down learning.
Empirical Analysis of Mixing for Continual RL: We back up our theory with empiricalanalysis of mixing time scaling in Continual RL settings based on the Atari benchmark.
Ouranalysis provides insight into why agents experience signiﬁcant catastrophic forgetting orother optimization instability when learning in these domains.
We hope that our analysis intothe fundamental driver of these issues can open the door for more successful and principledapproaches to Continual RL domains like these moving forward.
We refer readers to Appendix D for detailed proofs of all propositions and theorems in our paper.
Formalizing Our Continual RL Setting
Aiming to characterize a broad range of settings, we focus on the formulation of RL in continuingenvironments.
As explained in Khetarpal et al.
[1], not only are supervised learning and episodic RLspecial cases of RL in continuing environments, but so are their non-stationary variants (continualsupervised learning and continual episodic RL), despite violating the stationarity assumptions oftheir root settings.
Indeed, approaches that do not acknowledge the more general RL in continuingenvironments setting exhibit myopic bias in their optimization when faced with non-stationarity [1].
Unfortunately, the popular discounted reward setting inserts the very same kind of myopic biasin optimization that we would like to avoid [2].
As typically implemented, discounting does notcorrespond to the maximization of any objective function over a set of policies [3] and the policygradient is not the gradient of any function [4].
These fundamental issues do not resolve as the discountfactor approaches 1 [3] and discounting does not inﬂuence the ordering of policies, suggesting it likelyhas no role to play in the deﬁnition of the control problem [5].
In contrast, the average reward perstep objective, which explicitly includes an average over the stationary distribution, avoids inducingmyopic biases and hence is well-suited for continual RL problems [5].
2.1 Average Reward RL in Continuing Environments
RL in continuing environments is typically formulated using a ﬁnite, discrete-time, inﬁnite horizon
Markov Decision Process (MDP) [6, 7], which is a tuple M = (cid:104)S , A , T, R (cid:105) , where S is the set of 60 states,
A is the set of actions, R : S×A → [0 , R max ] is the reward function, and T : S×S×A → [0 , 1] 61 is the environment transition probability function.
At each time step, the learning agent perceivesa state s ∈ S and takes an action a ∈ A drawn from a policy π : S ×
A → [0 , 1] with internal 63 parameters θ ∈ Θ .
The agent then receives a reward R ( s, a ) and with probability T ( s (cid:48) | s, a ) entersnext state s (cid:48) .
Markov chains may be periodic and have multiple recurrent classes, but optimality isdifﬁcult to deﬁne in such cases [8], making the following assumption necessary for analysis:
Assumption 1 All stationary policies are aperiodic and unichain, meaning they give rise to a Markovchain with a single recurrent class that is recurrent in the Markov chain of every policy.
Any RL problem may be modiﬁed such that Assumption 1 holds by adding an arbitrarily smallpositive constant (cid:15) to all transition probabilities in T ( s (cid:48) | s, a ) and renormalizing in which case theeffect on the objective of each stationary policy is O ( (cid:15) )
An important corollary to Assumption 1is that the steady-state distribution µ π induced by the policy π is independent of the initial state:
Corollary 1 All stationary policies π induce a unique steady-state distributionµ
[Equation]
) that is independent of the initial state such that
[Equation]
Corollary 1 implies that the long-term rewards of any π will be independent of the current state.
Assuch, the average reward per step objective ρ ( π ) can be deﬁned independently of its starting state [5]:
[Equation]
Computing the average reward with the last expression is limited by the amount of time the Markovchain induced by the policy T π ( s (cid:48) | s ) = [Equation] ∈A π [Equation]
a | s ) T ( s
[Equation]
| s, a ) needs to be run for before reaching zz
[Figure]
z
[Figure]
Universal Rule [Figure] [Figure] the steady-state distribution µ π ( s ) .
This amount of time is referred to in the literature as the mixingtime of the induced Markov chain.
We denote t π mix ( (cid:15) ) as the (cid:15) -mixing time of the chain induced by π :
[Equation]
where d TV is the total variation distance.
The so-called conventional mixing time is deﬁned ast π mix ≡ t π mix (1 / 4) .
The conventional mixing time only gives insight about distributional mismatchwith respect to the steady-state distribution, which led [10] to introduce the notion of a mismatch withrespect to the reward rate.
The (cid:15) -return mixing time is a measure of the time it takes to formulate anaccurate estimate of the true reward rate.
More formally, if we denote the h -step average undiscountedreturn starting from state s as ρ ( π, s, h ) , then we deﬁne the (cid:15) -return mixing time as:
[Equation]
We will come back to this deﬁnition when we present our experiments.
As emphasized in Levin and
Peres [11], however, one should not get bogged down in the use-case speciﬁc deﬁnitions of mixingtime (we prove their equivalent scaling behaviour in fact, c.f. Proposition 1 ).
Rather, it is their sharedproperty of being determined by both a policy π and environment M , and not simply a property ofthe environment itself, that is important.
The environment’s contribution can be assessed throughmixing times obtained from extreme policies such as the optimal policy, t π ∗ mix .
An alternative approach for quantifying mixing is by considering the structure of the transition matrix
T π ( s (cid:48) | s ) induced by π .
It is well known that the mixing properties of a Markov chain are governed bythe spectral gap derived from the eigenvalues of the matrix T π ( s (cid:48) | s ) .
Unfortunately, it is difﬁcult toreason about the spectral gap of a class of MDPs directly.
Towards this end, one useful interpretationof a Markov chain is as a random walk over a graph G ( π, T ) with vertex set S and edge set { ( s, s (cid:48) ) } ,for all s and s (cid:48) satisfying T π ( s (cid:48) | s )
+ T π ( s | s (cid:48) ) > 0 .
The diameter D π of the Markov chain inducedby π is thus the diameter or maximal graph distance of G ( π, T ) .
It is deﬁned using the hitting timet π hit ( s 1 | s 0 ) , the ﬁrst time step in which s 1 is reached following the Markov chain T π ( s (cid:48) | s ) from s 0 ,
[Equation]
D ∗ then denotes the minimum diameter achieved by any policy.
All MDPs that follow Assumption1 have ﬁnite diameter D π for all policies [12].
Levin and Peres [11] (eq. 7.4) established therelationship, 2 t π mix ≥ D π , so that the diameter provides a lower bound for the mixing time and thusserves as a very relevant quantity when conducting mixing time analyses.
2.2 The Role Of Tasks And Non-stationarity
The preceding single-task formulation of RL in continuing environments can, in fact, be used toformulate continual RL, which typically makes notions of multiple tasks and non-stationarity explicit.
This is indeed a useful construct because arbitrary non-stationarity precludes any consistent signalto learn from, and thus further assumptions about the environment structure must be made to makeprogress [1].
In this paper, we consider tasks as sub-regions of the total MDP (which alternatively can be thought of as independent MDPs connected together) and we assume the transition dynamicsbetween and within tasks are both stationary.
As highlighted in [1], this setting is quite general asit is capable of modeling many continual RL problems.
For example, it easily extends to partiallyobservable variants: if the task component is not directly observed, the problem appears non-stationaryfrom the perspective of an agent that learns from observations of only the within-task state.
To help the reader picture t π mix in continual RL, we offer the example depicted in Figure 1.
Here, theagent’s state space S is decomposed into a task component, z ∈ Z and a within-task component, 118 x ∈ X z .
For a policy, π , the pair of Markov chains induced by marginalizing over x and over z havediameters D πz and D πx , respectively.
Using the result connecting mixing times and diameters here,we can establish a universal rule for problems of this type: 2 t π mix ≥ D π ≥ D πx ≥ D πz , where D π isthe diameter over the entire state space.
This rule highlights the intimate connection between mixingtimes and continual RL: task locality and bottleneck structure inherently lead to environments withcorrespondingly high mixing times due to high minimum diameters between states in different tasks.
Scalable MDPs
In this section, we formalize a scalable family of MDPs C σ , using a scaling function σ .
We consider MDPs with state space S parametrized by an n -dimensional vector q ∈ R n of intrinsic dimensions(see Figure 2). In general, q can contain extensive components determining spatial properties of S(e.g.
spatial dimensions of a grid world), as well as intensive components that determine qualitativefeatures (e.g. number of ‘cherry’ states in a rewarded sequence of target states in a grid world). Asa continuous parametrization of the MDP, q serves as a useful target for formalizing scaling. Weintroduce the set of scaling functions Σ on q as continuous functions σ : R n × R → R n parametrizedby a scalar ν ∈ R that takes any q 0 to q ν = σ ( q 0 , ν ) , with σ ( · , 0) as the identity map.
These functionsrespect the following expansive property: ∂q ν,i /∂ν ≥ 0 for all i = 1 , . . . , n , and ∂q ν,i /∂ν >
0for at least one i .[-
-]
Thus, each σ ∈ Σ induces C σ , a ν -parametrized abstract class of MDPs along asmooth path in q -space.
An important example of σ is proportional scaling , σ ( q, ν )
= c ( ν ) ◦ q withc i ( ν )
∈ Θ( ν ) .
In this case, |S ν | ∈ Θ( ν n ) where S ν is the state space speciﬁed by q ν .
Not all components must be scaled, however, so that scalings that affect only n (cid:48) ≤ n components ofq are valid. We give two such scaling examples in Figure 2 that scale only an intensive (1) and anextensive (2) parameter, respectively. Here, the intensive parameters increase the diameter via the number of bottlenecks, while the extensive parameters do so via the effective diameter of each region. Applying this distinction to our task formulation and example in Figure 1, intensive scaling increasesthe number of tasks, and extensive scaling increases the size of each task.
Concretely, a region R ⊆ S is a connected subset of states with steady-state probability µ π ( R )
=
144 (cid:80) s ∈R µ π ( s ) .
The boundary of R is a subset ∂ R ⊆ R with states having ﬁnite probability oftransitioning to at least one state that is outside of R , (cid:80) s (cid:48)
∈S\R T π ( s (cid:48) | s ) >
∀ s ∈ ∂ R .
As the statespace scales with ν , so too will the regions within it and their boundaries.
These do so at differentrates so that boundary-to-interior volume ratios also scale.
If the size of the boundary grows fasterthan the region with ν , there is a ﬁnite state space size |S| max , at which the region is undeﬁned.
In 149 that case, the problem type is of limited complexity and long-term interest by its nature.
We arethus interested in problems where regions maintain an interior as they are scaled (as measured bysteady-state probability) in the limit of large ν , leading to the following deﬁnition of a scalable MDP.
Deﬁnition
A scalable MDP is a family of MDPs C σ ⊆ M arising from some proportional scalingoperation σ on a subset n (cid:48) ≤ n of the intrinsic dimensions of S such that for any applicable scalableregion, R , the inequality µ π ∗ ( ∂ R ) < µ π ∗ ( R ) ≤ 1 / 2 is preserved as |S| → ∞ ∀M ∈ C σ .
Polynomial Mixing Times
Prior work analyzing the scaling of mixing times makes the practical distinction between slow mixing
Markov chains that scale exponentially with a size parameter and rapid mixing Markov chains thatscale at most polynomially.
Since the learning speed of any algorithm is lower bounded by the mixingtime of the optimal policy [10], environments with a slow mixing Markov chain induced by theoptimal policy may very well be outside the practical reach of RL algorithms.
In this section wewould like to further characterize mixing times over the space of practically solvable problems ofsufﬁcient complexity to still be of interest for real-world applications.
For continuing environments, the tightest known lower bound satisfying Assumption 1 1 has H -stepregret Regret ( H ) ∈ Ω( (cid:112)
D π ∗ |S||A| H ) ⊆
Ω( (cid:112) D ∗ |S||A| H ) [12].
Additionally, because we know 165 that D ∗ ≥ log |A| ( |S| ) − 3 [12], we can bound our regret in the general case as Regret ( H ) ∈˜Ω( (cid:112) |S||A| H ) .
Few practical problems are simple enough to exhibit diameters with such logarithmic 167 contributions.
More typical and thus of greater interest are diameters (and mixing times) havingpolynomial scaling in |S| .
Focusing on polynomial scaling is thus warranted, and provides more 169 stringent lower bounds on regret.
We thus consider the following deﬁnition:
Deﬁnition
A set or family of MDPs C ⊆ M has a polynomial mixing time if the environmentmixing dynamics contributes a Ω( |S| k ) multiplicative increase for some k > 0 to the intrinsic lower 172 bound on regret as [Equation]
An immediate utility of Deﬁnition 2 is that it subsumes the diversity of different diameter and mixingtime deﬁnitions by explicitly stating their equivalence in scaling with respect to the state space size:
Proposition 1 If all MDPs M within the subclass of MDPs C ⊆ M have t π ∗ ret , t π ∗ ces , t π ∗ mix , D π ∗ , or D ∗ ∈ Ω( |S| k ) for some k > 0 we can say that C has a polynomial mixing time.
We can thus hereon focus on formulating scaling MDPs via their state space size.
One way tounderstand how a region, R , contributes to the mixing time is through its residence time , t π ∗ R , i.e. theaverage time that π ∗ spends in R during a single visit.
Using results from the theory of bottleneckratios in Markov chains [11], we show that if scaling the MDP leads to t π ∗ R increasing by a polynomialfactor in |S| for any, not-too-big R , then the mixing time also increases at a polynomial rate:
Proposition 2 Any family of MDPs C ⊂ M with scalable regions has a polynomial mixing time ifthere exists a region R with µ π ∗ ( R ) ≤ 1 / 2 such that E µ π ∗ [ t π ∗ R ] ∈ Ω( |S| k ) for some k > 0 .
The purpose of Proposition 2 and Figure 2 are to provide intuition to readers about why scalable
MDPs inherently must have polynomial mixing times.
Our paper’s main result builds off Proposition2, to make a general statement about the set of all possible scalable MDPs M Σ , obtained from Σ , theset of all scaling functions:
Theorem 1 (Polynomial Mixing for Scalable MDPs): Any scalable MDP C σ ⊆ M Σ in the set of allpossible scalable MDPs M Σ has a polynomial mixing time.
Myopic Bias During Scaling
Monte Carlo sampling and bootstrapping are the two primary policy evaluation frameworks for RL[5].
Both implicitly assume that a ﬁnite and ﬁxed maximum frequency f ∗ ( π ) ∈ [0 , 1] of policyimprovement steps is possible for unbiased updates regardless of the problem size.
We now proposethree corollaries of Theorem 1 that together argue how polynomial mixing times invalidate thisassumption simply because f ∗ ( π ) ≥ 1 /t π mix , and t π mix grows over scalable MDPs.
Monte Carlo sampling: an approach for retrieving unbiased, although often high variance, estimatesof performance of a single policy by direct sampling from the environment or a high quality model ofthe environment.
In continuing environments with polynomial mixing times, this sampling procedureposes a problem when optimizing for ρ ( π ) : To get an unbiased estimate of ρ ( π ) , we must be ableto sample from the steady-state distribution µ π ( s ) , which is only available after t π mix steps in theenvironment.
Moreover, as demonstrated by Zahavy et al.
[13] in the general case where no upperbound on the mixing time is known apriori, O ( |S| t π mix ) samples are needed to retrieve a singleunbiased sample from µ π ( s ) .
As such, it is clear that the length of the policy evaluation phasestrongly depends on the mixing time of the current policy π and thus the maximum frequency f ∗ ( π )of unbiased policy improvement steps decreases as the mixing time increases:
Corollary 2 A Monte Carlo sampling algorithm for policy π in scalable MDP C σ has a maximumfrequency of unbiased policy improvement steps f ∗ ( π ) → 0 as |S| → ∞ .
For scalable MDPs of signiﬁcant size, Corollary 2 implies that model-free Monte Carlo methodsperform unbiased updates arbitrarily slowly and that model-based Monte Carlo methods will needarbitrary amounts of compute for unbiased updates even when a true environment model is known.
Assuch, to address scaling concerns over large horizons, bootstrapping methods based on the Bellmanequation are generally recommended [5].
Bootstrapping for Evaluation: a process of estimating the value function of the current policy atthe current state V π ( s ) in terms of our prior estimate of the value of our policy at the next state
V π ( s (cid:48) ) following the Bellman equation.
This process is referred to as iterative policy evaluation[7] and is known to converge to the true V π in the limit of inﬁnite evaluations of a ﬁxed policy.
However, in practice, RL algorithms constantly change their policy as they learn and only can affordpartial backups during policy evaluation when applied to large-scale domains.
Indeed, a foundationaltheoretical principle in RL is that as long as the agent constantly explores all states and actions withsome probability, bootstrapping with only partial backups will still allow an agent to learn π ∗ in thelimit of many samples [7].
Sample efﬁciency can still be quite poor since partial backups insert biasinto each individual policy evaluation step.
This bias, referred to as staleness [14], arises when thevalue function used for bootstrapping at the next state is reﬂective of an old policy that is currentlyout of date.
Unfortunately, if we want to avoid staleness bias during learning, the length of policyevaluation for each policy must once again depend strongly on the mixing time:
Corollary 3 A bootstrapping algorithm with policy π in scalable MDP C σ has a maximum frequencyof unbiased policy improvement steps f ∗ ( π ) → 0 as |S| → ∞ .
Bootstrapping for Improvement: the process of using bootstrapping for policy improvement aspopularized by dynamic programming algorithms such as policy iteration and value iteration, as wellas by RL frameworks such as temporal difference (TD) learning and actor-critic.
The theoreticalfoundation of these approaches is the policy improvement theorem , which demonstrates the valueof taking a greedy action with respect to the estimated action-value function Q π ( s, a ) at each step.
At each state s ∈ S we can consider a modiﬁed policy π m ( a, s, π ) for each potential action a ∈ A 234 that defaults to the policy π at all other states.
The greedy policy π (cid:48) is then deﬁned as π (cid:48) ( s ) = argmax a Q π ( s, π m ( a, s, π )) and π (cid:48) ( s (cid:48) ) =
π ( s (cid:48) ) ∀ s (cid:48) ∈ S \ s .
The policy improvement theoremthen tells us that policy changes towards π (cid:48) are worthwhile as V π ( s ) ≤ Q π ( s, π (cid:48)
( s ))
≤ V π (cid:48)
( s )eventually yielding π ∗ in the limit of many changes [5].
However, this improvement is unlikely to beefﬁcient when mixing times are high.
To demonstrate this, we decompose value into the transient andlimiting components respectively
V π ( s ) =
V π trans ( s ) + V π lim ( s ) .
While clearly V π ( s ) ≤ Q π ( s, π (cid:48) ( s )) ,the functions both follow π into the future, so their limiting distribution is the same implying that V π trans ( s ) ≤ Q π trans ( s, π (cid:48) ( s )) and V π lim ( s ) =
Q π lim ( s, π (cid:48)
( s )) .
Corollary 4 Policy improvement steps with bootstrapping based on the Bellman optimality operatorguarantee monotonic improvement for V π trans , but do not for V π lim .
See Appendix A for details on how these ideas connect to relevant off-policy or ofﬂine RL approachesand to the literature on catastrophic forgetting in continual RL.
Empirical Analysis of Mixing Behavior on Atari
We have established theoretically that scalable MDPs have polynomial mixing times and that polynomial mixing times present signiﬁcant optimization difﬁculties for current approaches to RL.
However,we still must demonstrate 1) that scalable MDPs are a useful construct for understanding the scalingprocess within modern continual RL benchmarks and 2) that large mixing times become a signiﬁcantpractical impediment to performing reliable policy evaluation in these domains.
Towards this end,we consider empirical scaling of the mixing time with respect to the number of distinct tasks, |Z| , 253 and to the task duration, τ , which controls the bottleneck structure.
For this purpose, we evaluate aset of high-quality pretrained policies.
In both cases, we report empirical results via the dependenceof the (cid:15) -return mixing time (equation 2) on the relative precision with which the reward rate isestimated.
Presenting the relationship in this way ensures we isolate the contribution that myopicpolicy evaluation has on the performance relative to the optimum and that high mixing times are notinﬂated by spurious sources of distributional mismatch that do not hinder policy evaluation.
Domain of Focus.
We perform experiments involving sequential interaction across 7 Atari environments: Breakout, Pong, SpaceInvaders, BeamRider, Enduro, SeaQuest and Qbert .
This is a typicalnumber of tasks explored in the continual RL literature [15, 16, 17, 18, 19] and high-performing,pretrained policies for them are publicly available [20].
We consider a sequential Atari setup wherethe individual environments can be seen as sub-regions of a larger environment that are stitchedtogether (see Figure 1; also Appendix A for further details).
We leverage task labels to use thepretrained model speciﬁc to each task as our behavior policy as appropriate when tasks change.
Empirical Mixing Time Estimation.
In order to quantitatively estimate the (cid:15) -return mixing timet π ret ( (cid:15) ) in equation 2, we need to estimate two terms: the average true reward rate ρ ( π ) (which isagnostic to the start state), and the h -step undiscounted return ρ ( h, s 0 , π ) from a start state s 0 for alls ∈ S .
To calculate ρ ( π ) we unroll the policy π in the environment for a large number of time steps(more than a million steps) while accumulating the rewards that we received and ﬁnally computethe average over the total number of environment interactions.
Estimating ρ ( h, s, π ) for every startstate s ∈ S in equation 2 is challenging because of the large state spaces in Atari.
Hence we rely onapproximations where we choose a subset of states from S to estimate ρ ( h, s, π ) .
However, sincethe estimates could be biased based on the start states we choose, instead of randomly sampling aﬁxed set of start states, we leverage reservoir sampling [21] to ensure that the limited fraction ofpossible start states that we consider is unbiased according to the on-policy distribution.
Conservativedeﬁnitions the mixing time take the maximum over start states [10].
Here, since we are interested inthe characteristic mixing time size, we report the average among start states weighted according tothe on-policy distribution, which ensures that the mixing times reported are representative of thoseactually needed for evaluation over the course of on-policy training.
Finally, in order to provide anintuitive interpretation for (cid:15) across domains, we parametrize (cid:15) using the relative error , (cid:15)/ρ ( π ) , andcalculate the (cid:15) -return mixing times for ﬁxed relative error.
See Appendix B for further details.
Setting of Interest.
We focus our experiments on the following scalable MDP formulation of Atarithat is broadly representative of the majority of work on continual RL [15, 16, 17, 18, 19]:
Example 1 (Continual Learning with Passive Task Switching): Consider an environment with tasksz ∈ Z and within task states x ∈ X z .
The residence time of the region encompassed by task z before moving to another region is ﬁxed for all tasks at τ for any π such that t π X z = τ ∀ z ∈ Z .
Regardlessof the way that tasks are connected, the diameter of such an MDP must scale as D ∗ ∈ Ω( τ |Z| ) 289 because the residence time bounds the minimum possible time to travel between two states in differenttasks.
Expected mixing times will thus grow at least linearly as either τ or |Z| are increased.
Mixing as a Function of Relative Error.
As the task connection structure following Example 1 cantake any arbitrary form, we have focused our experiments on random transitions between tasks.
In
Figure 3 we plot both t π ret ( (cid:15) ) and the value normalized by the diameter lower bound t π ret ( (cid:15) ) /τ |Z| forτ = 10,000 and |Z| =
7 as a function of the relative error of ρ ( π, s, h ) w.r.t ρ ( π ) .
From this analysisit is clear that the lower bound on the diameter is a very conservative estimate of the mixing timewhere t π ret ( (cid:15) ) /τ |Z| < 1 only for reward rates estimated to a poor tolerance of more than 30% error.
Scaling |Z| .
We now analyze Example 1 in the case where the state space and diameter grow as we 298 increase the number of tasks |Z| .
Speciﬁcally we scale |Z| from 2 to 4 to 6 as τ is kept ﬁxed at 1,000 299 and generate 10 random task combinations from the list of 7 for each value of |Z| .
In Figure 4 we 300 plot the resulting mixing times for a precision of 10% error in approximating the reward rate.
Weﬁnd that there is a linear trend in the expected mixing time and not just the theoretical lower bound.
Scaling τ .
We ﬁnally analyze Example 1 in the case where the state space is held constant but thediameter grows with the increasingly severe bottleneck structure as we increase τ .
Here we consider|Z| = 7 and τ = 1,000, 3,000, 10,000, and 30,000, while calculating ρ ( h, s, π ) and ρ ( π ) by unrollingthe policy π for τ × 1,000 environment steps each.
In Figure 5 we once again plot both t π ret ( (cid:15) )
andthe value normalized by the diameter lower bound t π ret ( (cid:15) ) /τ |Z| to ﬁnd that empirical average mixingtimes scale linearly with increasing τ and not just the lower bound.
In summary, we have observed from Figure 3 that, in the case of Atari, average mixing times reacha staggering high for lower relative errors, and shorter mixing times are associated with very highrelative errors.
From Figures 4 and 5 we observed that, for a broad class of algorithms, there seems tobe a linear dependency of the expected mixing time with increasing |Z| and increasing τ .  Implications of Polynomial Mixing
Optimization Instability on Atari.
While we have empirically established that mixing timesscale to be quite large in the Atari domain following Example 1 and established theoretically how RL approaches struggle when this is the case in Section 5, we would like to emphasize that theprior literature applying deep RL to continual learning on Atari has empirically come to the sameconclusion.
Indeed, [16] looked at continual learning across 10 Atari games with a randomizedduration between task switches and found that all approaches were signiﬁcantly outperformed by thesingle task baseline, reﬂecting signiﬁcant instability in optimizing across tasks.
Additionally, perhapsthe most common setting in the literature considered by [17], [18], and [22] is to train on a set oftasks where |Z| = 6 and τ = 50,000,000.
Note that extrapolating from our results would indicateaverage mixing times on the order of ≈ 15 billion steps for a precision of 5% error or ≈ 5 billionsteps for a precision of 10% error.
Across a number of RL approaches tried on these domains, as wewould expect, all approaches have experienced signiﬁcant instability in their learning curves.
A verypopular failure mode is approaches that display too much plasticity such that their performance fora task goes through big peaks and valleys resulting from myopic bias towards optimization on thecurrent task.
This phenomena is often known in the literature as catastrophic forgetting [23].
Development of New Approaches.
While the focus of our paper is on highlighting the difﬁcultyof the polynomial mixing time problem, not proposing an approach to solve it, we believe thatour analysis provides signiﬁcant insights that the community will be able to draw on to buildbetter and more theoretically grounded approaches for continual RL.
In the context of commonlyconsidered problems that follow the high-level structure of Example 1, we have showcased the deepconnection between catastrophic forgetting and myopic bias in the presence of very large mixingtimes.
In Appendix A we demonstrate how this perspective explains some of the value of replay basedapproaches in this setting.
Moreover, this perspective naturally points towards approaches that directlyreason over the steady-state distribution as a natural solution to the catastrophic forgetting problem.
In Appendix C we consider simple tabular RL experiments across 3 example classes of scalable MDPs.
Our analysis demonstrates that on-policy and off-policy versions of Q-Learning that perform policyevaluation using an estimate of the steady-state distribution derived from matrix inversion of a tabularenvironment model, consistently outperform standard model-based and model-free baselines in termsof lifelong regret as environments are scaled up.
Even though estimating the steady-state distributionwould clearly be quite challenging in complex state and observation spaces, recent approaches havemade great strides towards developing practical and scalable approaches leveraging data buffers [24].
Looking Forward.
In this work, we have considered the implicit premise of the current continual
RL literature that results from currently manageable small-scale experiments are indicative ofperformance for the large-scale aspirational use cases of the future.
In particular, we have highlightedthat mixing times will scale signiﬁcantly as the problems we deal with are scaled and how traditionalapproaches to RL are ill-suited to deal with this.
We hope that our work will encourage the communityto carefully consider the theoretical implications of scaling environments, especially with regard tohow proposed continual RL algorithms scale with the mixing time.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
[Table]