Continual Learning In Environments With Polynomial Mixing Times
Abstract
The mixing time of the Markov chain induced by a policy limits performance inreal-world continual learning scenarios.
Yet, the effect of mixing times on learningin continual reinforcement learning (RL) remains underexplored.
In this paper,we characterize problems that are of long-term interest to the development ofcontinual RL, which we call scalable MDPs, through the lens of mixing times.
Inparticular, we theoretically establish that scalable MDPs have mixing times thatscale polynomially with the size of the problem.
We go on to demonstrate thatpolynomial mixing times present signiﬁcant difﬁculties for existing approaches,which suffer from myopic bias and stale bootstrapped estimates.
To validate ourtheory, we study the empirical scaling behavior of mixing times with respect to thenumber of tasks and task duration for high performing policies deployed acrossmultiple Atari games.
Our analysis demonstrates both that polynomial mixingtimes do emerge in practice and how their existence may lead to unstable learningbehavior like catastrophic forgetting in continual learning settings.
Introduction
Continual reinforcement learning (RL) [1] is an aspirational ﬁeld of research confronting the difﬁculties of long-term, real-world applications by studying problems of increasing scale, diversity,and non-stationarity.
The practical requirement for researchers to work on problems of reasonablecomplexity in the short-term presents a meta-challenge: choosing the right small-scale problemsso that the approaches we develop scale up to the use cases of the future.
Here, we address thismeta-challenge by formalizing RL problems that vary in size and by analyzing the scaling behaviorof popular RL algorithms.
In particular, we analyze on the often-ignored mixing time that expressesthe amount of time until the agent-environment dynamics converge to some stationary behaviour.
Towards this end, we speciﬁcally make the following contributions in this work:
Scalable MDPs: We propose the formalism of scalable MDPs in Deﬁnition 1 to characterizean abstract class of MDPs where the MDPs within the class are differentiated based on a
changing scale parameter.
Understanding the inﬂuence of this scale parameter on learning
can promote better understanding of the meta-challenge of extrapolating our results on small
scale problems to large scale problems of the same class.
Polynomial Mixing Times: Theorem 1 establishes the key result of this paper that as any
scalable MDP is scaled, its mixing time must grow polynomially as a function of the growing
state space.
This has major implications for regret analysis in these MDPs.
Myopic Bias
During Scaling: We demonstrate in Corollaries 2, 3, and 4 that Theorem
1 implies traditional approaches to RL cannot efﬁciently scale to problems of large size
without experiencing myopic bias in optimization that slows down learning.
Empirical Analysis of Mixing for Continual RL: We back up our theory with empiricalanalysis of mixing time scaling in Continual RL settings based on the Atari benchmark.
Ouranalysis provides insight into why agents experience signiﬁcant catastrophic forgetting orother optimization instability when learning in these domains.
We hope that our analysis intothe fundamental driver of these issues can open the door for more successful and principledapproaches to Continual RL domains like these moving forward.
We refer readers to Appendix D for detailed proofs of all propositions and theorems in our paper.
Formalizing Our Continual RL Setting
Aiming to characterize a broad range of settings, we focus on the formulation of RL in continuingenvironments.
As explained in Khetarpal et al.
[1], not only are supervised learning and episodic RLspecial cases of RL in continuing environments, but so are their non-stationary variants (continualsupervised learning and continual episodic RL), despite violating the stationarity assumptions oftheir root settings.
Indeed, approaches that do not acknowledge the more general RL in continuingenvironments setting exhibit myopic bias in their optimization when faced with non-stationarity [1].
Unfortunately, the popular discounted reward setting inserts the very same kind of myopic biasin optimization that we would like to avoid [2].
As typically implemented, discounting does notcorrespond to the maximization of any objective function over a set of policies [3] and the policygradient is not the gradient of any function [4].
These fundamental issues do not resolve as the discountfactor approaches 1 [3] and discounting does not inﬂuence the ordering of policies, suggesting it likelyhas no role to play in the deﬁnition of the control problem [5].
In contrast, the average reward perstep objective, which explicitly includes an average over the stationary distribution, avoids inducingmyopic biases and hence is well-suited for continual RL problems [5].
2.1 Average Reward RL in Continuing Environments
RL in continuing environments is typically formulated using a ﬁnite, discrete-time, inﬁnite horizon
Markov Decision Process (MDP) [6, 7], which is a tuple M = (cid:104)S , A , T, R (cid:105) , where S is the set of 60 states,
A is the set of actions, R : S×A → [0 , R max ] is the reward function, and T : S×S×A → [0 , 1] 61 is the environment transition probability function.
At each time step, the learning agent perceivesa state s ∈ S and takes an action a ∈ A drawn from a policy π : S ×
A → [0 , 1] with internal 63 parameters θ ∈ Θ .
The agent then receives a reward R ( s, a ) and with probability T ( s (cid:48) | s, a ) entersnext state s (cid:48) .
Markov chains may be periodic and have multiple recurrent classes, but optimality isdifﬁcult to deﬁne in such cases [8], making the following assumption necessary for analysis:
Assumption 1 All stationary policies are aperiodic and unichain, meaning they give rise to a Markovchain with a single recurrent class that is recurrent in the Markov chain of every policy.
Any RL problem may be modiﬁed such that Assumption 1 holds by adding an arbitrarily smallpositive constant (cid:15) to all transition probabilities in T ( s (cid:48) | s, a ) and renormalizing in which case theeffect on the objective of each stationary policy is O ( (cid:15) )
An important corollary to Assumption 1is that the steady-state distribution µ π induced by the policy π is independent of the initial state:
Corollary 1 All stationary policies π induce a unique steady-state distribution
[Equation]
that is independent of the initial state such that
[Equation]
Corollary 1 implies that the long-term rewards of any π will be independent of the current state.
Assuch, the average reward per step objective ρ ( π ) can be deﬁned independently of its starting state [5]:
[Equation]
Computing the average reward with the last expression is limited by the amount of time the Markovchain induced by the policy T π
[Equation]
s
[Equation]
) needs to be run for before reaching the steady-state distribution µ π ( s ) .
This amount of time is referred to in the literature as the mixingtime of the induced Markov chain.
We denote t π mix ( (cid:15) ) as the (cid:15) -mixing time of the chain induced by π :
[Equation]
where d TV is the total variation distance between the two distributions.
The so-called conventionalmixing time is deﬁned as t π mix ≡ t π mix (1 / 4) .
The conventional mixing time only gives insight aboutdistributional mismatch with respect to the steady-state distribution, which led [10] to introduce thenotion of a mismatch with respect to the reward rate.
The (cid:15) -return mixing time is a measure of thetime it takes to formulate an accurate estimate of the true reward rate.
More formally, if we denotethe h -step average undiscounted return starting from state s as ρ ( π, s, h ) , then we deﬁne the (cid:15) -returnmixing time as:
[Equation]
We will come back to this deﬁnition when we present our experiments.
As emphasized in Levin and
Peres [11], however, one should not get bogged down in the use-case speciﬁc deﬁnitions of mixingtime (we prove their equivalent scaling behaviour in fact, c.f. Proposition 1 ).
Rather, it is their sharedproperty of being determined by both a policy π and environment M , and not simply a property ofthe environment itself, that is important.
The environment’s contribution can be assessed throughmixing times obtained from extreme policies such as the optimal policy, t π ∗ mix .
An alternative approach for quantifying mixing is by considering the structure of the transition matrix
T π ( s (cid:48) | s ) induced by π .
It is well known that the mixing properties of a Markov chain are governed bythe spectral gap derived from the eigenvalues of the matrix T π ( s (cid:48) | s ) .
Unfortunately, it is difﬁcult toreason about the spectral gap of a class of MDPs directly.
Towards this end, one useful interpretationof a Markov chain is as a random walk over a graph G ( π, T ) with vertex set S and edge set { ( s, s (cid:48) ) } ,for all s and s (cid:48) satisfying T π ( s (cid:48) | s )
+ T π ( s | s (cid:48) ) > 0 .
The diameter D π of the Markov chain inducedby π is thus the diameter or maximal graph distance of G ( π, T ) .
It is deﬁned using the hitting timet π hit ( s 1 | s 0 ) , the ﬁrst time step in which s 1 is reached following the Markov chain T π ( s (cid:48) | s ) from s 0 ,
[Equation]
D ∗ then denotes the minimum diameter achieved by any policy.
All MDPs that follow Assumption1 have ﬁnite diameter D π for all policies [12].
Levin and Peres [11] (eq. 7.4) established therelationship, 2 t π mix ≥ D π , so that the diameter provides a lower bound for the mixing time and thusserves as a very relevant quantity when conducting mixing time analyses.
2.2 The Role Of Tasks And Non-stationarity
The preceding single-task formulation of RL in continuing environments can, in fact, be used toformulate continual RL, which typically makes notions of multiple tasks and non-stationarity explicit.
This is indeed a useful construct because arbitrary non-stationarity precludes any consistent signalto learn from, and thus further assumptions about the environment structure must be made to makeprogress [1].
In this paper, we consider tasks as sub-regions of the total MDP (which alternativelycan be thought of as independent MDPs connected together) and we assume the transition dynamicsbetween and within tasks are both stationary.
As highlighted in [1], this setting is quite general asit is capable of modeling many continual RL problems.
For example, it easily extends to partiallyobservable variants: if the task component is not directly observed, the problem appears non-stationaryfrom the perspective of an agent that learns from observations of only the within-task state.
Thatsaid, the continual learning literature demonstrates that optimization issues are generally experiencedwhether task labels are observed or not [13].
Our analysis of mixing times in this paper is an attemptto understand these difﬁculties even when the total MDP is fully observable and stationary.
To help the reader picture t π mix in continual RL, we offer the example depicted in Figure 1.
Here, theagent’s state space S is decomposed into a task component, z ∈ Z and a within-task component, 122 x ∈ X z .
For a policy, π , the pair of Markov chains induced by marginalizing over x and over z havediameters D πz and D πx , respectively.
Using the result connecting mixing times and diameters here,we can establish a universal rule for problems of this type: 2 t π mix ≥ D π ≥ D πx ≥ D πz , where D π isthe diameter over the entire state space.
This rule highlights the intimate connection between mixingtimes and continual RL: task locality and bottleneck structure inherently lead to environments withcorrespondingly high mixing times due to high minimum diameters between states in different tasks. zzzz N Rule xxx [Figure]
[Figure]
Scalable MDPs
In this section, we formalize the notion of scaling in the context of MDPs and provide intuition aboutthe effect of this scaling with the aid of the schematic shown in Figure 2.
We assume MDPs canbe described in terms of an n -dimensional parameter vector q ∈ R n , and that some subset of thesedimensions are the parameters of interest that will be scaled up.
We would like to analyze the effectof this scaling on mixing times.
2 With reference to Figure 1, these parameters can be non-spatialsuch as the number of tasks, or spatial such as the size of an individual task.
More generally andwith reference to Figure 2, the effect of scaling up parameters can contribute to longer mixing timesby adding bottleneck states (akin to increasing the number of tasks) and increasing the size of theregions generated by a bottleneck (akin to increasing the size of each task). How parameters arescaled is speciﬁed by a scaling function, σ controlled by scaling parameter, ν . {+
+}
All MDPs accessiblethrough σ by varying ν form a family of MDPs, C σ = {M ν } , where M ν is the MDP speciﬁed by 140 q ν .
In particular, we consider proportional scaling functions, for which a subset of elements of qare scaled linearly with ν .
In our experiments, we show results for proportional scaling of two MDPparameters central to continual RL: the number of tasks and the time between task switches.
Figure
As the MDP scales up according to σ , the overall state space size will grow with |S| → ∞ as ν → ∞ .
144 Regions within the state space can also grow and the steady-state probability on them from π maychange as a result.
A region R ⊆ S is a connected subset of states with steady-state probability 146 µ π (
R ) =
(cid:80) s ∈R µ π ( s ) .
The boundary of R is a subset ∂ R ⊆ R with states having ﬁnite probabilityof transitioning to at least one state that is outside of R , (cid:80) s (cid:48)
∈S\R T π ( s (cid:48) | s ) >
∀ s ∈ ∂ R .
Wedenote the scalable regions and boundaries R ν and ∂ R ν , respectively.
If the size of a region’s 149 boundary grows faster than the region’s interior, there is a ﬁnite state space size at which the regionno longer has an interior.
In that case, the problem type is of bounded complexity and not relevant tothe development of scaling friendly algorithms.
We are thus interested in problems where scalableregions can maintain an interior as they are scaled (as measured by steady-state probability) in thelimit of large ν . See Appendix D for a precise formulation leading to the following deﬁnition:
Deﬁnition
A scalable MDP is a family of MDPs C σ = {M ν } arising from a proportional scaling 155 function σ satisfying the property that there exists an initial scalable region R 0 with ﬁnite interior,µ π ∗ ( ∂ R 0 ) < µ π ∗ ( R 0 ) , that scales so that µ π ∗ ( ∂ R ν ) < µ π ∗ ( R ν ) as ν → ∞ and thus |S| → ∞ .
Polynomial Mixing Times
Prior work analyzing the scaling of mixing times makes the practical distinction between slow mixing
Markov chains that scale exponentially with a size parameter and rapid mixing Markov chains thatscale at most polynomially.
Since the learning speed of any algorithm is lower bounded by the mixingtime of the optimal policy [10], environments with a slow mixing Markov chain induced by theoptimal policy may very well be outside the practical reach of RL algorithms.
In this section wewould like to further characterize mixing times over the space of practically solvable problems ofsufﬁcient complexity to still be of interest for real-world applications.
For continuing environments, the tightest known lower bound satisfying Assumption 1 4 has H -stepregret Regret ( H ) ∈ Ω( (cid:112)
D π ∗ |S||A| H ) ⊆
Ω( (cid:112) D ∗ |S||A| H ) [12].
Additionally, because we know 167 that D ∗ ≥ log |A| ( |S| ) − 3 [12], we can bound our regret in the general case as Regret ( H ) ∈˜Ω( (cid:112) |S||A| H ) .
Few practical problems are simple enough to exhibit diameters with such logarithmic 169 contributions.
More typical and thus of greater interest are diameters (and mixing times) havingpolynomial scaling in |S| .
Focusing on polynomial scaling is thus warranted, and provides more 171 stringent lower bounds on regret.
We thus consider the following deﬁnition:
Deﬁnition
A set or family of MDPs C has a polynomial mixing time if the environment mixingdynamics contributes a Ω( |S| k ) multiplicative increase for some k > 0 to the intrinsic lower bound 174 on regret as |S| → ∞ ∀M ∈ C .
An immediate utility of Deﬁnition 2 is that it subsumes the diversity of different diameter and mixingtime deﬁnitions by explicitly stating their equivalence in scaling with respect to the state space size:
Proposition 1 If all MDPs M within the subclass of MDPs C have t π ∗ ret , t π ∗ ces , t π ∗ mix , D π ∗ , or D ∗ ∈Ω( |S| k ) for some k > 0 we can say that C has a polynomial mixing time.
We can thus hereon focus on formulating scaling MDPs via their state space size.
One way tounderstand how a scalable region, R ν , contributes to the mixing time is through its residence time ,t π ∗ R , i.e. the average time that π ∗ spends in R during a single visit.
Using results from the theory ofbottleneck ratios in Markov chains [11], we show that if scaling the MDP leads to t π ∗ R increasing by apolynomial factor in |S| , then the mixing time also increases at a polynomial rate:
Proposition 2 Any scalable MDP family C σ exhibits a polynomial mixing time if there exists ascalable region R ν such that E µ π ∗ [ t π ∗ R ν ] ∈ Ω( |S| k ) for some k > 0 .
The purpose of Proposition 2 and Figure 2 are to provide intuition to readers about why scalable
MDPs inherently must have polynomial mixing times.
Our paper’s main result builds off Proposition2, to make a general statement about the set of all possible scalable MDPs:
Theorem 1 (Mixing Time Scaling): Any scalable MDP C σ has a polynomial mixing time.
Myopic Bias During Scaling
Monte Carlo sampling and bootstrapping are the two primary policy evaluation frameworks for RL[5].
Both implicitly assume that a ﬁnite and ﬁxed maximum frequency f ∗ ( π ) ∈ [0 , 1] of policyimprovement steps is possible for unbiased updates regardless of the problem size.
We now proposethree corollaries of Theorem 1 that together argue how polynomial mixing times invalidate thisassumption simply because f ∗ ( π ) ≥ 1 /t π mix , and t π mix grows over scalable MDPs.
In practice,algorithm designers cannot afford to wait until reaching the mixing time before updating their modelwhen mixing times are high. This results in a myopic bias in the policy improvement steps taken. Monte Carlo sampling:
In continuing environments with polynomial mixing times, this samplingprocedure poses a problem when optimizing for ρ ( π ) . To get an unbiased estimate of ρ ( π ) , we mustbe able to sample from the steady-state distribution µ π ( s ) , which is only available after t π mix stepsin the environment.
Moreover, as demonstrated by Zahavy et al.
[14] in the general case whereno upper bound on the mixing time is known apriori, O ( |S| t π mix ) samples are needed to retrieve asingle unbiased sample from µ π ( s ) .
As such, it is clear that the length of the policy evaluation phasestrongly depends on the mixing time of the current policy π and thus the maximum frequency f ∗ ( π )of unbiased policy improvement steps decreases as the mixing time increases:
Corollary 2 A Monte Carlo sampling algorithm for policy π in scalable MDP C σ has a maximumfrequency of unbiased policy improvement steps f ∗ ( π ) → 0 as |S| → ∞ .
For scalable MDPs of signiﬁcant size, Corollary 2 implies that model-free Monte Carlo methodsperform unbiased updates arbitrarily slowly and that model-based Monte Carlo methods will needarbitrary amounts of compute for unbiased updates even when a true environment model is known.
Assuch, to address scaling concerns over large horizons, bootstrapping methods based on the Bellmanequation are generally recommended [5].
Bootstrapping for Evaluation: This process is referred to as iterative policy evaluation [7] and isknown to converge to the true V π in the limit of inﬁnite evaluations of a ﬁxed policy.
However,in practice, RL algorithms constantly change their policy as they learn and only can afford partialbackups during policy evaluation when applied to large-scale domains.
Indeed, a foundationaltheoretical principle in RL is that as long as the agent constantly explores all states and actions withsome probability, bootstrapping with only partial backups will still allow an agent to learn π ∗ in thelimit of many samples [7].
Sample efﬁciency can still be quite poor since partial backups insert biasinto each individual policy evaluation step.
This bias, referred to as staleness [15], arises when thevalue function used for bootstrapping at the next state is reﬂective of an old policy that is currentlyout of date.
Unfortunately, if we want to avoid staleness bias during learning, the length of policyevaluation for each policy must once again depend strongly on the mixing time:
Corollary 3 A bootstrapping algorithm with policy π in scalable MDP C σ has a maximum frequencyof unbiased policy improvement steps f ∗ ( π ) → 0 as |S| → ∞ .
Bootstrapping for Improvement: the process of using bootstrapping for policy improvement aspopularized by dynamic programming algorithms such as policy iteration and value iteration, as wellas by RL frameworks such as temporal difference (TD) learning and actor-critic.
The theoreticalfoundation of these approaches is the policy improvement theorem , which demonstrates the valueof taking a greedy action with respect to the estimated action-value function Q π ( s, a ) at each step.
The policy improvement theorem then tells us that policy changes towards π (cid:48) are worthwhile as
V π ( s ) ≤ Q π ( s, π (cid:48) ( s ))
≤ V π (cid:48)
( s ) eventually yielding π ∗ in the limit of many changes [5].
However,this improvement is unlikely to be efﬁcient when mixing times are high.
To demonstrate this, wedecompose value into the transient and limiting components respectively
V π ( s ) =
V π trans ( s )+ V π lim ( s ) .
While clearly V π ( s ) ≤ Q π ( s, π (cid:48) ( s )) , the functions both follow π into the future, so their limitingdistribution is the same implying that V π trans ( s ) ≤ Q π trans ( s, π (cid:48) ( s )) and V π lim ( s ) =
Q π lim ( s, π (cid:48)
( s )) .
Corollary 4 Policy improvement steps with bootstrapping based on the Bellman optimality operatorguarantee monotonic improvement for V π trans , but do not for V π lim .
See Appendix A for details on how these ideas connect to relevant off-policy or ofﬂine RL approachesand to the literature on catastrophic forgetting in continual RL.
Empirical Analysis of Mixing Behavior on Atari
We have established theoretically that scalable MDPs have polynomial mixing times and that polynomial mixing times present signiﬁcant optimization difﬁculties for current approaches to RL.
However,we still must demonstrate 1) that scalable MDPs are a useful construct for understanding the scalingprocess within modern continual RL benchmarks and 2) that large mixing times become a signiﬁcantpractical impediment to performing reliable policy evaluation in these domains.
Towards this end,we consider empirical scaling of the mixing time with respect to the number of distinct tasks, |Z| , 248 and to the task duration, τ , which controls the bottleneck structure.
For this purpose, we evaluate aset of high-quality pretrained policies.
In both cases, we report empirical results via the dependenceof the (cid:15) -return mixing time (equation 2) on the relative precision with which the reward rate isestimated.
Presenting the relationship in this way ensures we isolate the contribution that myopicpolicy evaluation has on the performance relative to the optimum and that high mixing times are notinﬂated by spurious sources of distributional mismatch that do not hinder policy evaluation.
Domain of Focus.
We perform experiments involving sequential interaction across 7 Atari environments: Breakout, Pong, SpaceInvaders, BeamRider, Enduro, SeaQuest and Qbert .
This is a typicalnumber of tasks explored in the continual RL literature [16, 17, 18, 19, 20] and high-performing,pretrained policies for them are publicly available [21].
We consider a sequential Atari setup wherethe individual environments can be seen as sub-regions of a larger environment that are stitchedtogether (see Figure 1; also Appendix A for further details).
We leverage task labels to use thepretrained model speciﬁc to each task as our behavior policy as appropriate when tasks change.
Empirical Mixing Time Estimation.
In order to quantitatively estimate the (cid:15) -return mixing timet π ret ( (cid:15) ) in equation 2, we need to estimate two terms: the average true reward rate ρ ( π ) (which isagnostic to the start state), and the h -step undiscounted return ρ ( h, s 0 , π ) from a start state s 0 for alls ∈ S . We can then provide t π ret ( (cid:15) ) for any desired value of (cid:15) .
To calculate ρ ( π ) we unroll the policyπ in the environment for a large number of time steps (more than a million steps) while accumulatingthe rewards that we received and ﬁnally compute the average over the total number of environmentinteractions.
Estimating ρ ( h, s, π ) for every start state s ∈ S in equation 2 is challenging because ofthe large state spaces in Atari.
Hence we rely on approximations where we choose a subset of statesfrom S to estimate ρ ( h, s, π ) .
However, since the estimates could be biased based on the start stateswe choose, instead of randomly sampling a ﬁxed set of start states, we leverage reservoir sampling[22] to ensure that the limited fraction of possible start states that we consider is unbiased accordingto the on-policy distribution.
Conservative deﬁnitions the mixing time take the maximum over startstates [10].
Here, since we are interested in the characteristic mixing time size, we report the averageamong start states weighted according to the on-policy distribution, which ensures that the mixingtimes reported are representative of those actually needed for evaluation over the course of on-policytraining.
Finally, in order to provide an intuitive interpretation for (cid:15) across domains, we leveragethe relative error , (cid:15)/ρ ( π ) , as a common reference point for comparison. We can then calculate the(cid:15) -return mixing times for a ﬁxed relative error.
See Appendix B for further details.
Setting of Interest.
We focus our experiments on the following scalable MDP formulation of Atarithat is broadly representative of the majority of work on continual RL [16, 17, 18, 19, 20]:
Example 1 (Continual Learning with Passive Task Switching): Consider an environment with tasksz ∈ Z and within-task states x ∈ X z .
The residence time of the region encompassed by task z before 283 moving to another region is ﬁxed for all tasks at τ for any π such that t π X z = τ ∀ z ∈ Z .
Regardless of 284 the way that tasks are connected, the diameter of such an MDP must scale as D ∗ ∈ Ω( τ |Z| ) because 285 the residence time bounds the minimum possible time to travel between two states in different tasks.
Utilizing our formulation of scalable MDPs with initial MDP parameters q 0 = ( τ, |Z| ) , the space 287 of possible proportional scaling functions on τ only, |Z| only, and both τ and |Z| simultaneously 288 are σ τ ( q 0 , ν )
= ( τ + aν, |Z| ) , σ |Z| ( q 0 , ν ) = ( τ, |Z| + bν ) , and σ τ, |Z| ( q 0 , ν ) = ( τ + aν, |Z| + bν ) , 289 respectively, where a, b ∈ R span the respective spaces.
6.1 Overview of Empirical Findings
Mixing Time as a Function of Relative Error.
As the task connection structure following Example1 can take arbitrary form, we have focused our experiments on random transitions between tasks.
In
Figure 3 we plot both t π ret ( (cid:15) ) and t π ret ( (cid:15) ) /τ |Z| ,the value normalized by the diameter lower bound, forτ = 10,000 and |Z| = 7 as a function of the relative error of ρ ( π, s, h ) w.r.t ρ ( π ) , (cid:15) =
ρ ( π, s, h ) − ρ ( π ) .
We see the intuitive fact that higher demands for precision deﬁne longer mixing times and that it isclear that the lower bound on the diameter is a very conservative estimate of the mixing time wheret π ret ( (cid:15) ) /τ |Z| < 1 only for reward rates estimated to a poor tolerance of more than 30% error.
Scaling |Z| .
We now analyze Example 1 in the case where the state space and diameter grow as we 299 increase the number of tasks |Z| .
Speciﬁcally we scale |Z| from 2 to 4 to 6 as τ is kept ﬁxed at 1,000 300 and generate 10 random task combinations from the list of 7 for each value of |Z| .
In Figure 4 we 301 plot the resulting mixing times for a precision of 10% error in approximating the reward rate.
Weﬁnd that there is a linear trend in the expected mixing time and not just the theoretical lower bound.
Scaling τ .
We ﬁnally analyze Example 1 in the case where the state space is held constant but thediameter grows with the increasingly severe bottleneck structure as we increase τ .
Here we consider|Z| = 7 and τ = 1,000, 3,000, 10,000, and 30,000, while calculating ρ ( h, s, π ) and ρ ( π ) by unrollingthe policy π for τ × 1,000 environment steps each.
In Figure 5 we once again plot both t π ret ( (cid:15) )
andthe value normalized by the diameter lower bound t π ret ( (cid:15) ) /τ |Z| to ﬁnd that empirical average mixingtimes scale linearly with increasing τ and not just the lower bound. We also perform experiments [Figure]
sequential MuJoco with |Z| = 5 in order to verify the generality of our results across different 310 domains.
As expected, we can see in Figure 5a that the empirical mixing time scales linearly with τ .
6.2 Implications For Continual RL
In summary, we have observed from Figure 3 that, in the case of Atari, average mixing times reacha staggering high for lower relative errors, and shorter mixing times are associated with very highrelative errors.
From Figures 4 and 5 we observed that, for a broad class of algorithms, there seems tobe a linear dependency of the expected mixing time with increasing |Z| and increasing τ .
Optimization Instability for Continual Atari.
While we have empirically established that mixingtimes scale to be quite large in the Atari domain following Example 1 and established theoreticallyhow RL approaches struggle when this is the case in Section 5, we would like to emphasize thatthe prior literature applying deep RL to continual learning on Atari has empirically come to thesame conclusion.
Indeed, [17] looked at continual learning across 10 Atari games with a randomizedduration between task switches and found that all approaches were signiﬁcantly outperformed by thesingle task baseline, reﬂecting signiﬁcant instability in optimizing across tasks.
Additionally, perhapsthe most common setting in the literature considered by [18], [19], and [23] is to train on a set oftasks where |Z| = 6 and τ = 50,000,000.
Note that extrapolating from our results would indicateaverage mixing times on the order of ≈ 15 billion steps for a precision of 5% error or ≈ 5 billionsteps for a precision of 10% error.
Across a number of RL approaches tried on these domains, as wewould expect, all approaches have experienced signiﬁcant instability in their learning curves.
A verypopular failure mode is approaches that display too much plasticity such that their performance fora task goes through big peaks and valleys resulting from myopic bias towards optimization on thecurrent task.
This phenomena is often known in the literature as catastrophic forgetting [24].
Inspiring New Approaches for Continual RL
While the focus of our paper is on highlighting the difﬁculty of the polynomial mixing time problem,not proposing an approach to solve it, we believe that our analysis provides signiﬁcant insights thatthe community will be able to draw on to build better approaches for continual RL: Direct Steady-State Estimation:
In the context of commonly considered problems that follow thehigh-level structure of Example 1, we have showcased the deep connection between catastrophicforgetting and myopic bias in the presence of very large mixing times.
In Appendix A we demonstratehow this perspective explains some of the value of replay based approaches in this setting.
Moreover,this perspective naturally points towards approaches that directly reason over the steady-state distribution as a natural solution to the catastrophic forgetting problem.
In Appendix C we consider simpletabular RL experiments across 3 example classes of scalable MDPs.
Our analysis demonstrates thaton-policy and off-policy versions of Q-Learning that perform policy evaluation using an estimateof the steady-state distribution derived from matrix inversion of a tabular environment model, consistently outperform standard model-based and model-free baselines in terms of lifelong regret asenvironments are scaled up.
Even though estimating the steady-state distribution would clearly bequite challenging in complex state and observation spaces, recent approaches have made great stridestowards developing practical and scalable approaches leveraging data buffers [25]. Tracking Approaches:
It was originally noted by [26] that it could be a better strategy even instationary environments with temporal coherence to learn to track the best local solution insteadof attempting to converge to a globally optimal ﬁxed policy.
This indeed, may be a necessity inenvironments that experience extreme temporal coherence governed by high mixing times and couldprovide a formalization of the beneﬁt for approaches based on meta-learning in this context as notedby [26].
For example, [27] recently was successful in addressing the non-stationarity of multi-agent
RL by learning to converge to a recurrent set of joint policies rather than a single joint policy.
Looking Forward.
In this work, we have considered the implicit premise of the current continual
RL literature that results from currently manageable small-scale experiments are indicative ofperformance for the large-scale aspirational use cases of the future.
In particular, we have highlightedthat mixing times will scale signiﬁcantly as the problems we deal with are scaled and how traditionalapproaches to RL are ill-suited to deal with this.
We hope that our work will encourage the communityto carefully consider the theoretical implications of scaling environments, especially with regard tohow proposed continual RL algorithms scale with the mixing time.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]