Original: M INIMUM D ESCRIPTION L ENGTH C ONTROL
Revised: M INIMUM D ESCRIPTION L ENGTH C ONTROL

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel framework for multitask reinforcement learning based on the minimum description length (MDL) principle.
Revised: We propose a novel framework for multitask reinforcement learning based on the minimum description length (MDL) principle.

Original: In this approach, which we term MDL-control (MDL-C), the agent learns the common structure among the tasks with which it is faced and then distills it into a simpler representation which facilitates faster convergence and generalization to new tasks.
Revised: In this approach, which we term MDL-control (MDL-C), the agent learns the common structure among the tasks with which it is faced and then distills it into a simpler representation which facilitates faster convergence and generalization to new tasks.

Original: In doing so, MDLC naturally balances adaptation to each task with epistemic uncertainty about the task distribution.
Revised: In doing so, MDLC naturally balances adaptation to each task with epistemic uncertainty about the task distribution.

Original: We motivate MDL-C via formal connections between the
Revised: We motivate MDL-C via formal connections between the

Original: MDL principle and Bayesian inference, derive theoretical performance guarantees, and demonstrate MDL-C’s empirical effectiveness on both discrete and highdimensional continuous control tasks.
Revised: MDL principle and Bayesian inference, derive theoretical performance guarantees, and demonstrate MDL-C’s empirical effectiveness on both discrete and highdimensional continuous control tasks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In order to learn efﬁciently in a complex world with multiple rapidly changing objectives, both animals and machines must leverage past experience.
Revised: In order to learn efficiently in a complex world with multiple rapidly changing objectives, both animals and machines must leverage past experience.

Original: This is a challenging task, as processing and storing all relevant information is computationally infeasible.
Revised: This is a challenging task, as processing and storing all relevant information is computationally infeasible.

Original: How can an intelligent agent address this problem?
Revised: How can an intelligent agent address this problem?

Original: We hypothesize that one route may lie in the dual process theory of cognition, a longstanding framework in cognitive psychology introduced by William James (James, 1890) which lies at the heart of many dichotomies in both cognitive science and machine learning.
Revised: We hypothesize that one route may lie in the dual process theory of cognition, a longstanding framework in cognitive psychology introduced by William James (James, 1890) which lies at the heart of many dichotomies in both cognitive science and machine learning.

Original: Examples include goal-directed versus habitual behavior (Graybiel, 2008), model-based versus model-free reinforcement learning (Daw et al., 2011; Sutton and Barto, 2018), and “System 1” versus “System 2” thinking (Kahneman, 2011).
Revised: Examples include goal-directed versus habitual behavior (Graybiel, 2008), model-based versus model-free reinforcement learning (Daw et al., 2011; Sutton and Barto, 2018), and “System 1” versus “System 2” thinking (Kahneman, 2011).

Original: In each of these paradigms, a complex, “control” process trades off with a simple, “default” process to guide actions.
Revised: In each of these paradigms, a complex, “control” process trades off with a simple, “default” process to guide actions.

Original: Why has this been such a successful and enduring conceptual motif?
Revised: Why has this been such a successful and enduring conceptual motif?

Original: Our hypothesis is that default processes often serve to distill common structure from the tasks consistently faced by animals and agents, facilitating generalization and rapid learning on new objectives.
Revised: Our hypothesis is that default processes often serve to distill common structure from the tasks consistently faced by animals and agents, facilitating generalization and rapid learning on new objectives.

Original: For example, drivers can automatically traverse commonly traveled roads en route to new destinations, and chefs quickly learn new dishes on the back of well-honed fundamental techniques.
Revised: For example, drivers can automatically traverse commonly traveled roads en route to new destinations, and chefs quickly learn new dishes on the back of well-honed fundamental techniques.

Original: Importantly, even intricate tasks can become automatic, if repeated often enough (e.g., the combination of ﬁne motor commands required to swing a tennis racket): the default process must be sufﬁciently expressive to learn common behaviors, regardless of their complexity.
Revised: Importantly, even intricate tasks can become automatic, if repeated often enough (e.g., the combination of fine motor commands required to swing a tennis racket): the default process must be sufficiently expressive to learn common behaviors, regardless of their complexity.

Original: In reality, most processes likely lie on a continuum between simplicity and complexity.
Revised: In reality, most processes likely lie on a continuum between simplicity and complexity.

Original: In reinforcement learning (RL; Sutton and Barto, 2018), improving sample efﬁciency on new tasks is crucial to the developement of general agents which can learn effectively in the real world (Botvinick et al., 2015; Kirk et al., 2021).
Revised: In reinforcement learning (RL; Sutton and Barto, 2018), improving sample efficiency on new tasks is crucial to the developement of general agents which can learn effectively in the real world (Botvinick et al., 2015; Kirk et al., 2021).

Original: Intriguingly, one family of approaches which have shown promise in this regard are regularized policy optimization algorithms, in which a goal-speciﬁc control policy is paired with a simple yet general default policy to facilitate learning across multiple tasks (Teh et al., 2017; Galashov et al., 2019; Goyal et al., 2020; 2019; Moskovitz et al., 2022a).
Revised: Intriguingly, one family of approaches which have shown promise in this regard are regularized policy optimization algorithms, in which a goal-specific control policy is paired with a simple yet general default policy to facilitate learning across multiple tasks (Teh et al., 2017; Galashov et al., 2019; Goyal et al., 2020; 2019; Moskovitz et al., 2022a).

Original: One difﬁculty in algorithm design, however, is how much or how little to constrain the default policy, and in what way.
Revised: One difficulty in algorithm design, however, is how much or how little to constrain the default policy, and in what way.

Original: An overly simple default policy will fail to identify and exploit commonalities among tasks, while a complex model may overﬁt to a single task and fail to generalize.
Revised: An overly simple default policy will fail to identify and exploit commonalities among tasks, while a complex model may overfit to a single task and fail to generalize.

Original: Most approaches manually specify an asymmetry between the control and default policies, such as hiding input information (Galashov et al., 2019) or constraining the model class (Lai and Gershman, 2021).
Revised: Most approaches manually specify an asymmetry between the control and default policies, such as hiding input information (Galashov et al., 2019) or constraining the model class (Lai and Gershman, 2021).

Original: Ideally, we’d like an adaptive approach that learns the appropriate degree of complexity via experience.
Revised: Ideally, we’d like an adaptive approach that learns the appropriate degree of complexity via experience.

Original: The minimum description length principle (MDL; Rissanen, 1978), which in general holds that one should prefer the simplest model that accurately ﬁts the data, offers a guiding framework for algorithm design that does just that, enabling the default policy to optimally trade off between adapting to information from new tasks and maintaining simplicity.
Revised: The minimum description length principle (MDL; Rissanen, 1978), which in general holds that one should prefer the simplest model that accurately fits the data, offers a guiding framework for algorithm design that does just that, enabling the default policy to optimally trade off between adapting to information from new tasks and maintaining simplicity.

Original: Inspired by dual process theory and the
Revised: Inspired by dual process theory and the

Original: MDL principle, we propose MDL-control (MDL-C, pronounced “middle-cee”), a principled RPO framework for multitask RL.
Revised: MDL principle, we propose MDL-control (MDL-C, pronounced “middle-cee”), a principled RPO framework for multitask RL.

Original: In Section 2, we formally introduce multitask RL and describe RPO approaches within this setting.
Revised: In Section 2, we formally introduce multitask RL and describe RPO approaches within this setting.

Original: In Section 3, we describe MDL and the variational coding framework, from which we extract MDL-C and derive its formal performance characteristics.
Revised: In Section 3, we describe MDL and the variational coding framework, from which we extract MDL-C and derive its formal performance characteristics.

Original: In Section 5, we demonstrate its empirical effectiveness in both discrete and continuous control settings.
Revised: In Section 5, we demonstrate its empirical effectiveness in both discrete and continuous control settings.

Original: Finally, we discuss related ideas from the the literature (Section 6) and conclude (Section 7).
Revised: Finally, we discuss related ideas from the the literature (Section 6) and conclude (Section 7).

Original:  R EINFORCEMENT L EARNING P RELIMINARIES
Revised:  R EINFORCEMENT L EARNING P RELIMINARIES

Original: The single-task setting We model a task as a Markov decision process (MDP; Puterman, 2010)
Revised: The single-task setting We model a task as a Markov decision process (MDP; Puterman, 2010)

Original: M = ( S , A , P , r, γ, ρ ) , where S , A are state and action spaces, respectively, P : S ×
Revised: M = ( S , A , P , r, γ, ρ ) , where S , A are state and action spaces, respectively, P : S ×

Original: A → P ( S ) is the state transition distribution, r : S ×
Revised: A → P ( S ) is the state transition distribution, r : S ×

Original: A → [0 , 1] is a reward function, γ ∈ [0 , 1) is a discount factor, and ρ ∈ P ( S ) is the starting state distribution.
Revised: A → [0 , 1] is a reward function, γ ∈ [0 , 1) is a discount factor, and ρ ∈ P ( S ) is the starting state distribution.

Original: P ( · ) is the space of probability distributions deﬁned over a given space.
Revised: P ( · ) is the space of probability distributions defined over a given space.

Original: The agent takes actions using a policy π : S → P ( A ) .
Revised: The agent takes actions using a policy π : S → P ( A ) .

Original: In large or continuous domains, the policy is often parameterized: π → π θ , θ ∈ Θ , where Θ ⊆ R d represents a particular model class with d parameters.
Revised: In large or continuous domains, the policy is often parameterized: π → π θ , θ ∈ Θ , where Θ ⊆ R d represents a particular model class with d parameters.

Original: In conjunction with the transition dynamics, the policy induces a distribution over trajectories τ
Revised: In conjunction with the transition dynamics, the policy induces a distribution over trajectories τ

Original: = ( s h , a h ) ∞ h =0 , P π θ ( τ ) .
Revised: = ( s h , a h ) ∞ h =0 , P π θ ( τ ) .

Original: In a single task, the agent seeks to maximize its value V π θ =
Revised: In a single task, the agent seeks to maximize its value V π θ =

Original: E τ ∼ P
Revised: E τ ∼ P

Original: πθ R ( τ ) , where R ( τ ) :
Revised: πθ R ( τ ) , where R ( τ ) :

Original: = (cid:80) h ≥ 

Revised: = (cid:80) h ≥

Original: γ h r ( s h , a h ) is called the return .
Revised: γ h r ( s h , a h ) is called the return .

Original: We denote by d πρ the state-occupancy distribution induced by policy π with starting state distribution ρ : d πρ ( s ) =
Revised: We denote by d πρ the state-occupancy distribution induced by policy π with starting state distribution ρ : d πρ ( s ) =

Original: E ρ (1 − γ ) (cid:80) h ≥ 

Revised: E ρ (1 − γ ) (cid:80) h ≥

Original: γ h Pr( s h
Revised: γ h Pr( s h

Original: = s | s 0 ) .
Revised: = s | s 0 ) .

Original: Multiple tasks There are a number of frameworks for multitask RL in the literature (Yu et al., 2019; Zahavy et al., 2021;
Revised: Multiple tasks There are a number of frameworks for multitask RL in the literature (Yu et al., 2019; Zahavy et al., 2021;

Original: Finn et al., 2017; Brunskill and Li, 2013).
Revised: Finn et al., 2017; Brunskill and Li, 2013).

Original: For a more detailed discussion, see Appendix Section B.
Revised: For a more detailed discussion, see Appendix Section B.

Original: In this paper, we focus primarily on what we term the sequential and parallel task settings.
Revised: In this paper, we focus primarily on what we term the sequential and parallel task settings.

Original: The objective in each case is to maximize average reward across tasks , equivalent to minimizing cumulative regret over the agent’s ‘lifetime.’
Revised: The objective in each case is to maximize average reward across tasks , equivalent to minimizing cumulative regret over the agent’s ‘lifetime.’

Original: More speciﬁcally, we assume a (possibly inﬁnite) set of tasks (MDPs)
Revised: More specifically, we assume a (possibly infinite) set of tasks (MDPs)

Original: M = { M } presented to the agent by sampling from some task distribution P M ∈ P ( M ) .
Revised: M = { M } presented to the agent by sampling from some task distribution P M ∈ P ( M ) .

Original: In the sequential task setting (Moskovitz et al., 2022a; Pacchiano et al., 2022), tasks (MDPs) are sampled one at a time from P M , with the agent training on each until convergence.
Revised: In the sequential task setting (Moskovitz et al., 2022a; Pacchiano et al., 2022), tasks (MDPs) are sampled one at a time from P M , with the agent training on each until convergence.

Original: In the parallel task training (Yu et al., 2019), a new MDP is sampled from P M at the start of every episode and is associated with a particular input feature g ∈ G that indicates to the agent which task has been sampled.
Revised: In the parallel task training (Yu et al., 2019), a new MDP is sampled from P M at the start of every episode and is associated with a particular input feature g ∈ G that indicates to the agent which task has been sampled.

Original: Regularized Policy Optimization One common approach which improves performance is regularized policy optimization (RPO; Schulman et al., 2017; 2018; Levine, 2018; Agarwal et al., 2020;
Revised: Regularized Policy Optimization One common approach which improves performance is regularized policy optimization (RPO; Schulman et al., 2017; 2018; Levine, 2018; Agarwal et al., 2020;

Original: Pacchiano et al., 2020; Tirumala et al., 2020; Abdolmaleki et al., 2018).
Revised: Pacchiano et al., 2020; Tirumala et al., 2020; Abdolmaleki et al., 2018).

Original: In RPO, a convex regularization term Ω( θ ) is added to the objective: J RPO λ ( θ ) =
Revised: In RPO, a convex regularization term Ω( θ ) is added to the objective: J RPO λ ( θ ) =

Original: V π θ − λ Ω( θ ) .
Revised: V π θ − λ Ω( θ ) .

Original: In the single-task setting, the regularization term is often used to approximate trust region (Schulman et al., 2015), proximal point (Schulman et al., 2017), or natural gradient (Kakade, 2002; Pacchiano et al., 2020; Moskovitz et al., 2020) optimization, or to prevent premature convergence to local maxima (Haarnoja et al., 2018; Lee et al., 2018).
Revised: In the single-task setting, the regularization term is often used to approximate trust region (Schulman et al., 2015), proximal point (Schulman et al., 2017), or natural gradient (Kakade, 2002; Pacchiano et al., 2020; Moskovitz et al., 2020) optimization, or to prevent premature convergence to local maxima (Haarnoja et al., 2018; Lee et al., 2018).

Original: In multitask settings, the regularization term for RPO typically takes the form of a divergence measure penalizing the policy responsible for taking actions π θ , which we’ll refer to as the control policy , for deviating from some default policy π w , which is intended to encode generally useful behavior for some family of tasks (Teh et al., 2017; Galashov et al., 2019; Goyal et al., 2019; 2020; Moskovitz et al., 2022a).
Revised: In multitask settings, the regularization term for RPO typically takes the form of a divergence measure penalizing the policy responsible for taking actions π θ , which we’ll refer to as the control policy , for deviating from some default policy π w , which is intended to encode generally useful behavior for some family of tasks (Teh et al., 2017; Galashov et al., 2019; Goyal et al., 2019; 2020; Moskovitz et al., 2022a).

Original: By capturing behavior which is on average useful across tasks, π w can provide a form of beneﬁcial supervision to π θ when obtaining reward is challenging, either because π θ has been insufﬁciently trained or rewards are sparse.
Revised: By capturing behavior which is on average useful across tasks, π w can provide a form of beneficial supervision to π θ when obtaining reward is challenging, either because π θ has been insufficiently trained or rewards are sparse.

Original:  T HE M INIMUM D ESCRIPTION L ENGTH P RINCIPLE
Revised:  T HE M INIMUM D ESCRIPTION L ENGTH P RINCIPLE

Original: General principle Storing all environment interactions across multiple tasks is computationally infeasible, so multitask RPO algorithms offer a compressed representation in the form of a default policy.
Revised: General principle Storing all environment interactions across multiple tasks is computationally infeasible, so multitask RPO algorithms offer a compressed representation in the form of a default policy.

Original: However, the type of information which is compressed (and that which is lost) is often hard-coded a priori .
Revised: However, the type of information which is compressed (and that which is lost) is often hard-coded a priori .

Original: Preferably, we’d like an approach which can distill structural regularities among tasks without needing to know what they are beforehand.
Revised: Preferably, we’d like an approach which can distill structural regularities among tasks without needing to know what they are beforehand.

Original: The minimum description length (MDL) framework offers a principled approach to this problem.
Revised: The minimum description length (MDL) framework offers a principled approach to this problem.

Original: So-called “ideal” MDL seeks to ﬁnd the shortest solution written in a general-purpose programming language 1 which accurately reproduces the data—an idea rooted in the concept of Kolmogorov complexity (Li and Vitnyi, 2008).
Revised: So-called “ideal” MDL seeks to find the shortest solution written in a general-purpose programming language 1 which accurately reproduces the data—an idea rooted in the concept of Kolmogorov complexity (Li and Vitnyi, 2008).

Original: Given the known impossibility of computing Kolmogorov complexity for all but the simplest cases, a more practical MDL approach instead prescribes selecting the hypothesis H (cid:63) from some hypothesis class
Revised: Given the known impossibility of computing Kolmogorov complexity for all but the simplest cases, a more practical MDL approach instead prescribes selecting the hypothesis H ⋆ from some hypothesis class

Original: H which minimizes the two-part code H (cid:63) = argmin H ∈H L ( D| H )
Revised: H which minimizes the two-part code H ⋆ = argmin H ∈H L ( D| H )

Original: + L ( H ) , where L ( D| H ) is the number of bits required to encode the data given the hypothesis and L ( H ) is the number of bits needed to encode the hypothesis itself.
Revised: + L ( H ) , where L ( D| H ) is the number of bits required to encode the data given the hypothesis and L ( H ) is the number of bits needed to encode the hypothesis itself.

Original: There are a variety of so-called universal coding schemes which can be used to model these quantities.
Revised: There are a variety of so-called universal coding schemes which can be used to model these quantities.

Original: Variational code One popular encoding scheme is the variational code (Blier and Ollivier, 2018;
Revised: Variational code One popular encoding scheme is the variational code (Blier and Ollivier, 2018;

Original: Hinton and Van Camp, 1993; Honkela and Valpola, 2004):
Revised: Hinton and Van Camp, 1993; Honkela and Valpola, 2004):

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where the hypothesis class is of a set of parametric models
Revised: where the hypothesis class is of a set of parametric models

Original: H =
Revised: H =

Original: { p θ ( D ) : θ ∈ Θ } .
Revised: { p θ ( D ) : θ ∈ Θ } .

Original: The model parameters are random variables with prior distribution p ( θ ) and ν ( θ ) is any distribution over Θ .
Revised: The model parameters are random variables with prior distribution p ( θ ) and ν ( θ ) is any distribution over Θ .

Original: Minimizing L var ν ( D ) with respect to ν is equivalent to performing variational inference, maximizing a lower-bound to the data log-likelihood log p ( D )
Revised: Minimizing L var ν ( D ) with respect to ν is equivalent to performing variational inference, maximizing a lower-bound to the data log-likelihood log p ( D )

Original: =
Revised: =

Original: log (cid:82) p
Revised: log (cid:82) p

Original: ( θ ) p θ ( D )
Revised: ( θ ) p θ ( D )

Original: dθ ≥ − L var ν ( D ) .
Revised: dθ ≥ − L var ν ( D ) .

Original: Roughly speaking, MDL encourages the choice of “simple” models when limited data are available (Grunwald, 2004).
Revised: Roughly speaking, MDL encourages the choice of “simple” models when limited data are available (Grunwald, 2004).

Original: In the variational coding scheme, simplicity is enforced via the choice of prior.
Revised: In the variational coding scheme, simplicity is enforced via the choice of prior.

Original: Sparsity-inducing priors and variational dropout Sparsity-inducing priors can be used to improve the compression rate within the variational coding scheme, as they encourage the model to prune out parameters that do not contribute to reducing L var ( D| θ ) .
Revised: Sparsity-inducing priors and variational dropout Sparsity-inducing priors can be used to improve the compression rate within the variational coding scheme, as they encourage the model to prune out parameters that do not contribute to reducing L var ( D| θ ) .

Original: Many sparsity-inducing priors belong to the family of scale mixtures of normal distributions (Andrews and Mallows, 1974): z ∼ p ( z ) , θ ∼ p ( θ | z ) =
Revised: Many sparsity-inducing priors belong to the family of scale mixtures of normal distributions (Andrews and Mallows, 1974): z ∼ p ( z ) , θ ∼ p ( θ | z ) =

Original: N ( w ; 0 , z 2 ) where p ( z ) deﬁnes a distribution over the variance z 2 .
Revised: N ( w ; 0 , z 2 ) where p ( z ) defines a distribution over the variance z 2 .

Original: Common choices of p ( z ) include the Jeffreys prior p ( z ) ∝ | z | − 1 (Jeffreys, 1946), the inverse-Gamma distribution, and the half-Cauchy distribution (Polson and Scott, 2012; Gelman, 2006).
Revised: Common choices of p ( z ) include the Jeffreys prior p ( z ) ∝ | z | − 1 (Jeffreys, 1946), the inverse-Gamma distribution, and the half-Cauchy distribution (Polson and Scott, 2012; Gelman, 2006).

Original: Such priors have deep connections to MDL theory.
Revised: Such priors have deep connections to MDL theory.

Original: For example, the Jeffreys prior in conjunction with an exponential family likelihood is asymptotically identical to the normalized maximum likelihood estimator, perhaps the most fundamental ‘MDL’ estimator (Gr ¨ unwald and Roos, 2019).
Revised: For example, the Jeffreys prior in conjunction with an exponential family likelihood is asymptotically identical to the normalized maximum likelihood estimator, perhaps the most fundamental ‘MDL’ estimator (Gr ¨ unwald and Roos, 2019).

Original: Variational dropout (VDO) is an effective algorithm for minimizing Equation (3.1) for these sparsityinducing priors (Louizos et al., 2017;
Revised: Variational dropout (VDO) is an effective algorithm for minimizing Equation (3.1) for these sparsity-inducing priors (Louizos et al., 2017;

Original: Kingma et al., 2015; Molchanov et al., 2017).
Revised: Kingma et al., 2015; Molchanov et al., 2017).

Original: Brieﬂy, this involves choosing an approximate posterior distribution with the form
Revised: Briefly, this involves choosing an approximate posterior distribution with the form

Original: [Equation]
Revised: [Equation]

Original: and optimizing Equation (3.1) via stochastic gradient descent on the variational parameters given by { α, µ z , σ 2 z , µ, σ 2 } .
Revised: and optimizing Equation (3.1) via stochastic gradient descent on the variational parameters given by { α, µ z , σ 2 z , µ, σ 2 } .

Original: As its name suggests—and importantly for its ease of application to large models—VDO can be implemented as a form of dropout (Srivastava et al., 2014) by reparameterizing the noise on the weights as activation noise (Kingma et al., 2015).
Revised: As its name suggests—and importantly for its ease of application to large models—VDO can be implemented as a form of dropout (Srivastava et al., 2014) by reparameterizing the noise on the weights as activation noise (Kingma et al., 2015).

Original: Application of VDO to Bayesian neural networks has achieved impressive compression rates, sparsifying deep neural networks while maintaining prediction performance on supervised learning problems (Molchanov et al., 2017;
Revised: Application of VDO to Bayesian neural networks has achieved impressive compression rates, sparsifying deep neural networks while maintaining prediction performance on supervised learning problems (Molchanov et al., 2017;

Original: Louizos et al., 2017).
Revised: Louizos et al., 2017).

Original: Equipped with a powerful approach for MDL-grounded posterior inference, we can now integrate these ideas with multitask RPO.
Revised: Equipped with a powerful approach for MDL-grounded posterior inference, we can now integrate these ideas with multitask RPO.

Original:  M INIMUM D ESCRIPTION L ENGTH C ONTROL
Revised:  M INIMUM D ESCRIPTION L ENGTH C ONTROL

Original: As part of its underlying philosophy, the MDL principle holds that 1) learning is the process of discovering regularity in data, and 2) any regularity in the data can be used to compress it (Grunwald, 2004).
Revised: As part of its underlying philosophy, the MDL principle holds that 1) learning is the process of discovering regularity in data, and 2) any regularity in the data can be used to compress it (Grunwald, 2004).

Original: Applying this perspective to RL is non-obvious—from the agent’s perspective, what ‘data’ is it trying to compress?
Revised: Applying this perspective to RL is non-obvious—from the agent’s perspective, what ‘data’ is it trying to compress?

Original: Our hypothesis, which forms the basis for the frameworkwe propose in this paper, is that an agent faced with a set of tasks in the world should seek to elucidate structural regularity from the environment interactions generated by the optimal policies for the tasks. This makes intuitive sense: the agent ought to compress information which indicates
Revised: Our hypothesis, which forms the basis for the framework

Original: 1: Require: task distribution P M , policy class Θ , non-increasing coefﬁcients { η k } Kk = 2: Initialize: default policy distribution ν 1 ∈ N ⊆ P (Θ) , default policy dataset D 0 ← ∅ 3: for tasks k = 1 , 2 , . . 
Revised: 1: Require: task distribution P M , policy class Θ , non-increasing coefficients { η k } Kk = 2: Initialize: default policy distribution ν 1 ∈ N ⊆ P (Θ) , default policy dataset D 0 ← ∅ 3: for tasks k = 1 , 2 , . . .

Original: , K do 4:
Revised: , K do 4:

Original: Sample a task M k = ( S , A , P k , r k , γ k , ρ k ) ∼ P M ( · ) 5:
Revised: Sample a task M k = ( S , A , P k [Equation] r [Equation]

Revised: γ

Revised: [Equation]

Original:  Optimize control policy:
Revised: ρ [Equation] ∼ P M ( · ) 5: Optimize control policy:

Original: [Equation]
Revised: [Equation]

Revised: ←

Original:  Add data to default policy replay ( M = |S| for ﬁnite/small state spaces):
Revised: [Equation] s ) [Equation] Add data to default policy replay ( M = |S| for finite/small state spaces):

Original: [Equation]
Revised: [Equation]

Original: Update default policy distribution: ν
Revised: Update default policy distribution: ν

Original: [Equation]
Revised: [Equation]

Original: ←
Revised: ←

Original: [Equation]
Revised: [Equation]

Original: [Equation] .4) 8: end for  how to correctly perform the tasks with which it is faced.
Revised: (4.4) 8: end for we propose in this paper, is that an agent faced with a set of tasks in the world should seek to elucidate structural regularity from the environment interactions generated by the optimal policies for the tasks. This makes intuitive sense: the agent ought to compress information which indicates how to correctly perform the tasks with which it is faced.

Original: That is, we propose that the data in multitask RL are the state-action interactions generated by the optimal policies for a set of tasks:
Revised: That is, we propose that the data in multitask RL are the state-action interactions generated by the optimal policies for a set of tasks:

Original: D =
Revised: D =

Original: {D M } M ∈M =
Revised: {D M } M ∈M =

Original: { ( s, a ) : ∀ s ∈ S , a ∼ π (cid:63)M ( ·| s ) } M ∈M
Revised: { ( s, a ) : ∀ s ∈ S , a ∼ π ⋆M ( ·| s ) } M ∈M

Original: This interpretation is in line with work suggesting that a useful operational deﬁnition of ‘task’ can be derived directly from the set of optimal (or near-optimal) policies it induces (Abel et al., 2021).
Revised: This interpretation is in line with work suggesting that a useful operational definition of ‘task’ can be derived directly from the set of optimal (or near-optimal) policies it induces (Abel et al., 2021).

Original: It also suggests a natural mapping to the multitask RPO framework.
Revised: It also suggests a natural mapping to the multitask RPO framework.

Original: In this view, the control policy is responsible for learning and the default policy for compression: by converging to the optimal policy for a given task, the control policy “discovers” regularity which is then distilled into a low-complexity representation by the default policy.
Revised: In this view, the control policy is responsible for learning and the default policy for compression: by converging to the optimal policy for a given task, the control policy “discovers” regularity which is then distilled into a low-complexity representation by the default policy.

Original: In our approach, the default policy is encouraged to learn a compressed representation not by artiﬁcially constraining the network architecture or via hand-designed information asymmetry, but rather through a prior distribution p ( w ) over its parameters which biases a variational posterior ν ( w ) towards simplicity.
Revised: In our approach, the default policy is encouraged to learn a compressed representation not by artificially constraining the network architecture or via hand-designed information asymmetry, but rather through a prior distribution p ( w ) over its parameters which biases a variational posterior ν ( w ) towards simplicity.

Original: The default policy is therefore trained to minimize the variational code:
Revised: The default policy is therefore trained to minimize the variational code:

Original: [Equation]
Revised: [Equation]

Original: where N is the distribution family for the posterior.
Revised: where N is the distribution family for the posterior.

Original: This suggests the approach presented in Algorithm 1, in which for each task M k , the control policy π θ is trained to approximate the optimal policy π (cid:63)k via RPO, and the result is compressed into a new default policy distribution ν k +1 .
Revised: This suggests the approach presented in Algorithm 1, in which for each task M k , the control policy π θ is trained to approximate the optimal policy π ⋆k via RPO, and the result is compressed into a new default policy distribution ν k +1 .

Original: We now further motivate sparsity-inducing priors for the default policy in multitask settings, derive formal performance guarantees for MDL-C, and demonstrate its empirical effectiveness.
Revised: We now further motivate sparsity-inducing priors for the default policy in multitask settings, derive formal performance guarantees for MDL-C, and demonstrate its empirical effectiveness.

Original: 4. M OTIVATING THE CHOICE OF SPARSITY - INDUCING PRIORS
Revised: 4. M OTIVATING THE CHOICE OF SPARSITY - INDUCING PRIORS

Original: In Section 3, compression (via pruning extraneous parameters) is the primary motivation for using sparsity-inducing priors that belong to the family of scaled-mixtures of normal distributions.
Revised: In Section 3, compression (via pruning extraneous parameters) is the primary motivation for using sparsity-inducing priors that belong to the family of scaled-mixtures of normal distributions.

Original: Intuitively, placing a distribution over the default parameters reﬂects the agent’s epistemic uncertainty about the task distribution—when few tasks have been sampled, a sparse prior prevents the default policy from overﬁtting to spurious correlations in the limited data that the agent has collected.
Revised: Intuitively, placing a distribution over the default parameters reflects the agent’s epistemic uncertainty about the task distribution—when few tasks have been sampled, a sparse prior prevents the default policy from overfitting to spurious correlations in the limited data that the agent has collected.

Original: Here, we make this motivation more precise, describing an example generative model of optimal policy parameters which provides a principled interpretation for prior choice p ( z ) in multitask RL.
Revised: Here, we make this motivation more precise, describing an example generative model of optimal policy parameters which provides a principled interpretation for prior choice p ( z ) in multitask RL.

Revised: [Figure]

Original: Generative model of optimal policy parameters Consider a set of tasks
Revised: Generative model of optimal policy parameters Consider a set of tasks

Original: M = { M ik } I,K i i =1 ,k = that are clustered into I groups, such that the MDPs in each group are more similar to one another than to members of other groups.
Revised: M = { M ik } I,K i i =1 ,k =1 that are clustered into I groups, such that the MDPs in each group are more similar to one another than to members of other groups.

Original: As an example, the overall family M could be all sports, while 
Revised: As an example, the overall family M could be all sports, while clusters

Original: [-wσ 2 /β σwˆ w(cid:107) w 1 − w 11 (cid:107) ≥ (cid:107) w 1 − ˆ w 1 (cid:107) (A) (B)      0 0.2 0.4 0.6 0.
 JeffreysInverse-Gamma Half-Cauchy p ( β ) β shrinks clusters-] M i ⊆
Revised: [-wσ 2 /β σwˆ w(cid:107) w 1 − w 11 (cid:107) ≥ (cid:107) w 1 − ˆ w 1 (cid:107) (A) (B)      0 0.2 0.4 0.6 0.
 JeffreysInverse-Gamma Half-Cauchy p ( β ) β shrinks clusters-] M i ⊆

Original: M could consist of, say, ball sports or endurance competitions.
Revised: M could consist of, say, ball sports or endurance competitions.

Original: To make this precise, we assume that the optimal policies of every MDP belong to a parametric family Π
Revised: To make this precise, we assume that the optimal policies of every MDP belong to a parametric family Π

Original: = { π w ( ·| s ) :
Revised: = { π w ( ·| s ) :

Original: w ∈ R d , ∀ s ∈ S} (e.g., softmax policies with parameters w ), and that the optimal policies for each group are randomly distributed within parameter space.
Revised: w ∈ R d , ∀ s ∈ S} (e.g., softmax policies with parameters w ), and that the optimal policies for each group are randomly distributed within parameter space.

Original: In particular, we assume that the parameters of the optimal policies of M have the following generative model:
Revised: In particular, we assume that the parameters of the optimal policies of M have the following generative model:

Original: [Equation]
Revised: [Equation]

Original: where I d is the d − dimensional identity matrix.
Revised: where I d is the d − dimensional identity matrix.

Original: If we marginalize out w i , we get the marginal distribution p ( w ik | β, σ 2 ) =
Revised: If we marginalize out w i , we get the marginal distribution p ( w ik | β, σ 2 ) =

Original: N ( w ik ; 0 , σ 2 β − 1 I d ) .
Revised: N ( w ik ; 0 , σ 2 β − 1 I d ) .

Original: We can then visualize the parameter distribution of the optimal policies for M as a d -dimensional Gaussian within which lie clusters of optimal policies for related tasks which are themselves normally distributed (see Fig.
Revised: We can then visualize the parameter distribution of the optimal policies for M as a d -dimensional Gaussian within which lie clusters of optimal policies for related tasks which are themselves normally distributed (see Fig.

Original: 4.1A for d = 2 ).
Revised: 4.1A for d = 2 ).

Original: Interpretation of β
Revised: Interpretation of β

Original: The parameter β ∈ (0 , 1] has thefollowing interpretation (see Figure 4.1A): β = squared distance between optimal policy parameters within a group  squared distance between optimal policies in M .
Revised: The parameter β ∈ (0 , 1] can be interpreted as encoding the squared distance between optimal policy parameters within a group divided by the squared distance between optimal policies in M .

Original: Intuitively, β determines how much information one gains about the optimal parameters of a task in a group, given knowledge about the optimal parameters of another task in the same group.
Revised: Intuitively, β determines how much information one gains about the optimal parameters of a task in a group, given knowledge about the optimal parameters of another task in the same group.

Original: To see this, we compute our posterior belief about the value w i given observation of w ik : p ( w i | w ik , β, σ 2 ) =
Revised: To see this, we compute our posterior belief about the value w i given observation of w ik : p ( w i | w ik , β, σ 2 ) =

Original: N (cid:0) w i ; (1 − β ) w ik , (1 − β ) σ 2 I d (cid:1) .
Revised: N (cid:0) w i ; (1 − β ) w ik , (1 − β ) σ 2 I d (cid:1) .

Original: When β = 1 (inner circle in Figure 4.1A has the same radius as the outer circle), our posterior mean estimate of w i is simply 0 , suggesting we have learned nothing new about the mean of the optimal parameters in group i , by observing w ik .
Revised: When β = 1 (inner circle in Figure 4.1A has the same radius as the outer circle), our posterior mean estimate of w i is simply 0 , suggesting we have learned nothing new about the mean of the optimal parameters in group i , by observing w ik .

Original: In the other extreme when β → 0 , the posterior mean approaches the maximum-likelihood estimator w ik , suggesting that observation of w ik provides maximal information about the optimal parameters in group i .
Revised: In the other extreme when β → 0 , the posterior mean approaches the maximum-likelihood estimator w ik , suggesting that observation of w ik provides maximal information about the optimal parameters in group i .

Original: Any β in between the two extremes results in an estimator that “shrinks” w ik towards 0 .
Revised: Any β in between the two extremes results in an estimator that “shrinks” w ik towards 0 .

Original: The value of β thus has important implications for multitask learning.
Revised: The value of β thus has important implications for multitask learning.

Original: Suppose an RL agent learns the optimal parameters w 11 (task 1, group 1), and proceeds to learn task 2 in group 1 .
Revised: Suppose an RL agent learns the optimal parameters w 11 (task 1, group 1), and proceeds to learn task 2 in group 1 .

Original: The value of β determines whether w 11 can be used to inform the agent’s learning of w 21 .
Revised: The value of β determines whether w 11 can be used to inform the agent’s learning of w 21 .

Original: In this way, β determines the effective degree of epistemic uncertainty the agent has about the task distribution.
Revised: In this way, β determines the effective degree of epistemic uncertainty the agent has about the task distribution.

Original: Choice of p ( β ) and connection to p ( z ) Given its importance, it’s natural to ask what value β should take.
Revised: Choice of p ( β ) and connection to p ( z ) Given its importance, it’s natural to ask what value β should take.

Original: Instead of treating β as a parameter, we can choose a prior p ( β ) and perform Bayesian inference.
Revised: Instead of treating β as a parameter, we can choose a prior p ( β ) and perform Bayesian inference.

Original: Ideally, p ( β ) should (i) encode our prior belief about the extent to which the optimal parameters cluster into groups and (ii) result in a posterior mean estimator ˆ w ( p ( β ))
Revised: Ideally, p ( β ) should (i) encode our prior belief about the extent to which the optimal parameters cluster into groups and (ii) result in a posterior mean estimator ˆ w ( p ( β ))

Original: ( x ) = 1 − E [ β | x ] x that is close to w for x | w ∼ N ( x ; w, σ 2 ) .
Revised: ( x ) = 1 − E [ β | x ] x that is close to w for x | w ∼ N ( x ; w, σ 2 ) .

Original: This condition encourages the expected default policy (under the posterior ν ; Equation (4.1)) to be close to optimal policies in the same MDP group (centered at w ).
Revised: This condition encourages the expected default policy (under the posterior ν ; Equation (4.1)) to be close to optimal policies in the same MDP group (centered at w ).

Original: One prior choice that satisﬁes both conditions is p ( β ) ∝ β − 1 .
Revised: One prior choice that satisfies both conditions is p ( β ) ∝ β − 1 .

Original: It places high probability for small β and low probability for high β , thus encoding the prior belief that the optimal task parameters are clustered (see Figure 4.1B; blue).
Revised: It places high probability for small β and low probability for high β , thus encoding the prior belief that the optimal task parameters are clustered (see Figure 4.1B; blue).

Original: It is instructive to compare p ( β ) ∝ β − 1 with two extreme choices of p ( β ) .
Revised: It is instructive to compare p ( β ) ∝ β − 1 with two extreme choices of p ( β ) .

Original: When p ( β ) = δ ( β − 1) , p ( z ) = δ ( σ ) and the marginal p ( w ) is the often-used Gaussian prior over the parameters w with ﬁxed variance σ 2 .
Revised: When p ( β ) = δ ( β − 1) , p ( z ) = δ ( σ ) and the marginal p ( w ) is the often-used Gaussian prior over the parameters w with fixed variance σ 2 .

Original: This corresponds to the prior belief that knowing w i 1 provides no information about w i 2 .
Revised: This corresponds to the prior belief that knowing w iprovides no information about w i 2 .

Original: On the other hand, p ( β ) = δ ( β ) recovers a uniform prior over the parameters w and reﬂects the prior belief that the MDP groups are inﬁnitely far apart.
Revised: On the other hand, p ( β ) = δ ( β ) recovers a uniform prior over the parameters w and reflects the prior belief that the MDP groups are infinitely far apart.

Original: In relation to (ii), one can show the ˆ w ( p ( β )) strictly dominates the maximum-likelihood estimator ˆ w ( ML ) ( x ) =
Revised: In relation to (ii), one can show the ˆ w ( p ( β )) strictly dominates the maximum-likelihood estimator ˆ w ( ML ) ( x ) =

Original: x (Efron and Morris, 1973; Section D), for p ( β ) ∝ β − 1 .
Revised: x (Efron and Morris, 1973; Section D), for p ( β ) ∝ β − 1 .

Original: This means MSE ( w, ˆ w ( p ( β )) )
Revised: This means MSE ( w, ˆ w ( p ( β )) )

Original: ≤ MSE ( w, ˆ w ( ML ) ) for all w , where MSE ( w, ˆ w ) =
Revised: ≤ MSE ( w, ˆ w ( ML ) ) for all w , where MSE ( w, ˆ w ) =

Original: E x ∼N ( x ; w,σ 2 ) (cid:107) w − ˆ w ( x ) (cid:107) 2 .
Revised: E x ∼N ( x ; w,σ 2 ) ∥ w − ˆ w ( x ) ∥ 2 .

Original: Connection to p ( z ) and application of VDO Deﬁning z2 = σ 2 β − 1 and applying the changeof-variable formula to p ( β ) ∝ β − 1 gives p ( z ) ∝ | z | − 1 and thus the Normal-Jeffreys prior in
Revised: Connection to p ( z ) and application of VDO Defining z = σ 2 β − 1 and applying the changeof-variable formula to p ( β ) ∝ β − 1 gives p ( z ) ∝ | z | − 1 and thus the Normal-Jeffreys prior in

Original: Section 3.
Revised: Section 3.

Original: VDO (see Section 3) can then be applied to obtain an approximate posterior ν ( w, z ) which minimizes the variational code Equation (4.1).
Revised: VDO (see Section 3) can then be applied to obtain an approximate posterior ν ( w, z ) which minimizes the variational code Equation (4.1).

Original: Similar correspondences may also be derived for the inverse-Gamma distribution and the half-Cauchy distribution (Figure 4.1B; Section D).
Revised: Similar correspondences may also be derived for the inverse-Gamma distribution and the half-Cauchy distribution (Figure 4.1B; Section D).

Original: 4. P ERFORMANCE A NALYSIS
Revised: 4. P ERFORMANCE A NALYSIS

Original: At a fundamental level, we’d like assurance (i) that MDL-C’s default policy will be able to effectively distill the optimal policies for previously observed tasks, and (ii) that regularization using this default policy gives strong performance guarantees for the control policy on future tasks.
Revised: At a fundamental level, we’d like assurance (i) that MDL-C’s default policy will be able to effectively distill the optimal policies for previously observed tasks, and (ii) that regularization using this default policy gives strong performance guarantees for the control policy on future tasks.

Original: Default policy performance One way we can verify (i) is to obtain an upper bound on the average
Revised: Default policy performance One way we can verify (i) is to obtain an upper bound on the average

Original: KL between default policies sampled from the default policy distribution and an optimal policy for a task sampled from the task distribution.
Revised: KL between default policies sampled from the default policy distribution and an optimal policy for a task sampled from the task distribution.

Original: This enables us to perform analysis using online convex optimization (OCO).
Revised: This enables us to perform analysis using online convex optimization (OCO).

Original: In OCO, the learner observes a series of convex loss functions (cid:96) k : N → R , k = 1 , . . .
Revised: In OCO, the learner observes a series of convex loss functions ℓ k : N → R , k = 1 , . . .

Original: , K , where N ⊆ R d is a convex set.
Revised: , K , where N ⊆ R d is a convex set.

Original: After each round, the learner produces an output x k ∈ N for which it will then incur a loss (cid:96) k ( x k ) (Orabona, 2019).
Revised: After each round, the learner produces an output x k ∈ N for which it will then incur a loss ℓ k ( x k ) (Orabona, 2019).

Original: At round k , the learner is usually assumed to have knowledge of (cid:96) 1 , . . . 
Revised: At round k , the learner is usually assumed to have knowledge of ℓ 1 , . . . ,

Original: , (cid:96) k − 1 , but no other assumptions are made about the sequence of loss functions.
Revised: ℓ k − 1 , but no other assumptions are made about the sequence of loss functions.

Original: The learner’s goal is to minimize its average regret.
Revised: The learner’s goal is to minimize its average regret.

Original: For further background, see
Revised: For further background, see

Original: Section F. Crucially, the MDL-C learning procedure for the default policy distribution is equivalent to follow the regularized leader (FTRL), an OCO algorithm which enjoys sublinear regret.
Revised: Section F. Crucially, the MDL-C learning procedure for the default policy distribution is equivalent to follow the regularized leader (FTRL), an OCO algorithm which enjoys sublinear regret.

Original: In each round of FTRL, the learner selects the solution x ∈ N according to the following general objective: x 
Revised: In each round of FTRL, the learner selects the solution x ∈ N according to the following general objective: x k +1 =

Original: [Equation] k 
Revised: argmin x ∈ N ψ k ( x ) + (cid:80)

Original: [Equation] i [Equation] ) , where ψ : N → R is a convex regularization function.
Revised: k − 1 i =1 ℓ i ( x ) , where ψ : N → R is a convex regularization function.

Original: Using standard results, this connection allows us to bound MDL-C’s regret in learning the default policy distribution.
Revised: Using standard results, this connection allows us to bound MDL-C’s regret in learning the default policy distribution.

Original: All proofs are provided in Section G.
Revised: All proofs are provided in Section G.

Original: Proposition 4.1 (Persistent Replay FTRL Regret; (Orabona, 2019), Corollary 7.9) .
Revised: Proposition 4.1 (Persistent Replay FTRL Regret) .

Original: Let tasks M k be independently drawn from P M at every round, and let them each be associated with a deterministic optimal policy
Revised: Let tasks M k be independently drawn from P M at every round, and let them each be associated with a deterministic optimal policy

Original: π (cid:63)k : S → A .
Revised: π ⋆k : S → A .

Original: We make the following mild assumptions: i) π w ( a (cid:63) | s ) ≥ (cid:15) > 0 ∀ s ∈ S , where a (cid:63) = 
Revised: We make the following mild assumptions: i) π w ( a ⋆ | s ) ≥ ϵ > 0 ∀ s ∈ S , where a ⋆ = π

Original: π (cid:63)k ( s ) and (cid:15) is a constant. ii) min ν KL
Revised: ⋆k ( s ) and ϵ is a constant. ii) min ν KL

Original: [ ν ( · ) , p ( · )]
Revised: [ ν ( · ) , p ( · )]

Original: = 0 asymptotically as Var[ ν ] → ∞ .
Revised: = 0 asymptotically as Var[ ν ] → ∞ .

Original: Then with η k − 1 = log(1 /(cid:15) ) √ k , Algorithm 1 guarantees
Revised: Then with η k − 1 = log(1 /ϵ ) √ k , Algorithm 1 guarantees

Original: [Equation]
Revised: [Equation]

Original: where ¯ ν K = argmin ν ∈ N [Table]
Revised: where ¯ ν K = argmin ν ∈ N (cid:80) Kk =1 ℓ k ( ν ) .

Original: Control policy performance Intuitively, this result shows that the average regret is upper-bounded by factors which depend on the divergence of the barycenter distribution from the prior and the “worstcase” prediction of the default policy.
Revised: Control policy performance Intuitively, this result shows that the average regret is upper-bounded by factors which depend on the divergence of the barycenter distribution from the prior and the “worstcase” prediction of the default policy.

Original: Importantly, the KL between the default policy distribution and the barycenter distribution goes to zero as K → ∞ .
Revised: Importantly, the KL between the default policy distribution and the barycenter distribution goes to zero as K → ∞ .

Original: We can also now be assured of point (ii) above, in that this result can be used to obtain a sample-complexity bound for the control policy.
Revised: We can also now be assured of point (ii) above, in that this result can be used to obtain a sample-complexity bound for the control policy.

Original: Speciﬁcally, we can use Proposition G.1 to place an upper-bound on the total variation distance between default policies sampled from ν and the KL between the maximum likelihood solution and a sparsity-inducing prior p .
Revised: Specifically, we can use Proposition G.1 to place an upper-bound on the total variation distance between default policies sampled from ν and the KL between the maximum likelihood solution and a sparsity-inducing prior p .

Original: This is useful, as it allows to translate low regret for the default policy into a sample complexity result for the control policy using Moskovitz et al.
Revised: This is useful, as it allows to translate low regret for the default policy into a sample complexity result for the control policy using Moskovitz et al.

Original: (2022a), Lemma 5.2.
Revised: (2022a), Lemma 5.2.

Original: Proposition 4.2 (Control Policy Sample Complexity) .
Revised: Proposition 4.2 (Control Policy Sample Complexity) .

Original: Under the setting described in Proposition G.1, denote by T k the number of iterations to reach (cid:15) -error for M k in the sense that min t ≤ T k { V π (cid:63)k −
Revised: Under the setting described in Proposition G.1, denote by T k the number of iterations to reach ϵ -error for M k in the sense that min t ≤ T k { V π ⋆k −

Original: V ( t ) } ≤ (cid:15). whenever t > T k .
Revised: V ( t ) } ≤ ϵ. whenever t > T k .

Original: Further, denote the upper-bound in Eq.
Revised: Further, denote the upper-bound in Eq.

Original: (G.1) by G ( K ) .
Revised: (G.1) by G ( K ) .

Original: In a ﬁnite
Revised: In a finite

Original: MDP, from any initial θ (0) , and following gradient ascent, E M k ∼P M
Revised: MDP, from any initial θ (0) , and following gradient ascent, E M k ∼P M

Original: [ T k ] satisﬁes:
Revised: [ T k ] satisfies:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: goa
Revised: goa

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: c on t i ngen
Revised: c on t i ngen

Original: [Figure]
Revised: [Figure]

Original: where α k ( s ) :
Revised: where α k ( s ) :

Original: = d TV ( π (cid:63)k ( ·| s ) , ˆ π 0 ( ·| s )) ≤ (cid:112) G ( K ) , κ α k A
Revised: = d TV ( π ⋆k ( ·| s ) , ˆ π 0 ( ·| s )) ≤ (cid:112) G ( K ) , κ α k A

Original: ( s )
Revised: ( s )

Original: = 2 |A| (1 − α ( s ))
Revised: = 2 |A| (1 − α ( s ))

Original: 2 |A| (1 − α ( s )) − 1 , and µ is a measure over S such that µ ( s ) > 0 ∀ s ∈ S .
Revised: 2 |A| (1 − α ( s )) − 1 , and µ is a measure over S such that µ ( s ) > 0 ∀ s ∈ S .

Revised: Intuitively, this means that when the average number of samples is sufficiently large, the control policy is guaranteed to have reached ε error.

Original: The core message is that as the agent is trained on more tasks, the default policy distribution regret, upper-bounded by G ( K ) , decreases asymptotically to zero, and as the default policy regret decreases, the control policy will learn more rapidly, as poly( G ( K )) .
Revised: Therefore, as the agent is trained on more tasks, the default policy distribution regret, upper-bounded by G ( K ) , decreases asymptotically to zero, and as the default policy regret decreases, the control policy will learn more rapidly, as poly( G ( K )) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We tested MDL-C applied to discrete and continuous control in both the sequential and parallel task settings.
Revised: We tested MDL-C applied to discrete and continuous control in both the sequential and parallel task settings.

Original: To quantify performance, in addition to measuring per-task reward, we also report the cumulative regret for each method in each experimental setting in Section I.1.
Revised: To quantify performance, in addition to measuring per-task reward, we also report the cumulative regret for each method in each experimental setting in Section I.1.

Original: 5. 2D N AVIGATION
Revised: 5. 2D N AVIGATION

Original: We ﬁrst test MDL-C in the classic F OUR R OOMS environment (Fig.
Revised: We first test MDL-C in the classic F OUR R OOMS environment (Fig.

Original: 5.1a, (Sutton et al., 1999)).
Revised: 5.1a, (Sutton et al., 1999)).

Original: The baselines in this case are PO entropy-regularized policy optimization (PO), regularized policy optimization with no constraint on the default policy (RPO), an agent with VDO applied to the control policy and no default policy (VDO-PO), and M ANUAL IA (Galashov et al., 2019) in which the goal feature is manually witheld from the default policy.
Revised: The baselines in this case are PO entropy-regularized policy optimization (PO), regularized policy optimization with no constraint on the default policy (RPO), an agent with VDO applied to the control policy and no default policy (VDO-PO), and M ANUAL IA (Galashov et al., 2019) in which the goal feature is manually witheld from the default policy.

Original: Details can be found in Section H.
Revised: Details can be found in Section H.

Original: Generalization Across Goals In the ﬁrst setting, we test MDL-C’s ability to facilitate rapid learning on previously unseen goals.
Revised: Generalization Across Goals In the first setting, we test MDL-C’s ability to facilitate rapid learning on previously unseen goals.

Original: In the ﬁrst phase of training, a single goal location is randomly sampled at the start of each episode, and may be placed anywhere in two of the four rooms in the environment (Fig.
Revised: In the first phase of training, a single goal location is randomly sampled at the start of each episode, and may be placed anywhere in two of the four rooms in the environment (Fig.

Original: 5.1a, top left).
Revised: 5.1a, top left).

Original: In the second phase, the goal location is again randomly sampled at the start of each episode, but in this case, only in the rooms which were held out in the ﬁrst phase.
Revised: In the second phase, the goal location is again randomly sampled at the start of each episode, but in this case, only in the rooms which were held out in the first phase.

Original: Additionally, the agent is limited to 25 rather than 100 steps per episode.
Revised: Additionally, the agent is limited to 25 rather than 100 steps per episode.

Original: Importantly, VDO induces the MDL-C default policy to ignore input features which are, on average, less predictive of the control policy’s behavior.
Revised: Importantly, VDO induces the MDL-C default policy to ignore input features which are, on average, less predictive of the control policy’s behavior.

Original: In this case, the default policy learns to ignore the goal feature and the reward obtained on the previous timestep.
Revised: In this case, the default policy learns to ignore the goal feature and the reward obtained on the previous timestep.

Original: This is because, when averaging across goal locations, the agent’s current position ( s h ) and its previous direction ( a h − 1 ) are more informative of its next action—typically, heading towards the nearest door.
Revised: This is because, when averaging across goal locations, the agent’s current position ( s h ) and its previous direction ( a h − 1 ) are more informative of its next action—typically, heading towards the nearest door.

Original: In contrast, the un-regularized default policy of the RPO agent does not drop these features (Section I for a visualization and Section H for more details).
Revised: In contrast, the un-regularized default policy of the RPO agent does not drop these features (Section I for a visualization and Section H for more details).

Original: By learning to ignore the goals present in phase 1 and encoding useful behavior regardless of goal location, MDL-C’s develops more effective regularization in phase 2, enabling it to adapt more quickly than other methods (Fig.
Revised: By learning to ignore the goals present in phase 1 and encoding useful behavior regardless of goal location, MDL-C’s develops more effective regularization in phase 2, enabling it to adapt more quickly than other methods (Fig.

Original: 5.1c, top), particularly RPO, which overﬁts to phase 1’s goals.
Revised: 5.1c, top), particularly RPO, which overfits to phase 1’s goals.

Original: M ANUAL IA also adapts quickly, as its default policy is hard-coded to ignore the goal feature.
Revised: M ANUAL IA also adapts quickly, as its default policy is hard-coded to ignore the goal feature.

Original: [Figure]
Revised: [Figure]

Original: Robustness to Rule Changes In this setting, there are only two possible goal locations, one at the top left of the environment, and the other at the bottom right.
Revised: Robustness to Rule Changes In this setting, there are only two possible goal locations, one at the top left of the environment, and the other at the bottom right.

Original: In training phase 1, the agent receives a goal feature as input which indicates the state index of the rewarded location for that episode.
Revised: In training phase 1, the agent receives a goal feature as input which indicates the state index of the rewarded location for that episode.

Original: In phase 2, the goal feature switches from marking the reward location to marking the unrewarded location.
Revised: In phase 2, the goal feature switches from marking the reward location to marking the unrewarded location.

Original: That is, if the reward is in the top left, the goal feature will point to the bottom right.
Revised: That is, if the reward is in the top left, the goal feature will point to the bottom right.

Original: Here, the danger for the agent isn’t overﬁtting to a particular goal or goals, but rather “overﬁtting” to the reward-based rules associated with a given feature.
Revised: Here, the danger for the agent isn’t overfitting to a particular goal or goals, but rather “overfitting” to the reward-based rules associated with a given feature.

Original: As we saw in Fig.
Revised: As we saw in Fig.

Original: 5.1c (top), an un-regularized default policy, will copy the control policy and overﬁt to a particular setting.
Revised: 5.1c (top), an un-regularized default policy, will copy the control policy and overfit to a particular setting.

Original: Fortunately, the MDL-C default policy learns to ignore features which are, on average, less useful for predicting the control policy’s behavior—the goal and previous reward features.
Revised: Fortunately, the MDL-C default policy learns to ignore features which are, on average, less useful for predicting the control policy’s behavior—the goal and previous reward features.

Original: This renders the agent more robust to contingency switches like the one described, as we can see in Fig.
Revised: This renders the agent more robust to contingency switches like the one described, as we can see in Fig.

Original: 5.1c (bottom).
Revised: 5.1c (bottom).

Original: These examples illustrate that MDL-C enables agents to effectively learn the consistent structure of a group of tasks, regardless of its semantics, and “compress out” information which is less informative on average.
Revised: These examples illustrate that MDL-C enables agents to effectively learn the consistent structure of a group of tasks, regardless of its semantics, and “compress out” information which is less informative on average.

Original: 5. C ONTINUOUS C ONTROL
Revised: 5. C ONTINUOUS C ONTROL

Original: A more challenging application area is that of high-dimensional continuous control.
Revised: A more challenging application area is that of high-dimensional continuous control.

Original: In this setting, we presented agents with multitask learning problems using environments from the DeepMind Control
Revised: In this setting, we presented agents with multitask learning problems using environments from the DeepMind Control

Original: Suite (DMC; (Tassa et al., 2018)).
Revised: Suite (DMC; (Tassa et al., 2018)).

Original: We used soft actor critic (SAC; (Haarnoja et al., 2018)) as the base agent.
Revised: We used soft actor critic (SAC; (Haarnoja et al., 2018)) as the base agent.

Original: We tested MDL-C in both the sequential and parallel settings on two domains from DMC: walker and cartpole (Fig. 5.2a).
Revised: We tested MDL-C in both the sequential and parallel settings on two domains from DMC: walker and cartpole (Fig. 5.2a).

Original: Additional details can be found in Section H.
Revised: Additional details can be found in Section H.

Original: Sequential Tasks
Revised: Sequential Tasks

Original: In the sequential setting, tasks are sampled one at a time uniformly without replacement from the available tasks within each domain, with the default policy distribution conserved across tasks.
Revised: In the sequential setting, tasks are sampled one at a time uniformly without replacement from the available tasks within each domain, with the default policy distribution conserved across tasks.

Original: For walker, these tasks are stand , walk , and run .
Revised: For walker, these tasks are stand , walk , and run .

Original: In stand , the agent is rewarded for increasing the height of its center of mass, and in the latter two tasks, an additional reward is given for forward velocity.
Revised: In stand , the agent is rewarded for increasing the height of its center of mass, and in the latter two tasks, an additional reward is given for forward velocity.

Original: For cartpole, there are four tasks: balance , balance-sparse , swingup , and swingup-sparse .
Revised: For cartpole, there are four tasks: balance , balance-sparse , swingup , and swingup-sparse .

Original: In the balance tasks, the agent must keep a rotating pole upright, and in the swingup tasks, it must additionally learn to swing the pole upwards from an initial downward orientation.
Revised: In the balance tasks, the agent must keep a rotating pole upright, and in the swingup tasks, it must additionally learn to swing the pole upwards from an initial downward orientation.

Original: Performance results for the hardest task within each domain ( run in walker and swingup-sparse in cartpole) for each method are plotted in Fig.
Revised: Performance results for the hardest task within each domain ( run in walker and swingup-sparse in cartpole) for each method are plotted in Fig.

Original: 5.2b, where k indicates the task round at which the task was sampled.
Revised: 5.2b, where k indicates the task round at which the task was sampled.

Original: We can see that as k increases (as more tasks have been seen previously), MDL-C’s performance improves.
Revised: We can see that as k increases (as more tasks have been seen previously), MDL-C’s performance improves.

Original: Importantly, the RPO agent’s default policy, which is un-regularized, overﬁts to the previous task, essentially copying the optimal policy’s behavior.
Revised: Importantly, the RPO agent’s default policy, which is un-regularized, overfits to the previous task, essentially copying the optimal policy’s behavior.

Original: This can severely hinder the agent’s performance when the subsequent task requires different behavior.
Revised: This can severely hinder the agent’s performance when the subsequent task requires different behavior.

Original: For example, on swingup-sparse , if the previous task is swingup , the RPO agent performs well, as the goal is identical.
Revised: For example, on swingup-sparse , if the previous task is swingup , the RPO agent performs well, as the goal is identical.

Original: However, if the previous task is balance or balance-sparse , the agent never learns to swing the pole upwards, signiﬁcantly reducing its average performance.
Revised: However, if the previous task is balance or balance-sparse , the agent never learns to swing the pole upwards, significantly reducing its average performance.

Original: Another important point to note is that because the agent is not given an explicit goal feature in this setting, methods like M ANUAL IA which rely on prior knowledge about the agent’s inputs cannot be applied.
Revised: Another important point to note is that because the agent is not given an explicit goal feature in this setting, methods like M ANUAL IA which rely on prior knowledge about the agent’s inputs cannot be applied.

Original: Parallel Tasks We also tested parallel-task versions of SAC, M ANUAL IA, and MDL-C based on the model of Yu et al.
Revised: Parallel Tasks We also tested parallel-task versions of SAC, M ANUAL IA, and MDL-C based on the model of Yu et al.

Original: In this framework, a task within each domain is randomly sampled at the start of each episodeand the agent learns a single control policy for all tasks.
Revised: In this framework, a task within each domain is randomly sampled at the start of each episodeand the agent learns a single control policy for all tasks.

Original: Performance is plotted in Fig.
Revised: Performance is plotted in Fig.

Original: 5.2c, where we can again see that MDL-C accelerates convergence relative to the baseline methods.
Revised: 5.2c, where we can again see that MDL-C accelerates convergence relative to the baseline methods.

Original: This marks a difference compared to the easier FourRooms environment, in which
Revised: This marks a difference compared to the easier FourRooms environment, in which

Original: MDL-C and M ANUAL IA performed roughly the same.
Revised: MDL-C and M ANUAL IA performed roughly the same.

Original: As before, one clue to the difference can be found in the input features that the MDL-C default policy chooses to ignore (Fig. 5.2d).
Revised: As before, one clue to the difference can be found in the input features that the MDL-C default policy chooses to ignore (Fig. 5.2d).

Original: For walker, inputs are 24-dimensional, with 14 features related to the joint orientations, 1 feature indicating the height of the agent’s center of mass, and 9 features indicating velocity components.
Revised: For walker, inputs are 24-dimensional, with 14 features related to the joint orientations, 1 feature indicating the height of the agent’s center of mass, and 9 features indicating velocity components.

Original: For cartpole, there are 5 input dimensions, with 3 pertaining to position and 2 to velocity.
Revised: For cartpole, there are 5 input dimensions, with 3 pertaining to position and 2 to velocity.

Original: In the walker domain, where the performance difference is greatest, the MDL-C agent not only ignores the added task ID feature, but also the several features related to velocity.
Revised: In the walker domain, where the performance difference is greatest, the MDL-C agent not only ignores the added task ID feature, but also the several features related to velocity.

Original: In contrast, in the cartpole domain, MDL-C only ignores the task ID feature, just as M ANUAL IA does, and the performance gap is smaller.
Revised: In contrast, in the cartpole domain, MDL-C only ignores the task ID feature, just as M ANUAL IA does, and the performance gap is smaller.

Original: This illustrates that MDL-C learns to compress out spurious information even in settings for which it is difﬁcult to identify a priori .
Revised: This illustrates that MDL-C learns to compress out spurious information even in settings for which it is difficult to identify a priori .

Original: In order to test the effect of the learned asymmetry on performance more directly, we implemented a variant of M ANUAL IA in which all of the features which MDL-C learned to ignore were manually hidden from the default policy (Fig.
Revised: In order to test the effect of the learned asymmetry on performance more directly, we implemented a variant of M ANUAL IA in which all of the features which MDL-C learned to ignore were manually hidden from the default policy (Fig.

Original: I.4).
Revised: I.4).

Original: Interestingly, while this method improved over standard M ANUAL IA, it didn’t completely close the gap with MDL-C, indicating there are downstream effects within the network beyond input processing which are important for the default policy’s effectiveness.
Revised: Interestingly, while this method improved over standard M ANUAL IA, it didn’t completely close the gap with MDL-C, indicating there are downstream effects within the network beyond input processing which are important for the default policy’s effectiveness.

Original: We hope to explore these effects in more detail in future work.
Revised: We hope to explore these effects in more detail in future work.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: MDL-C can be viewed as an extension of recent approaches to learning default policies (“behavioral priors”) from the optimal policies of related tasks (Teh et al., 2017; Tirumala et al., 2020).
Revised: MDL-C can be viewed as an extension of recent approaches to learning default policies (“behavioral priors”) from the optimal policies of related tasks (Teh et al., 2017; Tirumala et al., 2020).

Original: For a default policy to be useful for transfer learning, it is crucial to balance the ability of the default policy to “copy” the control policies with its expressiveness.
Revised: For a default policy to be useful for transfer learning, it is crucial to balance the ability of the default policy to “copy” the control policies with its expressiveness.

Original: If the default policy is too expressive, it is likely to overﬁt on past tasks and fail to generalize to unseen tasks.
Revised: If the default policy is too expressive, it is likely to overfit on past tasks and fail to generalize to unseen tasks.

Original: Whereas prior work primarily handcrafts structural constraints into the default policies to avoid overﬁtting (e.g., by hiding certain state information from the default policy; Galashov et al., 2019), MDL-C learns such a balance from data with sparsity-inducing priors via variational inference.
Revised: Whereas prior work primarily handcrafts structural constraints into the default policies to avoid overfitting (e.g., by hiding certain state information from the default policy; Galashov et al., 2019), MDL-C learns such a balance from data with sparsity-inducing priors via variational inference.

Original: MDL-C may also be derived from the RL-asinference framework (Levine, 2018; Section A).
Revised: MDL-C may also be derived from the RL-asinference framework (Levine, 2018; Section A).

Original: MDL-C thus has close connections with algorithms such as MPO (Abdolmaleki et al., 2018) and VIREL (Fellows et al., 2020), discussed in Section A.
Revised: MDL-C thus has close connections with algorithms such as MPO (Abdolmaleki et al., 2018) and VIREL (Fellows et al., 2020), discussed in Section A.

Original: As a general framework, MDL-C is also connected to the long and well-established literature on choosing appropriate Bayesian priors (Jeffreys, 1946; Bernardo, 2005; Casella, 1985), and more recent work that focuses on learning such priors for large-scale machine learning models (Nalisnick and Smyth, 2017; Nalisnick et al., 2021;
Revised: As a general framework, MDL-C is also connected to the long and well-established literature on choosing appropriate Bayesian priors (Jeffreys, 1946; Bernardo, 2005; Casella, 1985), and more recent work that focuses on learning such priors for large-scale machine learning models (Nalisnick and Smyth, 2017; Nalisnick et al., 2021;

Original: Atanov et al., 2018).
Revised: Atanov et al., 2018).

Original: For a further discussion of related work, particularly concerning the application of MDL to the RL setting, see Section C.
Revised: For a further discussion of related work, particularly concerning the application of MDL to the RL setting, see Section C.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Inspired by dual process theories and the MDL principle, we propose a regularized policy optimization framework for multitask RL which aims to learn a simple default policy encoding a low-complexity distillation of the optimal behavior for some family of tasks.
Revised: Inspired by dual process theories and the MDL principle, we propose a regularized policy optimization framework for multitask RL which aims to learn a simple default policy encoding a low-complexity distillation of the optimal behavior for some family of tasks.

Original: By encouraging the default policy to maintain a low effective description length, MDL-C ensures that it does not overﬁt to spurious correlations among the (approximately) optimal policies learned by the agent.
Revised: By encouraging the default policy to maintain a low effective description length, MDL-C ensures that it does not overfit to spurious correlations among the (approximately) optimal policies learned by the agent.

Original: We described MDLC’s formal properties and demonstrated its empirical effectiveness in discrete and continuous control tasks.
Revised: We described MDLC’s formal properties and demonstrated its empirical effectiveness in discrete and continuous control tasks.

Original: There are of course limitations of MDL-C, which we believe represent opportunities for future work (see Section E).
Revised: There are of course limitations of MDL-C, which we believe represent opportunities for future work (see Section E).

Original: Promising research directions include integrating MDL-C with multitask RL approaches which balance a larger set of policies (Barreto et al., 2020; Moskovitz et al., 2022b;
Revised: Promising research directions include integrating MDL-C with multitask RL approaches which balance a larger set of policies (Barreto et al., 2020; Moskovitz et al., 2022b;

Original: Thakoor et al., 2022) as well considering nonstationary environments (Parker-Holder et al., 2022).
Revised: Thakoor et al., 2022) as well considering nonstationary environments (Parker-Holder et al., 2022).

Original: We hope MDL-C inspires further advances in multitask RL.
Revised: We hope MDL-C inspires further advances in multitask RL.

Revised: Acknowledgements The authors would like to thank Kevin Miller, DJ Strouse, Marcel Binz, and

Revised: Alexander Galashov for useful discussions and suggested improvements to the manuscript.

Revised: Work funded by the Gatsby Charitable Foundation.

Original: Supplementary Information A R EINFORCEMENT L EARNING AS I NFERENCE
Revised: Supplementary Information A R EINFORCEMENT L EARNING AS I NFERENCE

Original: The control as inference framework (Levine, 2018) associates every time step h with a binary “optimality” random variable O h ∈ { 0 , 1 } that indicates whether a h is optimal at state s h
Revised: The control as inference framework (Levine, 2018) associates every time step h with a binary “optimality” random variable O h ∈ { 0 , 1 } that indicates whether a h is optimal at state s h

Original: ( O h =for optimal, and O h
Revised: ( O h =for optimal, and O h

Original: = 0 for not).
Revised: = 0 for not).

Original: The optimality variable has the conditional distribution P ( O h = 1 | s h , a h ) =
Revised: The optimality variable has the conditional distribution P ( O h = 1 | s h , a h ) =

Original: exp( r ( s h , a h )) , which scales exponentially with the reward received taking action a h in state s h .
Revised: exp( r ( s h , a h )) , which scales exponentially with the reward received taking action a h in state s h .

Original: Denote O H as the event that O s = 1 for s = 0 , . .
Revised: Denote O H as the event that O s = 1 for s = 0 , . .

Original: , H − 1 .
Revised: , H − 1 .

Original: Then the log-likelihood that a policy π w ( a | s ) is optimal over a horizon H is given by:
Revised: Then the log-likelihood that a policy π w ( a | s ) is optimal over a horizon H is given by:

Original: [Equation]
Revised: [Equation]

Original: By performing variational inference, we can lower-bound the log-likelihood with the ELBO:
Revised: By performing variational inference, we can lower-bound the log-likelihood with the ELBO:

Original: [Equation]
Revised: [Equation]

Original: where ν θ,φ ( τ, w ) =
Revised: where ν θ,ϕ ( τ, w ) =

Original: ν θ ( τ ) ν φ ( w ) is the variational posterior,
Revised: ν θ ( τ ) ν ϕ ( w ) is the variational posterior,

Original: [Equation]
Revised: [Equation]

Original: and { θ, φ } are the variational parameters.
Revised: and { θ, ϕ } are the variational parameters.

Original: We can maximize this objective iteratively by performing coordinate ascent on { θ, φ } : 
Revised: We can maximize this objective iteratively by performing coordinate ascent on { θ, ϕ } : [Equation] h ) − [Equation]

Revised: KL

Original: [Equation]
Revised: [Equation]

Revised: ] (cid:1)(cid:33)

Revised: [Equation]

Revised: (A.2) ϕ ←

Revised: [Equation]

Revised: ∇

Revised: [Equation]

Revised: (cid:32)

Original: a h ) − E ν θ ( w ) KL [ π θ ( a h | s h ) , π w ( a h | s h )] (cid:1)(cid:33) , (A.2) φ ← φ − η ∇ φ (cid:32)
Revised: E [Equation] [Equation] (cid:88)

Original:  Eν θ ( τ )
Revised: h [Equation] E

Original: H − 1 (cid:88) h =
Revised: [Equation]

Original: E ν θ ( w ) KL [ π θ ( a h | s h ) , π w ( a h | s h )]
Revised: [Equation] ( w ) [Equation]

Original: + KL [ ν φ ( w ) , p ( w )] (cid:33) (A.3) where η is a learning rate parameter.
Revised: p ( w )] (cid:33) (A.3) where η is a learning rate parameter.

Original: Note that Equation (A.3) is equivalent to Equation (4.1) and
Revised: Note that Equation (A.3) is equivalent to Equation (4.1) and

Original: Equation (G.8), and Equation (A.2) is equivalent to Equation (G.7) with the KL reversed.
Revised: Equation (G.8), and Equation (A.2) is equivalent to Equation (G.7) with the KL reversed.

Original: Connection to Maximum a Posteriori Policy Optimization (MPO)
Revised: Connection to Maximum a Posteriori Policy Optimization (MPO)

Original: MDL-C is closely related to MPO (Abdolmaleki et al., 2018), with three key differences.
Revised: MDL-C is closely related to MPO (Abdolmaleki et al., 2018), with three key differences.

Original: First, MDL-C performs variational inference on the parameters of the default policy with an approximate posterior ν φ ( w ) , whereas MPO performs MAP inference.
Revised: First, MDL-C performs variational inference on the parameters of the default policy with an approximate posterior ν ϕ ( w ) , whereas MPO performs MAP inference.

Original: Second, MPO places a normal prior on w , which in effect penalizes the
Revised: Second, MPO places a normal prior on w , which in effect penalizes the

Original: L2 norm of w .
Revised: L2 norm of w .

Original: In contrast, MDL-C uses sparsity-inducing priors such as the normal-Jeffreys prior.
Revised: In contrast, MDL-C uses sparsity-inducing priors such as the normal-Jeffreys prior.

Original: Third, MDL-C uses a parametric π θ , whereas MPO uses a non-parametric one 2 .
Revised: Third, MDL-C uses a parametric π θ , whereas MPO uses a non-parametric one 2 .

Original: While there is also a parametric variant of MPO, this variant does not maintain θ and φ separately.
Revised: While there is also a parametric variant of MPO, this variant does not maintain θ and ϕ separately.

Original: Instead, this variant directly sets θ to φ in Equation (A.2).
Revised: Instead, this variant directly sets θ to ϕ in Equation (A.2).

Original: This illustrates the key conceptual difference between MDL-C and MPO.
Revised: This illustrates the key conceptual difference between MDL-C and MPO.

Original: MDL-C makes a clear distinction between the control policy π θ and the default policy π w , with the two policies serving two distinct purposes: the control policy for performing on the current task, the default policy for distilling optimal policies across tasks and generalizing to new ones.
Revised: MDL-C makes a clear distinction between the control policy π θ and the default policy π w , with the two policies serving two distinct purposes: the control policy for performing on the current task, the default policy for distilling optimal policies across tasks and generalizing to new ones.

Original: MPO, on the other hand, treats π θ and π w as fundamentally the same object.
Revised: MPO, on the other hand, treats π θ and π w as fundamentally the same object.

Original: Like MPO, VIREL (Fellows et al., 2020) can be derived from the control as inference framework.
Revised: Like MPO, VIREL (Fellows et al., 2020) can be derived from the control as inference framework.

Original: In fact, Fellows et al.
Revised: In fact, Fellows et al.

Original: showed that a parametric variant of MPO can be derived from VIREL (Fellows et al., 2020).
Revised: showed that a parametric variant of MPO can be derived from VIREL (Fellows et al., 2020).

Original: The key novelty that sets VIREL apart from both MPO and MDL-C is an adaptive temperature parameter that dynamically updates the inﬂuence of the KL term in Equation (A.2).
Revised: The key novelty that sets VIREL apart from both MPO and MDL-C is an adaptive temperature parameter that dynamically updates the influence of the KL term in Equation (A.2).

Original: B M ULTITASK RL F RAMEWORKS
Revised: B M ULTITASK RL F RAMEWORKS

Original: We believe the objective which best captures naturalistic settings is the average reward obtained over the agent’s “lifetime”: lim T →∞ 1 T E (cid:80)
Revised: We believe the objective which best captures naturalistic settings is the average reward obtained over the agent’s “lifetime”: lim T →∞ 1 T E (cid:80)

Original: Tt =1 r ( s t , a t ) .
Revised: Tt =1 r ( s t , a t ) .

Original: Typical objectives include ﬁnding either a single policy or a set of policies which maximize worst- or average-case value: max π min M ∈M V πM (Zahavy et al., 2021) or max π E P M V πM (Moskovitz et al., 2022a).
Revised: Typical objectives include finding either a single policy or a set of policies which maximize worst- or average-case value: max π min M ∈M V πM (Zahavy et al., 2021) or max π E P M V πM (Moskovitz et al., 2022a).

Original: When the emphasis is on decreasing the required sample complexity of learning new tasks, a useful metric is cumulative regret : the agent’s total shortfall across training compared to an optimal agent.
Revised: When the emphasis is on decreasing the required sample complexity of learning new tasks, a useful metric is cumulative regret : the agent’s total shortfall across training compared to an optimal agent.

Original: In practice, it’s often simplest to consider the task distribution P M to be a categorical distribution deﬁned over a discrete set of tasks
Revised: In practice, it’s often simplest to consider the task distribution P M to be a categorical distribution defined over a discrete set of tasks

Original: M :=
Revised: M :=

Original: { M k } Kk =1 , though continuous densities over MDPs are also possible.
Revised: { M k } Kk =1 , though continuous densities over MDPs are also possible.

Original: Two multitask settings which we consider here are parallel task RL and sequential task RL.
Revised: Two multitask settings which we consider here are parallel task RL and sequential task RL.

Original: In typical parallel task training (Yu et al., 2019), a new MDP is sampled from P M at the start of every episode and is associated with a particular input feature g ∈ G that indicates to the agent which task has been sampled.
Revised: In typical parallel task training (Yu et al., 2019), a new MDP is sampled from P M at the start of every episode and is associated with a particular input feature g ∈ G that indicates to the agent which task has been sampled.

Original: The agent’s performance is evaluated on all tasks M ∈ M together.
Revised: The agent’s performance is evaluated on all tasks M ∈ M together.

Original: In the sequential task setting (Moskovitz et al., 2022a; Pacchiano et al., 2022), tasks (MDPs) are sampled one at a time from P M , with the agent training on each until convergence.
Revised: In the sequential task setting (Moskovitz et al., 2022a; Pacchiano et al., 2022), tasks (MDPs) are sampled one at a time from P M , with the agent training on each until convergence.

Original: In contrast to continual learning (Kessler et al., 2021), the agent’s goal is simply to learn a new policy for each task more quickly as more are sampled, rather than learning a single policy which maintains its performance across tasks.
Revised: In contrast to continual learning (Kessler et al., 2021), the agent’s goal is simply to learn a new policy for each task more quickly as more are sampled, rather than learning a single policy which maintains its performance across tasks.

Original: Another important setting is meta-RL , which we do not consider here.
Revised: Another important setting is meta-RL , which we do not consider here.

Original: In the meta-RL setting, the agent trains on each sampled task for only a few episodes each with the goal of improving few-shot performance and is meta-tested on a set of held-out tasks (Yu et al., 2019; Finn et al., 2017).
Revised: In the meta-RL setting, the agent trains on each sampled task for only a few episodes each with the goal of improving few-shot performance and is meta-tested on a set of held-out tasks (Yu et al., 2019; Finn et al., 2017).

Original: Another strain of work in multitask RL assumes some form of shared structure in the transition dynamics (Pacchiano et al., 2022; ?
Revised: Another strain of work in multitask RL assumes some form of shared structure in the transition dynamics (Pacchiano et al., 2022; ?

Original: Speciﬁcally, the core assumption made by these works is that the transition dynamics are linearly decodable from a set of features which is shared across tasks or in which the transition matrix admits a low-rank decomposition.
Revised: Specifically, the core assumption made by these works is that the transition dynamics are linearly decodable from a set of features which is shared across tasks or in which the transition matrix admits a low-rank decomposition.

Original: This is very different from our own structural assumption—that is, in its simplest form, that the optimal policies of the tasks with which our agents are faced take similar actions in at least some part of the state space.
Revised: This is very different from our own structural assumption—that is, in its simplest form, that the optimal policies of the tasks with which our agents are faced take similar actions in at least some part of the state space.

Original: Beyond this, the MDPs in M need only share the same state and action space, with no direct assumptions about transitions or rewards.
Revised: Beyond this, the MDPs in M need only share the same state and action space, with no direct assumptions about transitions or rewards.

Original: This is important, because the assumed structures in the transition distribution made by Pacchiano et al.
Revised: This is important, because the assumed structures in the transition distribution made by Pacchiano et al.

Original: ; ? ); ? act as the starting points for algorithm development.
Revised: ; ? ); ? act as the starting points for algorithm development.

Original: MDL-C/RPO/TVPO however, can leverage similarity among optimal policies when it exists, but are not dependent on it as a prerequisite.
Revised: MDL-C/RPO/TVPO however, can leverage similarity among optimal policies when it exists, but are not dependent on it as a prerequisite.

Original: (E.g., TVPO (and RPO/MDL-C) is guaranteed to perform no worse than log-barrier regularization, which has a polynomial sample complexity guarantee.)
Revised: (E.g., TVPO (and RPO/MDL-C) is guaranteed to perform no worse than log-barrier regularization, which has a polynomial sample complexity guarantee.)

Original: Ideally, we’d like a generalist method which can identify on its own and exploit different types of structure in the environment.
Revised: Ideally, we’d like a generalist method which can identify on its own and exploit different types of structure in the environment.

Original: C A DDITIONAL R ELATED W ORK
Revised: C A DDITIONAL R ELATED W ORK

Original: Previous work has also applied the MDL principle in an RL context, though primarily in the context of unsupervised skill learning (Zhang et al., 2021; Thrun and Schwartz, 1994).
Revised: Previous work has also applied the MDL principle in an RL context, though primarily in the context of unsupervised skill learning (Zhang et al., 2021; Thrun and Schwartz, 1994).

Original: For example, Thrun and Schwartz (1994) are concerned with a set of “skills” which are policies deﬁned only over a subset of the state space that are reused across tasks.
Revised: For example, Thrun and Schwartz (1994) are concerned with a set of “skills” which are policies defined only over a subset of the state space that are reused across tasks.

Original: They consider tabular methods, measuring a pseudo-description length as
Revised: They consider tabular methods, measuring a pseudo-description length as

Original: [Equation]
Revised: [Equation]

Original: where P ∗ M ( s ) is the probability that no skill selects an action in state s for task M and the agent must compute the optimal Q -values in state s for M , N is the number of skills, and | S n | is the number states for which skill n is deﬁned.
Revised: where P ∗ M ( s ) is the probability that no skill selects an action in state s for task M and the agent must compute the optimal Q -values in state s for M , N is the number of skills, and | S n | is the number states for which skill n is defined.

Original: They then trade off this description length term with performance across a series of tabular environments.
Revised: They then trade off this description length term with performance across a series of tabular environments.

Original: One other related method is D ISTRAL (Teh et al., 2017), which uses the following objective in the parallel task setting:
Revised: One other related method is D ISTRAL (Teh et al., 2017), which uses the following objective in the parallel task setting:

Original: [Equation]
Revised: [Equation]

Original: That is, like the un-regularized RPO method, D ISTRAL can be seen as performing maximumlikelihood estimation to learn the (unconstrained) default policy, while adding an entropy bonus to the control policy.
Revised: That is, like the un-regularized RPO method, D ISTRAL can be seen as performing maximumlikelihood estimation to learn the (unconstrained) default policy, while adding an entropy bonus to the control policy.

Original: Another important method in the sequential setting is TVPO (Moskovitz et al., 2022a), in which (in the tabular case) the default policy is deﬁned as a softmax over the average action frequencies of the optimal policies for the tasks that the agent has seen so far.
Revised: Another important method in the sequential setting is TVPO (Moskovitz et al., 2022a), in which (in the tabular case) the default policy is defined as a softmax over the average action frequencies of the optimal policies for the tasks that the agent has seen so far.

Original: That is, if the average optimal action in a state s is given by
Revised: That is, if the average optimal action in a state s is given by

Original: [Equation]
Revised: [Equation]

Original: then the TVPO default policy is
Revised: then the TVPO default policy is

Original: [Equation]
Revised: [Equation]

Original: where β ( k ) is a temperature which decays as k → ∞ .
Revised: where β ( k ) is a temperature which decays as k → ∞ .

Original: In high-dimensional state and action spaces, this tabular solution can be approximated by training a default policy to predict the converged control policy’s actions in each task.
Revised: In high-dimensional state and action spaces, this tabular solution can be approximated by training a default policy to predict the converged control policy’s actions in each task.

Original: Importantly, this is equivalent to using KL distillation in that the default policies will converge to the same barycenter policy (Moskovitz et al., 2022a), as long as the distillation is only performed once the control policy has converged in each task.
Revised: Importantly, this is equivalent to using KL distillation in that the default policies will converge to the same barycenter policy (Moskovitz et al., 2022a), as long as the distillation is only performed once the control policy has converged in each task.

Original: Using KL distillation in this way is exactly the RPO baseline that we use in this paper.
Revised: Using KL distillation in this way is exactly the RPO baseline that we use in this paper.

Original: Crucially, the use of the softmax with decaying temperature was introduced by Moskovitz et al.
Revised: Crucially, the use of the softmax with decaying temperature was introduced by Moskovitz et al.

Original: (2022a) as a useful ‘hack’ to prevent the default policy from overﬁtting to early tasks, as the optimal default policy is the barycenter policy (approximated as the number of draws from the task distribution grows).
Revised: (2022a) as a useful ‘hack’ to prevent the default policy from overfitting to early tasks, as the optimal default policy is the barycenter policy (approximated as the number of draws from the task distribution grows).

Original: Thus, MDL-C can itself be seen as a scalable advancement of TVPO which models the agent’s epistemic uncertainty about the task distribution by placing a sparse prior over the default policy parameters (and uses a distillation loss rather than action prediction).
Revised: Thus, MDL-C can itself be seen as a scalable advancement of TVPO which models the agent’s epistemic uncertainty about the task distribution by placing a sparse prior over the default policy parameters (and uses a distillation loss rather than action prediction).

Original: In other words, MDL-C represents a principled approach to reducing the risk of default policy overﬁtting in the low-data regime.
Revised: In other words, MDL-C represents a principled approach to reducing the risk of default policy overfitting in the low-data regime.

Original: Finally, Brunskill and Li (2013) consider a similar training and task structure to our own, but use a model-based approach to learn the underlying MDPs.
Revised: Finally, Brunskill and Li (2013) consider a similar training and task structure to our own, but use a model-based approach to learn the underlying MDPs.

Original: D M OTIVATING THE CHOICE OF SPARSITY - INDUCING PRIORS
Revised: D M OTIVATING THE CHOICE OF SPARSITY - INDUCING PRIORS

Original: As a reminder, the generative model of optimal parameters in Section 4.1 is given by:
Revised: As a reminder, the generative model of optimal parameters in Section 4.1 is given by:

Original: [Equation]
Revised: [Equation]

Original: with marginal and posterior densities
Revised: with marginal and posterior densities

Original: [Equation]
Revised: [Equation]

Original: In the rest of this section, we set σ 2 = 1 for simplicity and drop the indices on w and w to remove clutter.
Revised: In the rest of this section, we set σ 2 = 1 for simplicity and drop the indices on w and w to remove clutter.

Original: D. C ORRESPONDENCE BETWEEN p ( z ) AND p ( β )
Revised: D. C ORRESPONDENCE BETWEEN p ( z ) AND p ( β )

Original: In Section 4.1, we draw a connection between p ( β ) ∝ β − 1 and the normal-Jeffreys prior, which is commonly used for compressing deep neural networks (Louizos et al., 2017).
Revised: In Section 4.1, we draw a connection between p ( β ) ∝ β − 1 and the normal-Jeffreys prior, which is commonly used for compressing deep neural networks (Louizos et al., 2017).

Original: In Table 1, we expand on this connection and list p ( β ) for two other commonly-used priors for scale mixture of normal distributions: Jeffreys, Inverse-gamma, and Inverse-beta.
Revised: In Table 1, we expand on this connection and list p ( β ) for two other commonly-used priors for scale mixture of normal distributions: Jeffreys, Inverse-gamma, and Inverse-beta.

Original: Note that the half-Cauchy distribution p ( z ) ∝ (1 + z 2 ) − 1 is a special case of the inverse-beta distribution for s = t = 1 / 2 .
Revised: Note that the half-Cauchy distribution p ( z ) ∝ (1 + z 2 ) − 1 is a special case of the inverse-beta distribution for s = t = 1 / 2 .

Original: Half-cauchy prior is another commonly used prior for compressing Bayesian neural networks (Louizos et al., 2017).
Revised: Half-cauchy prior is another commonly used prior for compressing Bayesian neural networks (Louizos et al., 2017).

Original: D. MSE RISK
Revised: D. MSE RISK

Original: In this section, we prove that the Bayes estimators for the Jeffreys, inverse-gamma, and the inversebeta (by extension the half-Cauchy) distributions dominate the maximum-likelihood estimator with respect to the mean-squared error.
Revised: In this section, we prove that the Bayes estimators for the Jeffreys, inverse-gamma, and the inversebeta (by extension the half-Cauchy) distributions dominate the maximum-likelihood estimator with respect to the mean-squared error.

Original: Deﬁne the mean-squared error of an estimator ˆ w ( x ) of w as
Revised: Define the mean-squared error of an estimator ˆ w ( x ) of w as

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over N ( x ; w, α 2 ) .
Revised: where the expectation is taken over N ( x ; w, α 2 ) .

Original: Immediately, we have R ( w, ˆ w ( ML ) )
Revised: Immediately, we have R ( w, ˆ w ( ML ) )

Original: =
Revised: =

Original: d , where ˆ w ( ML ) ( x ) = x is the maximum-likelihood estimator.
Revised: d , where ˆ w ( ML ) ( x ) = x is the maximum-likelihood estimator.

Original: An estimator ˆ w ( a ) ( x ) is said to dominate another estimator ˆ w ( b ) ( x ) if MSE ( w, ˆ w a ) ≤ MSE ( w, ˆ w b ) for all w and the inequality is strict for a set of positive Lesbesgue measure.
Revised: An estimator ˆ w ( a ) ( x ) is said to dominate another estimator ˆ w ( b ) ( x ) if MSE ( w, ˆ w a ) ≤ MSE ( w, ˆ w b ) for all w and the inequality is strict for a set of positive Lesbesgue measure.

Original: It is well-known that the maximum-likelihood estimator is minimax (George et al., 2006), and thus any estimator that dominates the maximum-likelihood estimator is also minimax.
Revised: It is well-known that the maximum-likelihood estimator is minimax (George et al., 2006), and thus any estimator that dominates the maximum-likelihood estimator is also minimax.

Original: To compute the mean-squared error risk for an estimator ˆ w ( x ) , observe that
Revised: To compute the mean-squared error risk for an estimator ˆ w ( x ) , observe that

Original: [Equation]
Revised: [Equation]

Original: Taking expectations on both sides gives
Revised: Taking expectations on both sides gives

Original: [Equation]
Revised: [Equation]

Original: where ∇ =
Revised: where ∇ =

Original: ( ∂/∂x 1 , . . .
Revised: ( ∂/∂x 1 , . . .

Original: , ∂/∂x d ) and we apply Stein’s lemma cov ( ˆ w i ( x ) , x i ) =
Revised: , ∂/∂x d ) and we apply Stein’s lemma cov ( ˆ w i ( x ) , x i ) =

Original: E x ∂ ˆ w
Revised: E x ∂ ˆ w

Original: i
Revised: i

Original: /∂x i in the last line.
Revised: /∂x i in the last line.

Original: If the estimator takes the form ˆ w ( x ) = x + γ ( x ) , the expression simpliﬁes as:
Revised: If the estimator takes the form ˆ w ( x ) = x + γ ( x ) , the expression simplifies as:

Original: [Equation]
Revised: [Equation]

Original: Therefore, an estimator ˆ w ( x ) = x
Revised: Therefore, an estimator ˆ w ( x ) = x

Original: + γ ( x ) dominates ˆ w ( ML )
Revised: + γ ( x ) dominates ˆ w ( ML )

Original: ( x ) if
Revised: ( x ) if

Original: [Equation]
Revised: [Equation]

Original: for all w and the inequality is strict on a set of positive Lesbesgue measure.
Revised: for all w and the inequality is strict on a set of positive Lesbesgue measure.

Original: D.2. J AMES -S TEIN ESTIMATOR
Revised: D.2. J AMES -S TEIN ESTIMATOR

Original: The famous Jame-Stein estimator is deﬁned as
Revised: The famous Jame-Stein estimator is defined as

Original: [Equation]
Revised: [Equation]

Original: with ∇ · γ ( JS ) ( x ) =

Original: d (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:20) [Equation] (cid:107) [Equation]
Revised: with ∇

Original: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ( (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107) [Equation] (cid:21) [Equation]
Revised: (D.13)

Original:  (d [Equation] [Equation] (cid:107) x (cid:107) 2 , (D.12) (cid:107)
Revised: Substituting ∇ · γ ( JS ) ( x ) and ∥ γ ( JS ) ( x ) ∥ 2 into Equation (D.10), we have

Original: [Equation]
Revised: [Equation]

Original: ( JS ) ( x ) (cid:107) [Equation] = ( d −
Revised: D

Original: [Equation]
Revised: [Equation]

Original: )(cid:107) x (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (D.13) Substituting ∇ · γ ( JS ) ( x ) and (cid:107) γ ( JS ) ( x ) (cid:107) 2 into Equation (D.10), we have
Revised: Thus, the James-Stein estimator dominates the maximum-likelihood estimator for d > 2 .

Original:  [Equation]
Revised: [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (D.14) Thus, the James-Stein estimator dominates the maximum-likelihood estimator for d > 2 .
Revised: Inverse

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: Inverse

Revised: [Table]

Revised: beta p

Revised: [Equation]

Original:  D.2. B AYES ESTIMATORS
Revised: [Table] D.2. B AYES ESTIMATORS

Original: The Bayes estimator for a prior choice p ( β ) is given by ( ? ):
Revised: The Bayes estimator for a prior choice p ( β ) is given by ( ? ):

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Substituting γ ( p ( β ))
Revised: Substituting γ ( p ( β ))

Original: ( x ) into Equation (D.10), we ﬁnd that the condition for the Bayes estimator to be minimax is given by (George et al., 2006):
Revised: ( x ) into Equation (D.10), we find that the condition for the Bayes estimator to be minimax is given by (George et al., 2006):

Original: [Equation]
Revised: [Equation]

Original: where ∇ 2 = (cid:80) i ∂ 2 /∂x 

Revised: where ∇ 2 = (cid:80) i ∂ 2 /∂x

Original: i is the Laplace operator.
Revised: i is the Laplace operator.

Original: This condition holds when (cid:112) m ( x ) is superharmonic (i.e., (cid:112) m ( x ) ≤ 0 , ∀ x ∈ R d ), suggesting a recipe for constructing Bayes estimators that dominate the maximum likelihood estimator, summarized in the following proposition.
Revised: This condition holds when (cid:112) m ( x ) is superharmonic (i.e., (cid:112) m ( x ) ≤ 0 , ∀ x ∈ R d ), suggesting a recipe for constructing Bayes estimators that dominate the maximum likelihood estimator, summarized in the following proposition.

Original: Proposition D.1 (Extension of Theorem 1 in Fourdrinier et al., 1998) .
Revised: Proposition D.1 (Extension of Theorem 1 in Fourdrinier et al., 1998) .

Original: Let p ( β ) be a positive function such that f ( β ) =
Revised: Let p ( β ) be a positive function such that f ( β ) =

Original: βp (cid:48) ( β ) /p ( β ) can be decomposed as f 1 ( β ) + f 2 ( β ) where f 1 is non-decreasing, f 1 ≤ A , 0 < f 2 ≤ B , and A/ 2 + B ≤ ( d − 6) / 4 .
Revised: βp ′ ( β ) /p ( β ) can be decomposed as f 1 ( β ) + f 2 ( β ) where f 1 is non-decreasing, f 1 ≤ A , 0 < f 2 ≤ B , and A/ 2 + B ≤ ( d − 6) / 4 .

Original: Assume also that lim β
Revised: Assume also that lim β

Original: → 0 β d/ 2+2 p ( β ) = 0 .
Revised: → 0 β d/ 2+2 p ( β ) = 0 .

Original: Then, ∇ 2 (cid:112) m ( x ) ≤ 0 and the Bayes estimator is minimax.
Revised: Then, ∇ 2 (cid:112) m ( x ) ≤ 0 and the Bayes estimator is minimax.

Original: If A/ 2 + B < ( d − 6) / 4 , then the Bayes estimator dominates ˆ w ( ML ) ( x ) .
Revised: If A/ 2 + B < ( d − 6) / 4 , then the Bayes estimator dominates ˆ w ( ML ) ( x ) .

Original: Proof.
Revised: Proof.

Original: This proof largely follows the proof of Theorem 1 in (Fourdrinier et al., 1998).
Revised: This proof largely follows the proof of Theorem 1 in (Fourdrinier et al., 1998).

Original: Note that Equation (D.18) holds if
Revised: Note that Equation (D.18) holds if

Original: [Equation]
Revised: [Equation]

Original: or equivalently
Revised: or equivalently

Original: [Equation]
Revised: [Equation]

Original: Computing the derivatives, we get the condition
Revised: Computing the derivatives, we get the condition

Original: [Equation]
Revised: [Equation]

Original: Divide both sides by (cid:107) x (cid:107) and rearrange to get
Revised: Divide both sides by ∥ x ∥ and rearrange to get

Original: [Equation]
Revised: [Equation]

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we integrate by parts the numerator of the ﬁrst term on the left-hand side to get:
Revised: we integrate by parts the numerator of the first term on the left-hand side to get:

Original: [Equation]
Revised: [Equation]

Original: where the middle term is the same as the denominator of the ﬁrst term in Equation (D.23).
Revised: where the middle term is the same as the denominator of the first term in Equation (D.23).

Original: Integrating by parts the second term gives the same expression as that of the ﬁrst term, but with d − 2 in place of d everywhere.
Revised: Integrating by parts the second term gives the same expression as that of the first term, but with d − 2 in place of d everywhere.

Original: Substituting these expressions back into Equation (D.23), collecting like terms, and dividing both sides by 2 / (cid:107) x (cid:107) 2 , gives:
Revised: Substituting these expressions back into Equation (D.23), collecting like terms, and dividing both sides by 2 / ∥ x ∥ 2 , gives:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here, both κ 0 and κ 1 are nonpositive: (i) κ 0 is nonpositive because the ﬁrst term vanishes due to the boundary conditions and the second term is nonpositive, and (ii) κ 1 is nonpositive because the limits of the numerators of the two terms are equal while the denominator of the second term is larger than that of the ﬁrst.
Revised: Here, both κ 0 and κ 1 are nonpositive: (i) κ 0 is nonpositive because the first term vanishes due to the boundary conditions and the second term is nonpositive, and (ii) κ 1 is nonpositive because the limits of the numerators of the two terms are equal while the denominator of the second term is larger than that of the first.

Original: We can thus drop κ 0 and κ 1 to get the sufﬁcient condition:
Revised: We can thus drop κ 0 and κ 1 to get the sufficient condition:

Original: [Equation]
Revised: [Equation]

Original: where E d denotes expectation with respect to the density
Revised: where E d denotes expectation with respect to the density

Original: [Equation]
Revised: [Equation]

Original: and where f ( β ) = βp (cid:48) ( β ) /p
Revised: and where f ( β ) = βp ′ ( β ) /p

Original: ( β ) .
Revised: ( β ) .

Original: Because g d ( β ) is a family of monotone increasing likelihood ratio in d and f 1 is nonincreasing and bounded by A , we have E d ( f 1 ) − E d − 2 ( f 1 ) / 2 ≤ A/ 2 .
Revised: Because g d ( β ) is a family of monotone increasing likelihood ratio in d and f 1 is nonincreasing and bounded by A , we have E d ( f 1 ) − E d − 2 ( f 1 ) / 2 ≤ A/ 2 .

Original: We have E d ( f 2 ) − E d − 2 ( f 2 ) / 2 ≤ B because 0 < f 2 ≤ B .
Revised: We have E d ( f 2 ) − E d − 2 ( f 2 ) / 2 ≤ B because 0 < f 2 ≤ B .

Original: Taken together, we have
Revised: Taken together, we have

Original: [Equation]
Revised: [Equation]

Original: When the inequality is strict (i.e., A/ 2 + B < ( k − 6) / 4 ), then ∇ 2 (cid:112) m ( x ) < 0 and the Bayes estimator dominates the maximum-likelihood estimator.
Revised: When the inequality is strict (i.e., A/ 2 + B < ( k − 6) / 4 ), then ∇ 2 (cid:112) m ( x ) < 0 and the Bayes estimator dominates the maximum-likelihood estimator.

Original: Checking whether a given p ( β ) satisfy the conditions in Proposition D.1 may be tedious.
Revised: Checking whether a given p ( β ) satisfy the conditions in Proposition D.1 may be tedious.

Original: The following corollary is useful for construction p ( β ) that satisﬁes the conditions in Proposition D.1.
Revised: The following corollary is useful for construction p ( β ) that satisfies the conditions in Proposition D.1.

Original: Corollary D.1 (Extension of Corollary 1 in Fourdrinier et al., 1998) .
Revised: Corollary D.1 (Extension of Corollary 1 in Fourdrinier et al., 1998) .

Original: Let ψ be a continuous function that can be decomposed as ψ 1 + ψ 2 , with ψ 1 ≤ C , ψ 1 non-decreasing, 0 < ψ 2 ≤ D , and
Revised: Let ψ be a continuous function that can be decomposed as ψ 1 + ψ 2 , with ψ 1 ≤ C , ψ 1 non-decreasing, 0 < ψ 2 ≤ D , and

Original: C/ 2 + D ≤ 0 .
Revised: C/ 2 + D ≤ 0 .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: such that lim β → 0 β d/ 2+2 p ( β ) = 0 and β 0 ∈ (0 , 1) is a constant.
Revised: such that lim β → 0 β d/ 2+2 p ( β ) = 0 and β 0 ∈ (0 , 1) is a constant.

Original: Then, p ( β ) results in a minimax
Revised: Then, p ( β ) results in a minimax

Original: Bayes estimator, which dominates the maximum likelihood estimator when C/ 2 + D < 0 .
Revised: Bayes estimator, which dominates the maximum likelihood estimator when C/ 2 + D < 0 .

Original: Proof.
Revised: Proof.

Original: The proof is the same as that of Corollary 1 in Fourdrinier et al., 1998, with Proposition D. in place of Theorem 1 in Fourdrinier et al., 1998.
Revised: The proof is the same as that of Corollary 1 in Fourdrinier et al., 1998, with Proposition D. in place of Theorem 1 in Fourdrinier et al., 1998.

Original: Using Corollary D.1, we now check that the three priors listed in Table 1 and referenced in Section 4.
Revised: Using Corollary D.1, we now check that the three priors listed in Table 1 and referenced in Section 4.

Original: lead to Bayes estimators that dominate the maximum-likelihood estimator.
Revised: lead to Bayes estimators that dominate the maximum-likelihood estimator.

Original: Jeffreys prior Let ψ 1 ( u ) =
Revised: Jeffreys prior Let ψ 1 ( u ) =

Original: a for a ≤ 0 and ψ 2 ( u ) = 0 .
Revised: a for a ≤ 0 and ψ 2 ( u ) = 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: To satisfy lim β → 0 β d/ 2+2 p ( β ) = 0 , we require 1 − d < a ≤ 0 .
Revised: To satisfy lim β → 0 β d/ 2+2 p ( β ) = 0 , we require 1 − d < a ≤ 0 .

Original: We recover the improper normalJeffreys prior p ( β ) ∝ β − 1 , for a = 2 − d/ 2 .
Revised: We recover the improper normalJeffreys prior p ( β ) ∝ β − 1 , for a = 2 − d/ 2 .

Original: The corresponding Bayes estimator dominates the maximum likelihood estimator when d > 4 .
Revised: The corresponding Bayes estimator dominates the maximum likelihood estimator when d > 4 .

Original: Inverse-gamma prior Let ψ 1 ( u ) = a and ψ 2 ( u )
Revised: Inverse-gamma prior Let ψ 1 ( u ) = a and ψ 2 ( u )

Original: =
Revised: =

Original: b (1 − u ) / 2 for a ≤ 0 and b ≥ 0 .
Revised: b (1 − u ) / 2 for a ≤ 0 and b ≥ 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Setting C = a and D = b/ 2 , we get the followings conditions: a + b ≤ 0 and 1 − d ≤ a + b/ 2 .
Revised: Setting C = a and D = b/ 2 , we get the followings conditions: a + b ≤ 0 and 1 − d ≤ a + b/ 2 .

Original: Note that when these conditions are met with s = a + ( b + d − 4) / 2 and t = b , we recover the inverse-gamma prior in Table 1.
Revised: Note that when these conditions are met with s = a + ( b + d − 4) / 2 and t = b , we recover the inverse-gamma prior in Table 1.

Original: Inverse-beta (half-Cauchy) prior Let ψ 1 ( u ) = a and ψ 2 ( u ) =
Revised: Inverse-beta (half-Cauchy) prior Let ψ 1 ( u ) = a and ψ 2 ( u ) =

Original: b/ ( u + 1) for a ≤ 0 and b ≥ 0 .
Revised: b/ ( u + 1) for a ≤ 0 and b ≥ 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Setting C = a and D
Revised: Setting C = a and D

Original: = b , we get the condition a/ 2 + b ≤ 0 .
Revised: = b , we get the condition a/ 2 + b ≤ 0 .

Original: To satisfy lim β → 0 β d/ 2+2 p ( β ) = 0 , we require 1 − d < a + b ≤ 0 .
Revised: To satisfy lim β → 0 β d/ 2+2 p ( β ) = 0 , we require 1 − d < a + b ≤ 0 .

Original: Note that this corresponds to the inverse-beta prior in Table 1 with t = a + ( d − 8) / 2 and s = b − t .
Revised: Note that this corresponds to the inverse-beta prior in Table 1 with t = a + ( d − 8) / 2 and s = b − t .

Original: To recover the half-Cauchy prior, we set b = 1 and a = (5 − d ) / 2 .
Revised: To recover the half-Cauchy prior, we set b = 1 and a = (5 − d ) / 2 .

Original: All conditions in Corollary D. are satisﬁed when d > 9 .
Revised: All conditions in Corollary D. are satisfied when d > 9 .

Original: E L IMITATIONS
Revised: E L IMITATIONS

Original: One weakness of the current theoretical analysis regarding the choice of sparsity-inducing priors is the assumption of Gaussian (and in particular, isotropic Gaussian) structure in the parameter space of optimal policies for clusters of tasks.
Revised: One weakness of the current theoretical analysis regarding the choice of sparsity-inducing priors is the assumption of Gaussian (and in particular, isotropic Gaussian) structure in the parameter space of optimal policies for clusters of tasks.

Original: In reality, there is likely a nontrivial degree of covariance among task parameterizations.
Revised: In reality, there is likely a nontrivial degree of covariance among task parameterizations.

Original: Extending our analysis to more realistic forms of task structure is an important direction for future work.
Revised: Extending our analysis to more realistic forms of task structure is an important direction for future work.

Original: In a similar vein, the assumption that tasks are drawn iid from a ﬁxed distribution is also unrealistic in naturalistic settings.
Revised: In a similar vein, the assumption that tasks are drawn iid from a fixed distribution is also unrealistic in naturalistic settings.

Original: It would be interesting to introduce some form of sequential structure (e.g., tasks are drawn from a Markov process).
Revised: It would be interesting to introduce some form of sequential structure (e.g., tasks are drawn from a Markov process).

Original: Another direction for future work is expanding beyond the “one control policy, one default policy” setup–having, for example, one default policy per task cluster and the ability to reuse and select (for example, using successor feature-like representations (Barreto et al., 2020; Barth-Maron et al., 2018; Moskovitz et al., 2022b)) among an actively-maintained set of control policies across tasks and task clusters would be useful.
Revised: Another direction for future work is expanding beyond the “one control policy, one default policy” setup–having, for example, one default policy per task cluster and the ability to reuse and select (for example, using successor feature-like representations (Barreto et al., 2020; Barth-Maron et al., 2018; Moskovitz et al., 2022b)) among an actively-maintained set of control policies across tasks and task clusters would be useful.

Original: F OCO B ACKGROUND
Revised: F OCO B ACKGROUND

Original: In online convex optimization (OCO), the learner observes a series of convex loss functions (cid:96) k : N → R , k = 1 , . .
Revised: In online convex optimization (OCO), the learner observes a series of convex loss functions ℓ k : N → R , k = 1 , . .

Original: , K , where N ⊆ R d is a convex set.
Revised: , K , where N ⊆ R d is a convex set.

Original: After each round, the learner produces an output x k ∈ N for which it will then incur a loss (cid:96) k ( x k ) (Orabona, 2019).
Revised: After each round, the learner produces an output x k ∈ N for which it will then incur a loss ℓ k ( x k ) (Orabona, 2019).

Original: At round k , the learner is usually assumed to have knowledge of (cid:96) 1 , . . . 
Revised: At round k , the learner is usually assumed to have knowledge of ℓ 1 , . . . ,

Original: , (cid:96) k − 1 , but no other assumptions are made about the sequence of loss functions.
Revised: ℓ k − 1 , but no other assumptions are made about the sequence of loss functions.

Original: The learner’s goal is to minimize its average regret:
Revised: The learner’s goal is to minimize its average regret:

Original: [Equation]
Revised: [Equation]

Original: One OCO algorithm which enjoys sublinear regret is follow the regularized leader (FTRL).
Revised: One OCO algorithm which enjoys sublinear regret is follow the regularized leader (FTRL).

Original: In each round of FTRL, the learner selects the solution x ∈ N according to the following objective:
Revised: In each round of FTRL, the learner selects the solution x ∈ N according to the following objective:

Original: [Equation]
Revised: [Equation]

Original: where ψ k : N → R is a convex regularization function.
Revised: where ψ k : N → R is a convex regularization function.

Original: G P ROOFS OF P ERFORMANCE B OUNDS AND A DDITIONAL T HEORETICAL R ESULTS
Revised: G P ROOFS OF P ERFORMANCE B OUNDS AND A DDITIONAL T HEORETICAL R ESULTS

Original: The following result is useful.
Revised: The following result is useful.

Original: Lemma G.1.
Revised: Lemma G.1.

Original: The function (cid:96) ( ν ) =
Revised: The function ℓ ( ν ) =

Original: E w ∼ ν f ( w ) is L -Lipschitz
Revised: E w ∼ ν f ( w ) is L -Lipschitz

Original: as long as f : W → R lies within [0 , L ] ∀ w ∈ W , where
Revised: as long as f : W → R lies within [0 , L ] ∀ w ∈ W , where

Original: W ⊆ R
Revised: W ⊆ R

Original: d is a Hilbert space and L < ∞ .
Revised: d is a Hilbert space and L < ∞ .

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to Cauchy-Schwarz and the second is by assumption on f .
Revised: where the first inequality is due to Cauchy-Schwarz and the second is by assumption on f .

Original: Proposition G.1 (Default Policy Distribution Regret) .
Revised: Proposition G.1 (Default Policy Distribution Regret) .

Original: Let tasks M k be independently drawn from
Revised: Let tasks M k be independently drawn from

Original: P M at every round, and let them each be associated with a deterministic optimal policy
Revised: P M at every round, and let them each be associated with a deterministic optimal policy

Original: π (cid:63)k : S → A .
Revised: π ⋆k : S → A .

Original: We make the following mild assumptions: i) π w ( [Equation] s ) ≥ (cid:15) > 0 ∀ s ∈ S , where a (cid:63) = 
Revised: We make the following mild assumptions: i) π w ( a ⋆ | s ) ≥ ϵ > 0 ∀ s ∈ S , where a ⋆ = π

Original: π (cid:63)k ( s ) and (cid:15) is a constant. ii) min ν KL
Revised: ⋆k ( s ) and ϵ is a constant. ii) min ν KL

Original: [ ν ( · ) , p ( · )]
Revised: [ ν ( · ) , p ( · )]

Original: → 0 as Var[ ν ] → ∞ for an appropriate choice of sparsity-inducing prior p .
Revised: → 0 as Var[ ν ] → ∞ for an appropriate choice of sparsity-inducing prior p .

Original: Then Algorithm 2 guarantees
Revised: Then Algorithm 2 guarantees

Original: [Equation]
Revised: [Equation]

Original: where ¯ ν K = argmin [Equation]
Revised: where ¯ ν K = argmin ν ∈ N (cid:80) Kk =1 ℓ k ( ν ) .

Original: Proof.
Revised: Proof.

Original: The ﬁrst part of the proof sets up an application of Orabona (2019), Corollary 7.9.
Revised: The first part of the proof sets up an application of Orabona (2019), Corollary 7.9.

Original: To establish grounds for its application, we ﬁrst note the standard result that the regularization functional ψ ( ν ) =
Revised: To establish grounds for its application, we first note the standard result that the regularization functional ψ ( ν ) =

Original: KL [ ν ( w ) , p ( w )] for probability measures ν, p ∈ P ( W ) is 1 -strongly convex in ν (Melbourne, 2020).
Revised: KL [ ν ( w ) , p ( w )] for probability measures ν, p ∈ P ( W ) is 1 -strongly convex in ν (Melbourne, 2020).

Original: Finally, assumption (i) implies that the KL between the default policy and the optimal policy is upper-bounded: KL [ π (cid:63)k , π w ] ≤ log 1 /(cid:15) .
Revised: Finally, assumption (i) implies that the KL between the default policy and the optimal policy is upper-bounded: KL [ π ⋆k , π w ] ≤ log 1 /ϵ .

Original: Then by Lemma G.1, (cid:96) k ( ν ) is L -Lipschitz wrt the TV distance, where L = log 1 /(cid:15) .
Revised: Then by Lemma G.1, ℓ k ( ν ) is L -Lipschitz wrt the TV distance, where L = log 1 /ϵ .

Original: Note also that under a Gaussian parameterization for ν , the distribution space N is the Gaussian parameter space N = { ( µ, Σ) : µ ∈ R d , Σ ∈ R d × d , Σ (cid:23) 0 } , which is convex (Boyd and
Revised: Note also that under a Gaussian parameterization for ν , the distribution space N is the Gaussian parameter space N = { ( µ, Σ) : µ ∈ R d , Σ ∈ R d × d , Σ ⪰ 0 } , which is convex (Boyd and

Original: Vandenberghe, 2004).
Revised: Vandenberghe, 2004).

Original: Then Orabona (2019), Corollary 7.9 gives
Revised: Then Orabona (2019), Corollary 7.9 gives

Original: [Equation]
Revised: [Equation]

Original: where ¯ ν K = argmin ν (cid:80) Kk =1 (cid:96) k ( ν ) .
Revised: where ¯ ν K = argmin ν (cid:80) Kk =1 ℓ k ( ν ) .

Original: The constant α ∈ R
Revised: The constant α ∈ R

Original: + is a hyperparameter, so we are free to set it to 1 (Orabona, 2019).
Revised: + is a hyperparameter, so we are free to set it to 1 (Orabona, 2019).

Original: Finally, we observe that E P M i 1 K (cid:80)
Revised: Finally, we observe that E P M i 1 K (cid:80)

Original: Kk =1 (cid:96) ( ν k ) =
Revised: Kk =1 ℓ ( ν k ) =

Original: E P M i (cid:96) K ( ν K ) and take the expectation with respect to P M i of both sides of Eq.
Revised: E P M i ℓ K ( ν K ) and take the expectation with respect to P M i of both sides of Eq.

Original: (G.2) to get the desired result:
Revised: (G.2) to get the desired result:

Original: [Equation]
Revised: [Equation]

Original: Proposition 4.2 (Control Policy Sample Complexity) .
Revised: Proposition 4.2 (Control Policy Sample Complexity) .

Original: Under the setting described in Proposition G.1, denote by T k the number of iterations to reach (cid:15) -error for M k in the sense that min t ≤ T k { V π (cid:63)k −
Revised: Under the setting described in Proposition G.1, denote by T k the number of iterations to reach ϵ -error for M k in the sense that min t ≤ T k { V π ⋆k −

Original: V ( t ) } ≤ (cid:15). whenever t > T k .
Revised: V ( t ) } ≤ ϵ. whenever t > T k .

Original: Further, denote the upper-bound in Eq.
Revised: Further, denote the upper-bound in Eq.

Original: (G.1) by G ( K ) .
Revised: (G.1) by G ( K ) .

Original: In a ﬁnite
Revised: In a finite

Original: MDP, from any initial θ (0) , and following gradient ascent, E M k ∼P M
Revised: MDP, from any initial θ (0) , and following gradient ascent, E M k ∼P M

Original: [ T k ] satisﬁes:
Revised: [ T k ] satisfies:

Original: [Equation]
Revised: [Equation]

Original: where α k ( s ) :
Revised: where α k ( s ) :

Original: = d TV ( π (cid:63)k ( ·| s ) , ˆ π 0 ( ·| s )) ≤ (cid:112) G ( K ) , κ α k A
Revised: = d TV ( π ⋆k ( ·| s ) , ˆ π 0 ( ·| s )) ≤ (cid:112) G ( K ) , κ α k A

Original: ( s )
Revised: ( s )

Original: = 2 |A| (1 − α ( s ))
Revised: = 2 |A| (1 − α ( s ))

Original: 2 |A| (1 − α ( s )) − 1 , and µ is a measure over S such that µ ( s ) > 0 ∀ s ∈ S .
Revised: 2 |A| (1 − α ( s )) − 1 , and µ is a measure over S such that µ ( s ) > 0 ∀ s ∈ S .

Original: Note:
Revised: Note:

Original: In the above, there is a small error—it should be α k ( s ) :
Revised: In the above, there is a small error—it should be α k ( s ) :

Original: =
Revised: =

Original: E w ∼ ν d TV ( π (cid:63)k ( ·| s ) , π w
Revised: E w ∼ ν d TV ( π ⋆k ( [Equation] |

Original: [Equation]
Revised: [Equation]

Original:  ·| s )) ≤ (cid:113) 12 G ( K ) .
Revised: , π w ( ·| s )) ≤ (cid:113) 12 G ( K ) .

Original: d πρ refers to the discounted state-occupancy distribution under π with initial state distribution ρ :
Revised: d πρ refers to the discounted state-occupancy distribution under π with initial state distribution ρ :

Original: [Equation]
Revised: [Equation]

Original: Division between probability mass functions is assumed to be element-wise.
Revised: Division between probability mass functions is assumed to be element-wise.

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we prove the bound for a ﬁxed state s ∈ S , noting that the bound applies independently of our choice of s .
Revised: Without loss of generality [Equation] we prove the bound for a fixed state s ∈ S , noting that the bound applies independently of our choice of s [Equation]

Original: We use the shorthand KL [ π ( ·| s ) , π w ( ·| s )]
Revised: We use the shorthand KL [ π ( ·| s ) , π w ( ·| s )]

Original: → KL [ π, π w ] for brevity.
Revised: → KL [ π, π w ] for brevity.

Original: We start by multiplying both sides of the bound from Proposition G.1 by 1 / 2 and rearranging:
Revised: We start by multiplying both sides of the bound from Proposition G.1 by 1 / 2 and rearranging:

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from the deﬁnition of the variance, and ( ii ) follows from its non-negativity.
Revised: where ( i ) follows from the definition of the variance [Equation] and ( ii ) follows from its non-negativity.

Original: We can rearrange to get
Revised: We can rearrange to get

Original: [Equation]
Revised: [Equation]

Original: where ( ii ) follows from Pinsker’s inequality.
Revised: where ( ii ) follows from Pinsker’s inequality.

Original: Letting α K ( s )
Revised: Letting α K ( s )

Original: = (cid:113) 1 2 G ( K ) and applying Moskovitz et al.
Revised: = (cid:113) 1 2 G ( K ) and applying Moskovitz et al.

Original: (2022a)
Revised: (2022a)

Original: , Lemma 5.2 gives the desired result.
Revised: , Lemma 5.2 gives the desired result.

Original: This upper-bound is signﬁcant, as it shows that, all else being equal, a high complexity barycenter default policy distribution ¯ ν K (where complexity is measured by KL [¯ ν K , p ] ) leads to a slower convergence rate in the control policy.
Revised: This upper-bound is signficant, as it shows that, all else being equal, a high complexity barycenter default policy distribution ¯ ν K (where complexity is measured by KL [¯ ν K , p ] ) leads to a slower convergence rate in the control policy.

Original: 1: require: task distribution P M , policy class Π , coefﬁcients { η k } 2: initialize: default policy distribution ν 1 ∈ N 3: for tasks k
Revised: 1: require: task distribution P M , policy class Π , coefficients { η k } 2: initialize: default policy distribution ν 1 ∈ N 3: for tasks k

Original: = 1 , 2 , . .
Revised: = 1 , 2 , . .

Original: , K do 4:
Revised: , K do 4:

Original: Sample a task M k∼ P M
Revised: Sample a task M k

Original: [Equation]
Revised: [Equation]

Original: Optimize control policy:
Revised: Optimize control policy:

Original: [Equation]
Revised: [Equation]

Original: G
Revised: G.

Original: [Equation]
Revised: [Equation]

Original: Update default policy distribution:
Revised: Update default policy distribution:

Original: [Equation]
Revised: [Equation]

Original: (cid:63)k

Original: [Equation]

Original: π

Original: [Equation]

Original: (G.8) 7: end for
Revised: (G.8) 7: end for

Original: G. MDL-C WITH P ERSISTENT R EPLAY
Revised: G. MDL-C WITH P ERSISTENT R EPLAY

Original: Rather than rely on iid task draws to yield a bound on the expected regret under the task distribution, a more general formulation of MDL-C for sequential task learning is described in Algorithm 1.
Revised: Rather than rely on iid task draws to yield a bound on the expected regret under the task distribution, a more general formulation of MDL-C for sequential task learning is described in Algorithm 1.

Original: In this setting, the dataset of optimal agent-environment interactions is explicitly constructed by way of a replay buffer which persists across tasks and is used to train the default policy distribution.
Revised: In this setting, the dataset of optimal agent-environment interactions is explicitly constructed by way of a replay buffer which persists across tasks and is used to train the default policy distribution.

Original: This is much more directly in line with standard FTRL, and we can obtain the standard FTRL bound.
Revised: This is much more directly in line with standard FTRL, and we can obtain the standard FTRL bound.

Original: Proposition G.2 (Persistent Replay FTRL Regret; (Orabona, 2019), Corollary 7.9) .
Revised: Proposition G.2 (Persistent Replay FTRL Regret; (Orabona, 2019), Corollary 7.9) .

Original: Let tasks M k be independently drawn from P M at every round, and let them each be associated with a deterministic optimal policy
Revised: Let tasks M k be independently drawn from P M at every round, and let them each be associated with a deterministic optimal policy

Original: π (cid:63)k : S → A .
Revised: π ⋆k : S → A .

Original: We make the following mild assumptions: i) π w ( a (cid:63) | s ) ≥ (cid:15) > 0 ∀ s ∈ S , where a (cid:63) = 
Revised: We make the following mild assumptions: i) π w ( a ⋆ | s ) ≥ ϵ > 0 ∀ s ∈ S , where a ⋆ = π

Original: π (cid:63)k ( s ) and (cid:15) is a constant. ii) min ν KL
Revised: ⋆k ( s ) and ϵ is a constant. ii) min ν KL

Original: [ ν ( · ) , p ( · )]
Revised: [ ν ( · ) , p ( · )]

Original: = 0 asymptotically as Var[ ν ] → ∞ .
Revised: = 0 asymptotically as Var[ ν ] → ∞ .

Original: Then with η k − [Equation] =
Revised: Then with η k − 1 =

Original: L √ k , Algorithm 1 guarantees
Revised: L √ k , Algorithm 1 guarantees

Original: [Equation]
Revised: [Equation]

Original: G.

Original: [Equation]

Original: where ¯ ν K = argmin ν ∈ N (cid:80) Kk =1 (cid:96) k ( ν ) .
Revised: where ¯ ν K = argmin ν ∈ N (cid:80) Kk =1 ℓ k ( ν ) .

Original: Proof.
Revised: Proof.

Original: This follows directly from the arguments made in the proof of Proposition G.1.
Revised: This follows directly from the arguments made in the proof of Proposition G.1.

Original: As before, this result can be used to obtain a performance bound for the control policy.
Revised: As before, this result can be used to obtain a performance bound for the control policy.

Original: Proposition G.3 (Control Policy Sample Complexity for MDL-C with Persistent Replay) .
Revised: Proposition G.3 (Control Policy Sample Complexity for MDL-C with Persistent Replay) .

Original: Under the setting described in Proposition G.2, denote by T k the number of iterations to reach (cid:15) -error for M k in the sense that min t ≤ T k { V π (cid:63)k − V ( t ) } ≤ (cid:15). In a ﬁnite MDP, from any initial θ (0) , and following gradient ascent, E M k ∼P M [ T k ] satisﬁes:
Revised: Under the setting described in Proposition G.2, denote by T k the number of iterations to reach ϵ -error for M k in the sense that min t ≤ T k { V π ⋆k − V ( t ) } ≤ ϵ. In a finite MDP, from any initial θ (0) , and following gradient ascent, E M k ∼P M [ T k ] satisfies:

Original: [Equation]
Revised: [Equation]

Revised: ∞ 

Original:  where α k ( s ) :
Revised:  , where α k ( s ) :

Original: =
Revised: =

Original: E w ∼ ν d TV ( π (cid:63)k ( ·| s ) , π w ( ·| s )) ≤ (cid:113) 12 G ( K ) ,
Revised: E w ∼ ν d TV ( π ⋆k ( ·| s ) , π w ( ·| s )) ≤ (cid:113) 12 G ( K ) ,

Original: G ( K ) :
Revised: G ( K ) :=

Original: = (cid:96) K

Original:  (¯ ν K )
Revised: ℓ K (¯ ν K )

Original: +
Revised: +

Original: K − 1 (cid:88) k
Revised: K − 1 (cid:88) k

Original: = ( (cid:96) k 
Revised: = ( ℓ k [Equation]

Original: (¯ ν K ) − (cid:96) k
Revised: ¯ [Equation] K [Equation]

Original: ( ν k ))
Revised: k [Equation] k

Original:  + ( KL [¯ν, p ] + 1) L
Revised: [Equation] + ( KL [¯

Original: [Equation]
Revised: [Equation]

Original: , κ α k A ( s ) = 2 |A| (1 − α ( s )) 2 |A| (1 − α ( s )) − 1 , and µ is a probability measure over S such that µ ( s ) > 0 ∀ s ∈ S .
Revised: p ] + 1) L √ K, κ α k A ( s ) = 2 |A| (1 − α ( s )) 2 |A| (1 − α ( s )) − 1 , and µ is a probability measure over S such that µ ( s ) > 0 ∀ s ∈ S .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we select a single state s ∈ S , observing that the same analysis applies ∀ s ∈ S .
Revised: Without loss of generality, we select a single state s ∈ S , observing that the same analysis applies ∀ s ∈ S .

Original: For simplicity, we denote π ( ·| s ) by π .
Revised: For simplicity, we denote π ( ·| s ) by π .

Original: We start by multiplying each side of Eq.
Revised: We start by multiplying each side of Eq.

Original: (G.2) by K and rearranging: 
Revised: (G.2) by K and rearranging: K

Original: [Equation]
Revised: [Equation]

Original: We can multiply both sides by 1 / 2 and expand (cid:96) K ( ν K ) :
Revised: We can multiply both sides by 1 / 2 and expand ℓ K ( ν K ) :

Original: [Equation]
Revised: [Equation]

Original:  where ( i ) follows from the deﬁnition of the variance, ( ii ) follows from its non-negativity
Revised: ) where ( i ) follows from the definition of the variance, ( ii ) follows from its non-negativity

Original: [Equation]
Revised: [Equation]

Original: and ( iii ) follows from Pinsker’s inequality.
Revised: and ( iii ) follows from Pinsker’s inequality.

Original: We then have
Revised: We then have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Letting α K ( s )
Revised: Letting α K ( s )

Original: = (cid:113) 12 G ( K ) and applying Moskovitz et al.
Revised: = (cid:113) 12 G ( K ) and applying Moskovitz et al.

Original: (2022a)
Revised: (2022a)

Original: , Lemma 5.2 gives the desired result.
Revised: , Lemma 5.2 gives the desired result.

Original: G. C OMMENT ON I MPROVEMENT A CROSS T ASKS
Revised: G. C OMMENT ON I MPROVEMENT A CROSS T ASKS

Original: To gain intuition for these bounds, there are several important values of α ( s ) that we can consider.
Revised: To gain intuition for these bounds, there are several important values of α ( s ) that we can consider.

Original: First, as α ( s ) → 1 − 1 / |A| , which is the TV distance between a uniform default policy and a deterministic optimal policy, κ α A ( s ) → 2 .
Revised: First, as α ( s ) → 1 − 1 / |A| , which is the TV distance between a uniform default policy and a deterministic optimal policy, κ α A ( s ) → 2 .

Original: This is an important value because it’s the coefﬁcient obtained for log-barrier regularization—that is, when the default policy is uniform and encodes no information about the task distribution.
Revised: This is an important value because it’s the coefficient obtained for log-barrier regularization—that is, when the default policy is uniform and encodes no information about the task distribution.

Original: Next, as α ( s ) → 0 (that is, as the TV distance between the default policy and the optimal policy decreases), κ α A ( s ) → 2 |A| / (2 |A| − 1) < 2 for |A| > 1 .
Revised: Next, as α ( s ) → 0 (that is, as the TV distance between the default policy and the optimal policy decreases), κ α A ( s ) → 2 |A| / (2 |A| − 1) < 2 for |A| > 1 .

Original: So, we get faster as the distance between the default policy and the optimal policy decreases, as we would hope.
Revised: So, we get faster as the distance between the default policy and the optimal policy decreases, as we would hope.

Original: Another crucial point to note is that as |A| → ∞ in this case, κ
Revised: Another crucial point to note is that as |A| → ∞ in this case, κ

Original: α A ( s ) → 1 .
Revised: α A ( s ) → 1 .

Original: Finally, and importantly for MDL-C, as α ( s ) → 1 − 1 / 2 |A| from below, κ α A ( s ) → ∞ .
Revised: Finally, and importantly for MDL-C, as α ( s ) → 1 − 1 / 2 |A| from below, κ α A ( s ) → ∞ .

Original: In other words, a sufﬁciently bad default policy can preclude convergence entirely if it puts too much mass on a suboptimal action.
Revised: In other words, a sufficiently bad default policy can preclude convergence entirely if it puts too much mass on a suboptimal action.

Original: For an illustration of this phenomenon, see Moskovitz et al.
Revised: For an illustration of this phenomenon, see Moskovitz et al.

Original: (2022a)
Revised: (2022a)

Original: Figure 4.1.
Revised: Figure 4.1.

Original: Indeed, this is why our Proposition 4.1 is so useful–by effectively placing an upper bound on α ( s ) which shrinks as the number of tasks K increases, MDL-C’s default policy is guaranteed to a) avoid putting too much mass on a suboptimal action and thereby preclude or delay convergence for the control policy, and b) improve the rate as the default policy regret drops.
Revised: Indeed, this is why our Proposition 4.1 is so useful–by effectively placing an upper bound on α ( s ) which shrinks as the number of tasks K increases, MDL-C’s default policy is guaranteed to a) avoid putting too much mass on a suboptimal action and thereby preclude or delay convergence for the control policy, and b) improve the rate as the default policy regret drops.

Original: G. P ARALLEL T ASK S ETTING
Revised: G. P ARALLEL T ASK S ETTING

Original: An overview of MDL-C as applied in the parallel task setting is presented in Algorithm 3.
Revised: An overview of MDL-C as applied in the parallel task setting is presented in Algorithm 3.

Original: One important feature to note is the return threshold R (cid:63) .
Revised: One important feature to note is the return threshold R ⋆ .

Original: As a proxy for the control policy converging to π (cid:63)k , data are only added to the default policy replay buffer when a trajectory return is above this threshold performance (on DM control suite tasks, R (cid:63) corresponded to a test reward of at least 700).
Revised: As a proxy for the control policy converging to π ⋆k , data are only added to the default policy replay buffer when a trajectory return is above this threshold performance (on DM control suite tasks, R ⋆ corresponded to a test reward of at least 700).

Original: 1: require: task distribution P M , policy class Π 2: initialize: default policy distribution ν 1 ∈ N , control replay D 0 ← ∅ , default replay D φ 0 ← ∅ 3: initialize control policy parameters θ and default policy distribution parameters φ .
Revised: 1: require: task distribution P M , policy class Π 2: initialize: default policy distribution ν 1 ∈ N , control replay D 0 ← ∅ , default replay D ϕ 0 ← ∅ 3: initialize control policy parameters θ and default policy distribution parameters ϕ .

Original: 4: while not done do 5: for episodes k = 1 , 2 , . . .
Revised: 4: while not done do 5: for episodes k = 1 , 2 , . . .

Original: , K do 6:
Revised: , K do 6:

Original: Sample a task M k ∼ P M ( · ) with goal ID feature g k 7:
Revised: Sample a task M k ∼ P M ( · ) with goal ID feature g k 7:

Original: Collect trajectory τ =
Revised: Collect trajectory τ =

Original: (˜ s 0 , a 0 , r 0 , . . . ,
Revised: (˜ s 0 , a 0 , r 0 , . . . ,

Original: ˜ s H − 1 , a H − 1 , r H − 1 ) ∼ P π θ ( · ) , store experience
Revised: ˜ s H − 1 , a H − 1 , r H − 1 ) ∼ P π θ ( · ) , store experience

Original: D k ← D k − 1 ∪ { (˜ s h , a h , r h , ˜ s h +1 ) } H − 1 h =
Revised: D k ← D k − 1 ∪ { (˜ s h , a h , r h , ˜ s h +1 ) } H − 1 h =

Original: (G.13) where ˜ s h :=
Revised: (G.13) where ˜ s h :=

Original: ( s h , g k ) .
Revised: ( s h , g k ) .

Original: 8: if R ( τ ) ≥ R (cid:63) (i.e., π θ ≈ π (cid:63)k ) then 9:
Revised: 8: if R ( τ ) ≥ R ⋆ (i.e., π θ ≈ π ⋆k ) then 9:

Original: Add to default policy replay:
Revised: Add to default policy replay:

Original: D φk ← D φk − 1 ∪ { (˜ s h , π θ ( ·| ˜ s h ) } H − 1 h = (G.14)
Revised: D ϕk ← D ϕk − 1 ∪ { ( ˜ s h , π θ ( ·| ˜ s h ) } H − 1 h = (G.14)

Original: Note that, e.g., when π θ ( a | ˜ s ) =
Revised: Note that, e.g., when π θ ( a | ˜ s ) =

Original: N ( a ; µ (˜ s, g k ) , Σ(˜ s, g k )) is a Gaussian policy, µ (˜ s h , g k ) , Σ(˜ s h , g k ) are added to the replay with ˜ s h .
Revised: N ( a ; µ (˜ s, g k ) , Σ(˜ s, g k )) is a Gaussian policy, µ (˜ s h , g k ) , Σ(˜ s h , g k ) are added to the replay with ˜ s h .

Original: 10: end if 11: end for 12:
Revised: 10: end if 11: end for 12:

Original: Update Q -function(s) as in Haarnoja et al.
Revised: Update Q -function(s) as in Haarnoja et al.

Original: Update control policy:
Revised: Update control policy:

Original: [Equation]
Revised: [Equation]

Original: G [Equation] [Equation]
Revised: (G.15) 14:

Original: Update default policy distribution: 
Revised: Update default policy distribution: ϕ ←

Original: [Equation]
Revised: [Equation]

Original:  15: end while
Revised: KL [ ν ϕ ′ ( · ) , p ( · )] + E Unif D ϕk E w ∼ ν KL [ π θ ( ·| ˜ s h ) , π w ( ·| ˜ s h )] (G.16) 15: end while

Original: We leave more in-depth theoretical analysis of this setting to future work, but note that as the task experience is interleaved, ¯ π w = E ν π w will converge to the prior-weighted KL barycenter.
Revised: We leave more in-depth theoretical analysis of this setting to future work, but note that as the task experience is interleaved, ¯ π w = E ν π w will converge to the prior-weighted KL barycenter.

Original: If, in expectation, this distribution is a TV distance of less than 1 − 1 / |A| from π (cid:63)k , then the control policy will converge faster than for log-barrier regularization (Moskovitz et al., 2022a).
Revised: If, in expectation, this distribution is a TV distance of less than 1 − 1 / |A| from π ⋆k , then the control policy will converge faster than for log-barrier regularization (Moskovitz et al., 2022a).

Original: H A DDITIONAL E XPERIMENTAL D ETAILS
Revised: H A DDITIONAL E XPERIMENTAL D ETAILS

Original: Below, we describe experimental details for the two environment domains in the paper.
Revised: Below, we describe experimental details for the two environment domains in the paper.

Original: H. F OUR R OOMS
Revised: H. F OUR R OOMS

Original: As input, the agent receives a 16-dimensional vector containing the index of the current state, a ﬂattened 3 × 3 local view of its surrounding environment, its previous action taken encoded as a 4-dimensional one-hot vector, the reward on the previous timestep, and a feature indicating the goal state index.
Revised: As input, the agent receives a 16-dimensional vector containing the index of the current state, a flattened 3 × 3 local view of its surrounding environment, its previous action taken encoded as a 4-dimensional one-hot vector, the reward on the previous timestep, and a feature indicating the goal state index.

Original: The base learning algorithm in all cases is advantage actor critic (A2C; (Mnih et al., 2016)).
Revised: The base learning algorithm in all cases is advantage actor critic (A2C; (Mnih et al., 2016)).

Original: Environment The F OUR R OOMS experiments are set in an 11 × 11 gridworld.
Revised: Environment The F OUR R OOMS experiments are set in an 11 × 11 gridworld.

Original: The actions available to the agent are the four cardinal directions, up , down , left , and right , and transitions are deterministic.
Revised: The actions available to the agent are the four cardinal directions, up , down , left , and right , and transitions are deterministic.

Original: In both F OUR R OOMS experiments, the agent can begin an episode anywhere in the environment (sampled uniformly at random), and a single location with reward r = 50 is sampled at the beginning of each episode from a set of possible goal states which varies depending on the experiment and the current phase.
Revised: In both F OUR R OOMS experiments, the agent can begin an episode anywhere in the environment (sampled uniformly at random), and a single location with reward r = 50 is sampled at the beginning of each episode from a set of possible goal states which varies depending on the experiment and the current phase.

Original: A reward of r = − 1 is given if the agent contacts the walls.
Revised: A reward of r = − 1 is given if the agent contacts the walls.

Original: All other states give a reward of zero.
Revised: All other states give a reward of zero.

Original: Episodes end when either a time (number of timesteps) limit is reached or the agent reaches the goal state.
Revised: Episodes end when either a time (number of timesteps) limit is reached or the agent reaches the goal state.

Original: Observations were 16-dimensional vectors consisting of the current state index (1d), ﬂattened 3 × 3 local window surrounding the agent (includes walls, but not goals), a one-hot encoding of the action on the previous timestep (4d), the reward on the previous timestep (1d), and the state index of the current goal (1d).
Revised: Observations were 16-dimensional vectors consisting of the current state index (1d), flattened 3 × 3 local window surrounding the agent (includes walls, but not goals), a one-hot encoding of the action on the previous timestep (4d), the reward on the previous timestep (1d), and the state index of the current goal (1d).

Original: In the “goal generalization” experiment, goals may be sampled anywhere in either the top left or bottom right rooms in the ﬁrst phase and either the top right or bottom left rooms in the second phase.
Revised: In the “goal generalization” experiment, goals may be sampled anywhere in either the top left or bottom right rooms in the first phase and either the top right or bottom left rooms in the second phase.

Original: Each phase comprises 20,000 episodes, and in each phase, the agent may start each episode anywhere in the environment.
Revised: Each phase comprises 20,000 episodes, and in each phase, the agent may start each episode anywhere in the environment.

Original: In the ﬁrst phase, the agent was allowed 100 steps per episode, and in the second phase 25 steps.
Revised: In the first phase, the agent was allowed 100 steps per episode, and in the second phase 25 steps.

Original: In the “contingency change” experiment, the possible reward states in each phase were the top left state and bottom right state.
Revised: In the “contingency change” experiment, the possible reward states in each phase were the top left state and bottom right state.

Original: In the second phase of training, however, the semantics of the goal feature change from indicating the location of the reward to the location where it is absent.
Revised: In the second phase of training, however, the semantics of the goal feature change from indicating the location of the reward to the location where it is absent.

Original: Each phase consisted of 8, episodes with maximum length 100 timesteps.
Revised: Each phase consisted of 8, episodes with maximum length 100 timesteps.

Original: Results are averaged over 10 random seeds.
Revised: Results are averaged over 10 random seeds.

Original: Agents All agents were trained on-policy with advantage actor-critic (Mnih et al., 2016).
Revised: Agents All agents were trained on-policy with advantage actor-critic (Mnih et al., 2016).

Original: The architecture was a single-layer LSTM (Hochreiter and Schmidhuber, 1997) with 128 hidden units.
Revised: The architecture was a single-layer LSTM (Hochreiter and Schmidhuber, 1997) with 128 hidden units.

Original: To produce the feature sensitivity plots in Fig.
Revised: To produce the feature sensitivity plots in Fig.

Original: 5.1c, a gating function was added to the input layer of the network:
Revised: 5.1c, a gating function was added to the input layer of the network:

Original: [Equation]
Revised: [Equation]

Original: where o h is the current observation, σ ( · ) was the sigmoid funcion, b ∈ R is a constant (set to b
Revised: where o h is the current observation, σ ( · ) was the sigmoid funcion, b ∈ R is a constant (set to b

Original: =in all experiments), x
Revised: =in all experiments), x

Original: h ∈ R
Revised: h ∈ R

Original: d is the ﬁlter layer output, and κ ∈ R d is a parameter trained using backpropagation.
Revised: d is the filter layer output, and κ ∈ R d is a parameter trained using backpropagation.

Original: In this way, as κ d → ∞ , σ ( bκ d ) → 1 , allowing input feature o h , d through the gate.
Revised: In this way, as κ d → ∞ , σ ( bκ d ) → 1 , allowing input feature o h , d through the gate.

Original: As κ d → −∞ , the gate is shut.
Revised: As κ d → −∞ , the gate is shut.

Original: The plots in Fig.
Revised: The plots in Fig.

Original: 5.1c track σ ( bκ d ) over the course of training.
Revised: 5.1c track σ ( bκ d ) over the course of training.

Original: The baseline agent objective functions are as follows:
Revised: The baseline agent objective functions are as follows:

Original: [Equation]
Revised: [Equation]

Original: In all cases α = 0 . 1 , β = 1 . 0 , and learning rates for all agents were set to 0 . 0007 .
Revised: In all cases α = 0 . 1 , β = 1 . 0 , and learning rates for all agents were set to 0 . 0007 .

Original: Agents were optimized with Adam (Kingma and Ba, 2014).
Revised: Agents were optimized with Adam (Kingma and Ba, 2014).

Original: Agent control policies were reset after phase 1.
Revised: Agent control policies were reset after phase 1.

Original: H. D EEP M IND C ONTROL S UITE
Revised: H. D EEP M IND C ONTROL S UITE

Original: Environments/Task Settings We use the walker and cartpole environments from the DeepMind Control Suite (Tassa et al., 2018).
Revised: Environments/Task Settings We use the walker and cartpole environments from the DeepMind Control Suite (Tassa et al., 2018).

Original: We consider two multitask settings: sequential tasks and parallel tasks.
Revised: We consider two multitask settings: sequential tasks and parallel tasks.

Original: All results are averaged over 10 random seeds, and agents are trained for 500k timesteps.
Revised: All results are averaged over 10 random seeds, and agents are trained for 500k timesteps.

Original: In the sequential task setting, tasks are sampled one at a time without replacement and solved by the agent.
Revised: In the sequential task setting, tasks are sampled one at a time without replacement and solved by the agent.

Original: The control policy is reset after each task, but the default policy is preserved.
Revised: The control policy is reset after each task, but the default policy is preserved.

Original: For methods which have a default policy which can be preserved, performance on task k is averaged over runs with all possible previous tasks in all possible orders.
Revised: For methods which have a default policy which can be preserved, performance on task k is averaged over runs with all possible previous tasks in all possible orders.

Original: For example, when walker-run is the third task, performance is averaged over previous tasks being stand then walk and walk then stand .
Revised: For example, when walker-run is the third task, performance is averaged over previous tasks being stand then walk and walk then stand .

Original: In the parallel task setting, a different task is sampled randomly at the start of each episode, and a one-hot task ID vector is appended to the state observation.
Revised: In the parallel task setting, a different task is sampled randomly at the start of each episode, and a one-hot task ID vector is appended to the state observation.

Original: Learning was done directly from states, not from pixels.
Revised: Learning was done directly from states, not from pixels.

Original: Agents The base agent in all cases was SAC with automatic temperature tuning, following Haarnoja et al.
Revised: Agents The base agent in all cases was SAC with automatic temperature tuning, following Haarnoja et al.

Original: Standard SAC seeks to optimize the maximum-entropy RL objective:
Revised: Standard SAC seeks to optimize the maximum-entropy RL objective:

Original: [Equation]
Revised: [Equation]

Original: Effectively, then, SAC uses a uniform default policy.
Revised: Effectively, then, SAC uses a uniform default policy.

Original: The RPO algorithms with learned default policies replace KL [ π ( ·| s ) , Unif A ] with KL [ π ( ·| s ) , π w ( ·| s )] (or KL [ π w ( ·| s ) , π ( ·| s )] ).
Revised: The RPO algorithms with learned default policies replace KL [ π ( ·| s ) , Unif A ] with KL [ π ( ·| s ) , π w ( ·| s )] (or KL [ π w ( ·| s ) , π ( ·| s )] ).

Original: As MDL-C,
Revised: As MDL-C,

Original: RPO, and TVPO require that the control policy approximate the optimal policy before being used to generated the a learning signal for the default policy, in the sequential setting, the default policy is updated only after halfway through training.
Revised: RPO, and TVPO require that the control policy approximate the optimal policy before being used to generated the a learning signal for the default policy, in the sequential setting, the default policy is updated only after halfway through training.

Original: Because variational dropout can cause the network to over-sparsify (and not learn the learn adequately) if turned on too early in training, we follow the strategy of Molchanov et al.
Revised: Because variational dropout can cause the network to over-sparsify (and not learn the learn adequately) if turned on too early in training, we follow the strategy of Molchanov et al.

Original: (2017), linearly ramping up a coefﬁcient β on the variational dropout KL from 0 to 1 starting from 70% through training to 80% through training.
Revised: (2017), linearly ramping up a coefficient β on the variational dropout KL from 0 to 1 starting from 70% through training to 80% through training.

Original: Note that M ANUAL IA is not applicable to the sequential task setting, as there is no explicit goal feature.
Revised: Note that M ANUAL IA is not applicable to the sequential task setting, as there is no explicit goal feature.

Original: In the sequential task setting, we took inspiration from Haarnoja et al.
Revised: In the sequential task setting, we took inspiration from Haarnoja et al.

Original: (2018) and Abdolmaleki et al.
Revised: (2018) and Abdolmaleki et al.

Original: and reframed the soft KL penalty for methods with learned default policies as a constraint, i.e.,
Revised: and reframed the soft KL penalty for methods with learned default policies as a constraint, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where ε > 0 was a target KL divergence.
Revised: where ε > 0 was a target KL divergence.

Original: Under this formulation, α is treated as a dual variable via
Revised: Under this formulation, α is treated as a dual variable via

Original: Lagrangian relaxation and optimized with the following objective:
Revised: Lagrangian relaxation and optimized with the following objective:

Original: [Equation]
Revised: [Equation]

Original: In the parallel task setting, we convert the base SAC agent into the “multitask” variant used by Yu et al.
Revised: In the parallel task setting, we convert the base SAC agent into the “multitask” variant used by Yu et al.

Original: (2019), in which the agent learns a vector of temperature parameters [ α 1 , . .
Revised: (2019), in which the agent learns a vector of temperature parameters [ α 1 , . .

Original: . , α K ] , one for each task.
Revised: . , α K ] , one for each task.

Original: In this setting, we found it more effective to set α to a constant value.
Revised: In this setting, we found it more effective to set α to a constant value.

Original: Test performance was computed by averaging performance across all K tasks presented to the agent.
Revised: Test performance was computed by averaging performance across all K tasks presented to the agent.

Original: The baseline agent objectives are as in Eq.
Revised: The baseline agent objectives are as in Eq.

Original: (H.2), and the Distral objective is given by
Revised: (H.2), and the Distral objective is given by

Original: [Equation]
Revised: [Equation]

Original: TVPO is trained in the same way as RPO, with the difference being that the default policy objective is to predict the control policy action, rather than a distillation objective.
Revised: TVPO is trained in the same way as RPO, with the difference being that the default policy objective is to predict the control policy action, rather than a distillation objective.

Original: Hyperparameters shared by all agents can be viewed in Table 2.
Revised: Hyperparameters shared by all agents can be viewed in Table 2.

Original: As a note on performance, Distral performs very strongly in the parallel task setting, with overall performance slightly worse than MDL-C on Walker and virtually the same on Cartpole.
Revised: As a note on performance, Distral performs very strongly in the parallel task setting, with overall performance slightly worse than MDL-C on Walker and virtually the same on Cartpole.

Original: However, the gap is signiﬁcantly greater in the sequential setting, particularly on Walker.
Revised: However, the gap is significantly greater in the sequential setting, particularly on Walker.

Original: We hypothesize that this is due to the fact that by regularizing the control policy to be close to the default policy, but also encouraging the control policy to have high entropy (rather than regularizing the default policy as MDL-C does), Distral can in effect provide a conﬂicting objective to the control policy when strong structure is present.
Revised: We hypothesize that this is due to the fact that by regularizing the control policy to be close to the default policy, but also encouraging the control policy to have high entropy (rather than regularizing the default policy as MDL-C does), Distral can in effect provide a conflicting objective to the control policy when strong structure is present.

Original: In particular, on Walker, the optimal policies for each task have signiﬁcant overlap, and so by encouraging high entropy in the control policy even on the third task, Distral negates the effect of an informative default policy.
Revised: In particular, on Walker, the optimal policies for each task have significant overlap, and so by encouraging high entropy in the control policy even on the third task, Distral negates the effect of an informative default policy.

Original: As evidence, both RPO and TVPO, which only regularize the control policy to be close to the default policy, perform signiﬁcantly more strongly on
Revised: As evidence, both RPO and TVPO, which only regularize the control policy to be close to the default policy, perform significantly more strongly on

Original: Walker in the sequential setting.
Revised: Walker in the sequential setting

Original: [Table]
Revised: [Table]

Original: [ − 20 , 2]
Revised: [ − [Table]

Original: I A DDITIONAL E XPERIMENTAL R ESULTS
Revised: I A DDITIONAL E XPERIMENTAL R ESULTS

Original: I.
Revised: I.

Original: F OUR R OOMS
Revised: F OUR R OOMS

Original: I.
Revised: I.

Original: D EEP M IND C ONTROL S UITE
Revised: D EEP M IND C ONTROL S UITE

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Figure]
Revised: [Table] [Table]

Original: where ¯ w = E ν w for MDL-C, averaged over all possible goal states.
Revised: where ¯ w = E ν w for MDL-C, averaged over all possible goal states.

Original: The RPO default policy nearly perfectly matches the control policy, while the MDL-C default policy diverges most strongly from the control policy at the doorways.
Revised: The RPO default policy nearly perfectly matches the control policy, while the MDL-C default policy diverges most strongly from the control policy at the doorways.

Original: This is because the direction chosen by the policy in the doorways is highly goal-dependent.
Revised: This is because the direction chosen by the policy in the doorways is highly goal-dependent.

Original: Because the MDL-C default policy learns to ignore the goal feature, it’s roughly uniform in the doorways, whereas the control policy is highly deterministic, having access tothe goal feature
Revised: Because the MDL-C default policy learns to ignore the goal feature, it’s roughly uniform in the doorways, whereas the control policy is highly deterministic, having access to feature

Original: [Figure]
Revised: [Figure]

Original:  w
Revised: u r e w

Original: [Figure]
Revised: [Figure]

Revised: walker (parallel

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: , the performance gains of MDL-C can’t solely be ascribed to effective information asymmetry.
Revised: of MDL-C can’t solely be ascribed to effective information asymmetry.

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]