Original: A CMDP- WITHIN - ONLINE FRAMEWORK FOR M ETAS AFE R EINFORCEMENT L EARNING
Revised: A CMDP- WITHIN - ONLINE FRAMEWORK FOR M ETAS AFE R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience.
Revised: Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience.

Original: However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings.
Revised: However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings.

Original: In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting.
Revised: In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting.

Original: We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the taskaveraged optimality gap and constraint satisfaction improve with task-similarity in a static environment, or task-relatedness in a dynamic environment.
Revised: We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the taskaveraged optimality gap and constraint satisfaction improve with task-similarity in a static environment, or task-relatedness in a dynamic environment.

Original: Several technical challenges arise when making this framework practical.
Revised: Several technical challenges arise when making this framework practical.

Original: To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by offpolicy stationary distribution corrections.
Revised: To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by offpolicy stationary distribution corrections.

Original: Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle.
Revised: Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle.

Original: Finally, experiments are conducted to demonstrate the effectiveness of our approach.
Revised: Finally, experiments are conducted to demonstrate the effectiveness of our approach.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The field of meta-reinforcement learning (meta-RL) has recently evolved as one of the promising directions that enables reinforcement learning (RL) agents to learn quickly in dynamically changing environments Finn et al. ; Mitchell et al.
Revised: The field of meta-reinforcement learning (meta-RL) has recently evolved as one of the promising directions that enables reinforcement learning (RL) agents to learn quickly in dynamically changing environments (Finn et al., 2017; Mitchell et al., 2021;

Original: ; Zintgraf et al. ; Sodhani et al.
Revised: Zintgraf et al., 2021; Sodhani et al., 2021;

Original: Hospedales et al.
Revised: Hospedales et al., 2020).

Original: Many real-world applications, nevertheless, have safety constraints that should be rarely violated, which existing works do not fully address.
Revised: Many real-world applications, nevertheless, have safety constraints that should be rarely violated, which existing works do not fully address.

Original: These safe-RL problems are often modeled as constrained Markov decision processes (CMDPs), where the agent aims to maximize the value function while satisfying given constraints on the trajectory Altman (1999).
Revised: These safe-RL problems are often modeled as constrained Markov decision processes (CMDPs), where the agent aims to maximize the value function while satisfying given constraints on the trajectory (Altman, 1999).

Original: However, unlike meta-learning, CMDP algorithms are not designed to generalize efficiently over unseen tasks Chow et al.
Revised: However, unlike meta-learning, CMDP algorithms are not designed to generalize efficiently over unseen tasks (Chow et al., 2018;

Original: ; Paternain et al. ; Efroni et al. Ding et al. (2021a)
Revised: Paternain et al., 2022; Efroni et al., 2020; Ding et al., 2021a; Ding & Lavaei, 2022;

Original: ; Ding & Lavaei (2022); Ying et al. (2022); Yu et al. ; Xu et al. ; Chen et al. (2021b);
Revised: Ying et al., 2022; Yu et al., 2019; Xu et al., 2021; Chen et al., 2021b; Bura et al., 2021).

Original: Bura et al.

Original: Take the rehabilitation robotics example, where a certain torque motor control needs to be designed to aid a person in walking.
Revised: Take the rehabilitation robotics example, where a certain torque motor control needs to be designed to aid a person in walking.

Original: The required torque profiles would vary for each person, hence using a safe-RL algorithm for every task/person can take a substantial amount of interaction time.
Revised: The required torque profiles would vary for each person, hence using a safe-RL algorithm for every task/person can take a substantial amount of interaction time.

Original: In this paper, we study how meta-learning can be principally designed to help safe-RL algorithms adapt more quickly while satisfying critical constraints.
Revised: In this paper, we study how meta-learning can be principally designed to help safe-RL algorithms adapt more quickly while satisfying critical constraints.

Original: There are several unique challenges involved in meta-learning for the CMDP settings.
Revised: There are several unique challenges involved in meta-learning for the CMDP settings.

Original: First, there are multiple losses incurred at each time step, i.e., reward and constraints.
Revised: First, there are multiple losses incurred at each time step, i.e., reward and constraints.

Original: Since these functions are typically nonconvex and coupled through dynamics, adapting existing theories developed for stylized settings such as online convex optimization Hazan et al. (2016); Shalev-Shwartz et al. (2012) is not straightforward.
Revised: Since these functions are typically nonconvex and coupled through dynamics, adapting existing theories developed for stylized settings such as online convex optimization (Hazan et al., 2016; Shalev-Shwartz et al., 2012) is not straightforward.

Original: Second, it is unrealistic to assume the computation of a globally optimal policy or the corresponding state visitation distribution for CMDPs (unlike online learning Shalev-Shwartz et al. (2012)); even the characterization of the distance of a suboptimal policy to the set of global policies has been limited to some restricted settings, i.e., algorithm-dependent (Mei et al., 2020, Lemmasand 15, non-uniform Łojasiewicz for policy gradient).
Revised: Second, it is unrealistic to assume the computation of a globally optimal policy or the corresponding state visitation distribution for CMDPs (unlike online learning (Shalev-Shwartz et al., 2012)); even the characterization of the distance of a suboptimal policy to the set of global policies has been limited to some restricted settings, i.e., algorithm-dependent (Mei et al., 2020, Lemmasand 15, non-uniform Łojasiewicz for policy gradient).

Original: Thus, classical online learning algorithms that assume exact or unbiased estimator of the loss function do not apply Shalev-Shwartz et al.
Revised: Thus, classical online learning algorithms that assume exact or unbiased estimator of the loss function do not apply (Shalev-Shwartz et al., 2012;

Original: Khodak et al.
Revised: Khodak et al., 2019).

Original: Above all, there is an interplay among nonconvexity and stochasticity nature of the optimization problem, as well as algorithm and generalization considerations, posing significant complexity to leverage inter-task dependency and within-task geometries Denevi et al. Balcan et al.
Revised: Above all, there is an interplay among nonconvexity and stochasticity nature of the optimization problem, as well as algorithm and generalization considerations, posing significant complexity to leverage inter-task dependency and within-task geometries (Denevi et al., 2019; Balcan et al., 2019; 2021).

Original: To this end we propose a provably low-regret online learning framework that extends the current meta-learning algorithms to the safe-RL settings.
Revised: To this end we propose a provably low-regret online learning framework that extends the current meta-learning algorithms to the safe-RL settings.

Original: In view of the aforementioned challenges, our main contributions to the meta-learning and safe-RL literature are as follows:
Revised: In view of the aforementioned challenges, our main contributions to the meta-learning and safe-RL literature are as follows:

Original: Inexact CMDP-within-online framework: We propose a novel CMDP-within-online framework where the within-task is CMDP and the meta learner aims to learn the meta-initialization and learning rate.
Revised: Inexact CMDP-within-online framework: We propose a novel CMDP-within-online framework where the within-task is CMDP and the meta learner aims to learn the meta-initialization and learning rate.

Original: In our framework, the meta learner only requires the inexact optimal policies for each within-task CMDP and the approximate state visitation distributions estimated using collected offline trajectories to construct the upper bounds on the suboptimality gap and constraint violations.
Revised: In our framework, the meta learner only requires the inexact optimal policies for each within-task CMDP and the approximate state visitation distributions estimated using collected offline trajectories to construct the upper bounds on the suboptimality gap and constraint violations.

Original: An upper bound on these estimation errors is established in Theorem 3.1. 2.
Revised: An upper bound on these estimation errors is established in Theorem 3.1. 2.

Original: Task-averaged regret in terms of empirical task-similarity
Revised: Task-averaged regret in terms of empirical task-similarity

Original: : We show that the task-averaged regrets for optimality gap (TAOG) and constraint violations (TACV) diminish with respect to both the number of steps for the within-task algorithm M and the number of tasks T .
Revised: : We show that the task-averaged regrets for optimality gap (TAOG) and constraint violations (TACV) diminish with respect to both the number of steps for the within-task algorithm M and the number of tasks T .

Original: Specifically, we establish the order of O (cid:16) 1 √ M (cid:113) E T √ T + ˆ D ∗ 2 (cid:17) , where E T is the total inexactness in online learning and ˆ D ∗ is the empirical task-similarity (Theorem 3.2).
Revised: Specifically, we establish the order of O (cid:16) 1 √ M (cid:113) E T √ T + ˆ D ∗ 2 (cid:17) , where E T is the total inexactness in online learning and ˆ D ∗ is the empirical task-similarity (Theorem 3.2).

Original: Adapting to a dynamic environment: We consider adapting the learning rates for each task to a dynamic environment.
Revised: Adapting to a dynamic environment: We consider adapting the learning rates for each task to a dynamic environment.

Original: An improved rate of O (cid:16)
Revised: An improved rate of O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: for TAOG and TACV are shown, where ˆ V ψ is the empirical task-relatedness with respect to a sequence of changing comparator policies { ψ ∗ t }
Revised: for TAOG and TACV are shown, where ˆ V ψ is the empirical task-relatedness with respect to a sequence of changing comparator policies { ψ ∗ t }

Original: Tt =

Revised: Tt =

Original: (Corollary 1).
Revised: (Corollary 1).

Original: Incorporating
Revised: Incorporating

Original: all these components makes our Meta-safe-RL (Meta-SRL) approach highly practical and theoretically appealing for potential adaption to different RL settings.
Revised: all these components makes our Meta-safe-RL (Meta-SRL) approach highly practical and theoretically appealing for potential adaption to different RL settings.

Original: Due to the space restrictions, the related work can be found in the Appendix.
Revised: Due to the space restrictions, the related work can be found in the Appendix.

Original:  CMDP- WITHIN - ONLINE FRAMEWORK
Revised:  CMDP- WITHIN - ONLINE FRAMEWORK

Original: In this section, we introduce the CMDP-within-online framework for the Meta-SRL problems.
Revised: In this section, we introduce the CMDP-within-online framework for the Meta-SRL problems.

Original: In this framework, a within-task algorithm (such as CRPO Xu et al. for some CMDP task t ∈ [ T ] is encapsulated in an online learning algorithm
Revised: In this framework, a within-task algorithm (such as CRPO (Xu et al., 2021)) for some CMDP task t ∈ [ T ] is encapsulated in an online learning algorithm

Original: (meta-learning algorithm), which decides upon a sequence of initialization policy ϕ t and learning rate α t > 0 for each within-task algorithm.
Revised: (meta-learning algorithm), which decides upon a sequence of initialization policy ϕ t and learning rate α t > 0 for each within-task algorithm.

Original: The goal of the meta-learning algorithm is to minimize some proper notion of task-averaged performance regret to facilitate provably efficient adaptation to a new task.
Revised: The goal of the meta-learning algorithm is to minimize some proper notion of task-averaged performance regret to facilitate provably efficient adaptation to a new task.

Original: 2. CMDP AND THE PRIMAL APPROACH
Revised: 2. CMDP AND THE PRIMAL APPROACH

Original: Model.
Revised: Model.

Original: For each task t ∈ [ T ] , a CMDP M t is defined by the state space S , the action space A , the discount factor γ , the initial state distribution over the state-space ρ t , the transition kernel P t ( s ′ | s, a ) :
Revised: For each task t ∈ [ T ] , a CMDP M t is defined by the state space S , the action space A , the discount factor γ , the initial state distribution over the state-space ρ t , the transition kernel P t ( s ′ | s, a ) :

Original: S ×
Revised: S ×

Original: A → S , the reward functions c t, 0 : S ×
Revised: A → S , the reward functions c t, 0 : S ×

Original: A → [0 , 1] and cost functions c t,i : S ×
Revised: A → [0 , 1] and cost functions c t,i : S ×

Original: A → [0 , 1] for i = 1 , ..., p .
Revised: A → [0 , 1] for i = 1 , ..., p .

Original: The actions are chosen according to a stochastic policy π t :
Revised: The actions are chosen according to a stochastic policy π t :

Original: S → ∆( A ) where ∆( A ) is the simplex over the action space.
Revised: S → ∆( A ) where ∆( A ) is the simplex over the action space.

Original: We use ∆(
Revised: We use ∆(

Original: A ) |S| to denote the simplex over all states S .
Revised: A ) |S| to denote the simplex over all states S .

Original: The initial policy for task t is denoted as π t, 0 .
Revised: The initial policy for task t is denoted as π t, 0 .

Original: The discounted state visitation distribution of a policy π is defined as ν πt,s 0 ( s ) :
Revised: The discounted state visitation distribution of a policy π is defined as ν πt,s 0 ( s ) :

Original: = (1 − γ ) (cid:80) ∞ m
Revised: = (1 − γ ) (cid:80) ∞ m

Original: =

Revised: =

Original: γ m P t ( s m = s | π, s 0 ) and we write ν ∗ t ( s ) :
Revised: γ m P t ( s m = s | π, s 0 ) and we write ν ∗ t ( s ) :

Original: =
Revised: =

Original: E s 0 ∼ ρ t (cid:2)
Revised: E s 0 ∼ ρ t (cid:2)

Original: ν π ∗ t,s 0 ( s ) (cid:3) as the visitation distribution when the initial state follows ρ t at task t . 
Revised: ν π ∗ t,s 0 ( s ) (cid:3) as the visitation distribution when the initial state follows ρ t at task t . We denote π ∗ t as an optimal policy for task t , and ν ∗ t ( s ) : = E s 0 ∼ ρ t (cid:104)

Revised: ν

Revised: π ∗ t t,s 0 ( s ) (cid:105) is the corresponding state visitation distribution induced by policy π ∗ t when the initial state s 0 is sampled from initial state distribution ρ t at task t .

Original: Policy parametrization.
Revised: Policy parametrization.

Original: We consider the softmax parametrization.
Revised: We consider the softmax parametrization.

Original: For any θ ∈ R |S|×|A| , the corresponding softmax policy π θ is defined as π θ ( a | s ) :
Revised: For any θ ∈ R |S|×|A| , the corresponding softmax policy π θ is defined as π θ ( a | s ) :

Original: =
Revised: =

Original: exp( θ ( s,a )) (cid:80) a ′ ∈A exp( θ ( s,a ′ )) , ∀ ( s, a ) ∈ S × A .
Revised: exp( θ ( s,a )) (cid:80) a ′∈A exp( θ ( s,a ′ )) , ∀ ( s, a ) ∈ S × A .

Original: We neglect the dependence on θ to alleviate the notational burden.
Revised: We neglect the dependence on θ to alleviate the notational burden.

Original: Value function.
Revised: Value function.

Original: For task t and a policy π , we define the state-value function as V it,π ( s ) =
Revised: For task t and a policy π , we define the state-value function as V it,π ( s ) =

Original: E t [ (cid:80) ∞ m
Revised: E t [ (cid:80) ∞ m

Original: =

Revised: =

Original: γ m c t,i ( s m , a m , s m +1 ) | s 

Revised: γ m c t,i ( s m , a m , s m +1 ) | s

Original: = s, π ] and the action-value function as Q it,π ( s, a ) =
Revised: = s, π ] and the action-value function as Q it,π ( s, a ) =

Original: E t [ (cid:80) ∞ m
Revised: E t [ (cid:80) ∞ m

Original: =

Revised: =

Original: γ m c t,i ( s m , a m , s m +1 ) | s 

Revised: γ m c t,i ( s m , a m , s m +1 ) | s

Original: = s, a 0 = a, π ] , where m denotes the time steps.
Revised: = s, a 0 = a, π ] , where m denotes the time steps.

Original: Furthermore, the expected total reward/cost functions are J t,i ( π ) =
Revised: Furthermore, the expected total reward/cost functions are J t,i ( π ) =

Original: E ρ t (cid:2) V it,π ( s ) (cid:3) =
Revised: E ρ t (cid:2) V it,π ( s ) (cid:3) =

Original: E ρ t · π (cid:2) Q it,π ( s, a ) (cid:3) .
Revised: E ρ t · π (cid:2) Q it,π ( s, a ) (cid:3) .

Original: CMDP.
Revised: CMDP.

Original: In each task t , the goal of the agent is to solve the following CMDP problem
Revised: In each task t , the goal of the agent is to solve the following CMDP problem

Original: [Equation]
Revised: [Equation]

Original: where d t,i is a fixed limit on the expected total cost J t,i for task t and constraint i (among a total of p constraints).
Revised: where d t,i is a fixed limit on the expected total cost J t,i for task t and constraint i (among a total of p constraints).

Original: We denote the optimal solution of 1 for the task t as π ∗ t , which belongs to an optimal solution set Π ∗ t for each task t .
Revised: We denote the optimal solution of 1 for the task t as π ∗ t , which belongs to an optimal solution set Π ∗ t for each task t .

Original: Primal approach.
Revised: Primal approach.

Original: In this work, we focus on the primal approach, CRPO Xu et al. (2021), as an exemplary algorithm with guarantees for a single-task CMDP. 
Revised: In this work, we focus on the primal approach, CRPO (Xu et al., 2021), as an exemplary algorithm with guarantees for a single-task CMDP. CRPO is a primal-based online

Revised: CMDP algorithm, which performs policy optimization (natural gradient ascent on the reward) when constraints are not violated, or constraint minimization (natural gradient descent on the constraint function) for one of the violated constraints.

Revised: For more details about the CRPO algorithm, choice of parameters, and the convergence analysis for a single task, we refer the reader to Appendices B and

Original:  Specifically, with softmax parametrization and carefully chosen parameters, the suboptimality gap and constraint violation for task t are bounded as follows (if the exact action-value function { Q it,π } pi =0 are available for all π ) 1 :
Revised: D. Specifically, with softmax parametrization and carefully chosen parameters, the suboptimality gap and constraint violation for task t are bounded as follows (if the exact action-value function { Q i t,π } p i =0 are available for all π ) 1 :

Original: [Equation]
Revised: [Equation]

Original: where ˆ π t is the policy returned by running CRPO for M steps with learning rate α t in task t , π ∗ t is the optimal policy, c max is the upper bound on reward/cost function, and D KL ( ·|· ) is the KL divergence.
Revised: where ˆ π t is the policy returned by running CRPO for M steps with learning rate α t in task t , π ∗ t is the optimal policy, c max is the upper bound on reward/cost function, and D KL ( ·|· ) is the KL divergence.

Revised: 2 Note that we use a simplified version of CRPO in this section to illustrate the main idea of our framework.

Revised: A full analysis with the original CRPO algorithm is provided in Sectionto ensure practical applicability.

Revised: In particular, the results of Theorems 3.1, 3.3 , and Corollary 1 are based on the analysis with the non-simplified CRPO algorithm.

Original: 2. M ETA -SRL PROBLEM SETUP
Revised: 2. M ETA -SRL PROBLEM SETUP

Original: We now consider the lifelong extension of CMDPs in which safe-RL tasks arrive one at a time and t = 1 , 2 , . .
Revised: We now consider the lifelong extension of CMDPs in which safe-RL tasks arrive one at a time and t = 1 , 2 , . .

Original: , T denotes the index for a sequence of online learning problems.
Revised: , T denotes the index for a sequence of online learning problems.

Original: In each single task t , the agent must sequentially optimize the policy { π t,i } Mi =

Revised: In each single task t , the agent must sequentially optimize the policy { π t,i } Mi =

Original: so that the corresponding sub-optimality and the constraint violation, such as given in 2, decays sub-linearly in M .
Revised: so that the corresponding sub-optimality and the constraint violation, such as given in 2, decays sub-linearly in M .

Original: Beyond the single task, the meta-learner should aim to optimize the upper bounds in 2 over the initial policy π t, 0 and the learning rate α t so that the task-averaged sub-optimality and the task-averaged constraint violation are expected to improve as the meta-learner solves more tasks.
Revised: Beyond the single task, the meta-learner should aim to optimize the upper bounds in 2 over the initial policy π t, 0 and the learning rate α t so that the task-averaged sub-optimality and the task-averaged constraint violation are expected to improve as the meta-learner solves more tasks.

Original: Therefore, we will aim to minimize the task-averaged sub-optimality gap and the task-averaged constraint violation defined as follows:
Revised: Therefore, we will aim to minimize the task-averaged sub-optimality gap and the task-averaged constraint violation defined as follows:

Original: Definition 1.
Revised: Definition 1.

Original: The task-averaged optimality gap (TAOG) ¯ R 0 and the task-averaged constraintviolation (TACV)
Revised: The task-averaged optimality gap (TAOG) ¯ R 0 and the task-averaged constraintviolation (TACV)

Original: ¯ R i of a safe-RL algorithm after T tasks are 
Revised: ¯ R i of a safe-RL algorithm after T tasks are ¯ R

Original: [Equation]
Revised: [Equation]

Original: where ˆ π t is the policy returned by running some safe-RL algorithm for M time-steps at task t where the expectation is taken with respect to the randomness of the algorithm and environment.
Revised: where ˆ π t is the policy returned by running some safe-RL algorithm for M time-steps at task t where the expectation is taken with respect to the randomness of the algorithm and environment.

Original: Note that, unlike in the standard regret, one cannot achieve TAOG and TACV decreasing in T without further assumptions on the environment (e.g., Kwon et al. because the comparator π ∗ t is dynamic which can lead to suboptimality or constraint violation at each task t .
Revised: Note that, unlike in the standard regret, one cannot achieve TAOG and TACV decreasing in T without further assumptions on the environment (e.g., (Kwon et al., 2021)) because the comparator π ∗ t is dynamic which can lead to suboptimality or constraint violation at each task t .

Original: We can expect that the upper bounds on the task-averaged regrets depend on the policy initialization π t, 0 and the learning rate α t for each task.
Revised: We can expect that the upper bounds on the task-averaged regrets depend on the policy initialization π t, 0 and the learning rate α t for each task.

Original: Therefore, in this work, we aim to design the meta algorithm which can learn the initialization of the policy π t, 0 and the learning rate of the safe-RL algorithm α t so that TAOG and TACV are diminishing.
Revised: Therefore, in this work, we aim to design the meta algorithm which can learn the initialization of the policy π t, 0 and the learning rate of the safe-RL algorithm α t so that TAOG and TACV are diminishing.

Original: We will consider CRPO as the within-task algorithm.
Revised: We will consider CRPO as the within-task algorithm.

Original: 2. T ASK - SIMILARITY
Revised: 2. T ASK - SIMILARITY

Original: In Meta-SRL, we expect TAOG and TACV to improve with the similarity among the online CMDP tasks.
Revised: In Meta-SRL, we expect TAOG and TACV to improve with the similarity among the online CMDP tasks.

Original: Furthermore, the notion of similarity not only affects the evaluation of the meta-learning algorithm, but also impacts the quality of the meta initialization being learned and eventually the performance on an unseen task.
Revised: Furthermore, the notion of similarity not only affects the evaluation of the meta-learning algorithm, but also impacts the quality of the meta initialization being learned and eventually the performance on an unseen task.

Original: We now discuss the notions of similarity in a static environment; an extension to a dynamic environment, named task-relatedness, is introduced in Sec. 3.2.
Revised: We now discuss the notions of similarity in a static environment; an extension to a dynamic environment, named task-relatedness, is introduced in Sec. 3.2.

Original: Given optimal polices { π ∗ t } Tt =1 , where π ∗ t ∈ Π ∗ t for every t , the task-similarity can be measured by
Revised: Given optimal polices { π ∗ t } Tt =1 , where π ∗ t ∈ Π ∗ t for every t , the task-similarity can be measured by

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original:  | ϕ )] .
Revised: [ D [Equation] ( π ∗ t | ϕ )] .

Original: If the optimal policies are not unique, we take the worst case for D ∗ , i.e., set of policies for which D ∗ 2 is maximum.
Revised: If the optimal policies are not unique, we take the worst case for D ∗ , i.e., set of policies for which D ∗ 2 is maximum.

Original: This notion of task-similarity in the static environment is natural for studying gradient-based meta-learning, as it implies that there exists a meta initialization ϕ with respect to which optimal policies for individual tasks are all close together.
Revised: This notion of task-similarity in the static environment is natural for studying gradient-based meta-learning, as it implies that there exists a meta initialization ϕ with respect to which optimal policies for individual tasks are all close together.

Original: In particular, when the tasks are all identical, i.e., { π ∗ t } Tt =1 are all equal, we have
Revised: In particular, when the tasks are all identical, i.e., { π ∗ t } Tt =1 are all equal, we have

Original: D ∗ 2 = 

Revised: D ∗ 2 =

Original: (even though the optimal state distributions ν ∗ t may differ among tasks).
Revised: (even though the optimal state distributions ν ∗ t may differ among tasks).

Original: In the practical scenario where only suboptimal policies are accessible, we denote the empirical task-similarity as ˆD
Revised: In the practical scenario where only suboptimal policies are accessible, we denote the empirical task-similarity as ˆ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ] , which depends on the suboptimal policies { ˆ π t } Tt = returned by a within-task algorithm.
Revised: ϕ )] , which depends on the suboptimal policies { ˆ π t } Tt = returned by a within-task algorithm.

Original: Note that this is a natural notion of similarity that resembles
Revised: Note that this is a natural notion of similarity that resembles

Original: Bregman information introduced in the setting of clustering Banerjee et al.
Revised: Bregman information introduced in the setting of clustering (Banerjee et al., 2005).

Original: In this work, we aim to develop algorithms whose TAOG and TACV scale with the task-similarity, which implies that the method will do well if tasks are similar.
Revised: In this work, we aim to develop algorithms whose TAOG and TACV scale with the task-similarity, which implies that the method will do well if tasks are similar.

Original: To understand the CMDP-within-online framework and the impact of task-similarity on the upper bounds of TAOG and TACV for Meta-SRL, we first present a simplified result under the ideal setting where { ν ∗ t } Tt =1 and { π ∗ t } Tt =1 are available after each task t and the task-similarity D ∗ 2 is known.
Revised: To understand the CMDP-within-online framework and the impact of task-similarity on the upper bounds of TAOG and TACV for Meta-SRL, we first present a simplified result under the ideal setting where { ν ∗ t } Tt =1 and { π ∗ t } Tt =1 are available after each task t and the task-similarity D ∗ 2 is known.

Original: As it is desirable that the meta-initialization policy π t, 0 has good exploration properties, we require the initial policy to have full support over
Revised: As it is desirable that the meta-initialization policy π t, 0 has good exploration properties, we require the initial policy to have full support over

Original: S × A .
Revised: S × A .

Original: Assumption 1.
Revised: Assumption 1.

Original: The meta-initialization policy π t, 0 for any task t lies inside a shrinkage simplex set , i.e., π t,0 ( ·| s ) ∈ ∆ A ϱ :=
Revised: The meta-initialization policy π t, 0 for any task t lies inside a shrinkage simplex set , i.e., π t,

Original: { a 1 e 1 + ...
Revised: [Equation]

Original: [-+ a n a e n a | (cid:80) n a i =
-]
Revised: [-+ a n a e n a | (cid:80) n a i =
-][Equation]

Original: a i = 1 , a i ≥ ϱ ∀ i = 1 , ..., n a } for all s ∈ S , where e 1 , ..., e n a ∈ R n a are one-hot vectors for each action (e.g., e 1 is the vector of all 0 s exceptat the first location) and ϱ > 0 .
Revised: [Equation] } for all s ∈ S , where e 1 , ..., e n a ∈ R n a are one-hot vectors for each action (e.g., e 1 is the vector of all 0 s exceptat the first location) and ϱ > 0 .

Original: In particular, ∆ A ϱ lies inside the regular simplex set ∆ A ( ϱ = 0 ).
Revised: In particular, ∆ A ϱ lies inside the regular simplex set ∆ A ( ϱ = 0 ).

Original: If Assumption 1 is satisfied, the following holds for the meta-initialization policy π t, 0 for any state s ∈ S with positive constants C π , L g , L π and µ π : (1) | D KL ( π ∗ t ( ·| s ) | π t, 0 ( ·| s ))
Revised: If Assumption 1 is satisfied, the following holds for the meta-initialization policy π t, 0 for any state s ∈ S with positive constants C π , L g , L π and µ π : (1) | D KL ( π ∗ t ( ·| s ) | π t, 0 ( ·| s ))

Original: | , | D KL (ˆ π t ( ·| s ) | π t, 0 ( ·| s ))
Revised: | , | D KL (ˆ π t ( ·| s ) | π t, 0 ( ·| s ))

Original: | ≤ C π ; (2) D KL ( π ∗ t ( ·| s ) | π t, 0 ( ·| s )) is L g -Lipschitz and L π -smooth in π t, 0 ( ·| s ) ; (3) D KL ( π ∗ t ( ·| s ) | π t, 0 ( ·| s )) is µ π -strongly convex in π t, 0 ( ·| s ) .
Revised: | ≤ C π ; (2) D KL ( π ∗ t ( ·| s ) | π t, 0 ( ·| s )) is L g -Lipschitz and L π -smooth in π t, 0 ( ·| s ) ; (3) D KL ( π ∗ t ( ·| s ) | π t, 0 ( ·| s )) is µ π -strongly convex in π t, 0 ( ·| s ) .

Original: These conditions follow directly from Assumption 1 as the policy π t, 0 are bounded away from zero for all state and action pairs.
Revised: These conditions follow directly from Assumption 1 as the policy π t, 0 are bounded away from zero for all state and action pairs.

Original: The first condition can also be satisfied for compact Θ (Mei et al., 2020, Lemma 27).
Revised: The first condition can also be satisfied for compact Θ (Mei et al., 2020, Lemma 27).

Original: More relaxed conditions than Assumption 1 are possible (see, e.g., Zhang et al. ; Baby & Wang (2022)).
Revised: More relaxed conditions than Assumption 1 are possible (see, e.g., (Zhang et al., 2017; Baby & Wang, 2022)).

Original: Next, we present the bounds for TAOG and TACV assuming the task-similarity is known.
Revised: Next, we present the bounds for TAOG and TACV assuming the task-similarity is known.

Original: Lemma 1.
Revised: Lemma 1.

Original: Assume { ν ∗ t } Tt =1 and { π ∗ t } Tt =1 are given after each task and the task-similarity D ∗ 2 is known.
Revised: Assume { ν ∗ t } Tt =1 and { π ∗ t } Tt =1 are given after each task and the task-similarity D ∗ 2 is known.

Original: For each task t , we run CRPO for M iterations with α =
Revised: For each task t , we run CRPO for M iterations with α =

Original: [Equation]
Revised: [Equation]

Original: ∗ [Equation] In addition, the initialization { π t, 0 } Tt =1 are determined by playing Follow-the-Regularized-Leader (FTRL) or online gradient descent (OGD) on the functions E s ∼ ν ∗ t
Revised: In addition, the initialization { π t, 0 } Tt =1 are determined by playing Follow-the-Regularized-Leader (FTRL) or online gradient descent (OGD) on the functions E s ∼ ν ∗ t

Original: [ D KL ( π ∗ t |· )] , for t = 1 , . .
Revised: [ D KL ( π ∗ t |· )] , for t = 1 , . .

Original: , T . 

Revised: , T .

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: The above result reveals an interesting benefit brought by including more tasks (the regret decays at a rate of log( T ) /T ) with more similarity (i.e., lower D ∗ ), which improves upon single-task guarantee and serves as the initial point of our study.
Revised: The above result reveals an interesting benefit brought by including more tasks (the regret decays at a rate of log( T ) /T ) with more similarity (i.e., lower D ∗ ), which improves upon single-task guarantee and serves as the initial point of our study.

Original: However, there are several limitations.
Revised: However, there are several limitations.

Original: First, if the optimal  policies π ∗ t and the induced state distributions ν ∗ t are not revealed after each task, it is not likely that the plug-in estimator { E s ∼
Revised: First, if the optimal [Equation] [Equation] to obtain α policies π ∗ t and the induced state distributions ν ∗ t are not revealed after each task, it is not likely that the plug-in estimator { E s ∼

Original: ˆ
Revised: ˆ

Original: ν t [ D KL (ˆ π t |· )] } Tt =1 with the learned policy ˆ π t and estimated visitation distribution ˆ ν t is an unbiased estimator, ruling out existing analysis for FTRL or OGD in the bandit setting.
Revised: ν t [ D KL (ˆ π t |· )] } T t =1 with the learned policy ˆ π t and estimated visitation distribution ˆ ν t is an unbiased estimator, ruling out existing analysis for FTRL or OGD in the bandit setting.

Original: Besides, while the knowledge of D ∗ used to determine the learning rate can be relaxed Khodak et al. ; Balcan et al. (2019), the resulting scheme is complex to implement—we expect that the learning rates can be chosen adaptively for different tasks based on losses observed in the past.
Revised: Besides, while the knowledge of D ∗ used to determine the learning rate can be relaxed (Khodak et al., 2019; Balcan et al., 2019), the resulting scheme is complex to implement—we expect that the learning rates can be chosen adaptively for different tasks based on losses observed in the past.

Original: We aim to address these challenges with a series of developments in the next section.
Revised: We aim to address these challenges with a series of developments in the next section.

Original:  P ROVABLE GUARANTEES FOR PRACTICAL CMDP- WITHIN - ONLINE FRAMEWORK 3. I NEXACT CMDP- WITHIN - ONLINE FRAMEWORK
Revised:  P ROVABLE GUARANTEES FOR PRACTICAL CMDP- WITHIN - ONLINE FRAMEWORK 3. I NEXACT CMDP- WITHIN - ONLINE FRAMEWORK

Original: One of the key steps to generalize the online-within-online methodology Balcan et al. Alquier et al. (2017) to Meta-SRL is to relax the assumption of accessing the exact upper-bounds of within-task performance by designing algorithms to estimate and update on their inexact versions.
Revised: One of the key steps to generalize the online-within-online methodology (Balcan et al., 2019; Alquier et al., 2017) to Meta-SRL is to relax the assumption of accessing the exact upper-bounds of within-task performance by designing algorithms to estimate and update on their inexact versions.

Original: Estimation of upper bounds.
Revised: Estimation of upper bounds.

Original: Once a CMDP task t is complete, the meta-learner only has access to a suboptimal policy ˆ π t and the trajectory dataset D t produced by some safe-RL algorithm.
Revised: Once a CMDP task t is complete, the meta-learner only has access to a suboptimal policy ˆ π t and the trajectory dataset D t produced by some safe-RL algorithm.

Original: Let
Revised: Let

Original: ˜ ν t denote the discounted state visitation distribution induced by policy ˆ π t .
Revised: ˜ ν t denote the discounted state visitation distribution induced by policy ˆ π t .

Original: To obtain an estimate ˆ ν t from D t , state-of-the-art methods often rely on estimating discounted state visitation distribution ratios or corrections Hallak & Mannor (2017); Liu et al. ; Gelada & Bellemare (2019).
Revised: To obtain an estimate ˆ ν t from D t , state-of-the-art methods often rely on estimating discounted state visitation distribution ratios or corrections (Hallak & Mannor, 2017; Liu et al., 2018; Gelada & Bellemare, 2019).

Original: However, the main issues are that D t is collected by multiple behavior policies during the learning period, and depending on how far these behavior policies are from the target policy, the per-step importance ratios involved in these methods may have large variance, which may result in a detrimental effect on stochastic algorithms.
Revised: However, the main issues are that D t is collected by multiple behavior policies during the learning period, and depending on how far these behavior policies are from the target policy, the per-step importance ratios involved in these methods may have large variance, which may result in a detrimental effect on stochastic algorithms.

Original: In this work, we use one of the methods from the distribution correction estimation (DICE) family, namely DualDICE Nachum et al. (2019), which is agnostic to the number or type of behavior policies used and does not involve any per-step importance ratios, thus is less likely to be affected by their high variance.
Revised: In this work, we use one of the methods from the distribution correction estimation (DICE) family, namely DualDICE (Nachum et al., 2019), which is agnostic to the number or type of behavior policies used and does not involve any per-step importance ratios, thus is less likely to be affected by their high variance.

Original: In particular, for each state-action pair ( s, a ) , the method aims to estimate the quantity ω π/ D t ( s, a ) =
Revised: In particular, for each state-action pair ( s, a ) , the method aims to estimate the quantity ω π/ D t ( s, a ) =

Original: d π ( s,a ) d D t ( s,a ) , i.e., the likelihood that the target policy π will experience the state-action pair normalized by the probability with which the state-action pair appears in the off-policy data D t .
Revised: d π ( s,a ) d D t ( s,a ) , i.e., the likelihood that the target policy π will experience the state-action pair normalized by the probability with which the state-action pair appears in the off-policy data D t .

Original: Thereby, we estimate E ν ∗ t [ D KL ( π ∗ t | π )] with E
Revised: Thereby, we estimate E ν ∗ t [ D KL ( π ∗ t | π )] with E

Original: ˆ ν t [ D KL (ˆ π t
Revised: ˆ ν t [ D KL (ˆ π t

Original: | π )] by plugging in ˆ π t from the within-task CMDP
Revised: | π )] by plugging in ˆ π t from the within-task CMDP

Original: and ˆ ν t from DualDICE in lieu of the optimal policy π ∗ t and the corresponding discounted state visitation distribution ν ∗ t .
Revised: and ˆ ν t from DualDICE in lieu of the optimal policy π ∗ t and the corresponding discounted state visitation distribution ν ∗ t .

Original: Bounding the estimation error.
Revised: Bounding the estimation error.

Original: We breakdown the error by sources of origin:
Revised: We breakdown the error by sources of origin:

Original: [Equation]
Revised: [Equation]

Original: where ( A ) accounts for the mismatch between the discounted state visitation distributions of an optimal policy π ∗ t and a suboptimal one ˆ π t , ( B ) originates from the estimation error of DualDICE, and ( C ) is due to the difference between π ∗ t and ˆ π t measured according to ˆ π t .
Revised: where ( A ) accounts for the mismatch between the discounted state visitation distributions of an optimal policy π ∗ t and a suboptimal one ˆ π t , ( B ) originates from the estimation error of DualDICE, and ( C ) is due to the difference between π ∗ t and ˆ π t measured according to ˆ π t .

Original: By triangle inequality, we can bound the total error by controlling each term separately.
Revised: By triangle inequality, we can bound the total error by controlling each term separately.

Original: To bound ( A ) , we need to control the distance between ν ∗ t and ˜ ν t , which can be bounded by the distance between the inducing policy parameters as long as they are Lipschitz continuous (Xu et al., 2020, Lemma 3).
Revised: To bound ( A ) , we need to control the distance between ν ∗ t and ˜ ν t , which can be bounded by the distance between the inducing policy parameters as long as they are Lipschitz continuous (Xu et al., 2020, Lemma 3).

Original: In addition, the bound on ( C ) also depends on the distance between policies.
Revised: In addition, the bound on ( C ) also depends on the distance between policies.

Original: In general, controlling the distance between a policy to an optimal policy based on the suboptimality gap requires the optimization to have some curvatures around the optima (e.g., quadratic growth Drusvyatskiy & Lewis (2018) or Hölderian growth Johnstone & Moulin (2020)).
Revised: In general, controlling the distance between a policy to an optimal policy based on the suboptimality gap requires the optimization to have some curvatures around the optima (e.g., quadratic growth (Drusvyatskiy & Lewis, 2018) or Hölderian growth (Johnstone & Moulin, 2020)).

Original: However, to the best of knowledge of the authors, the only available results are an algorithm-dependent PL inequalities for policy gradient Mei et al. (2020) or quadratic growth with entropy regularization Ding et al. (2021b).
Revised: However, to the best of knowledge of the authors, the only available results are an algorithm-dependent PL inequalities for policy gradient (Mei et al., 2020) or quadratic growth with entropy regularization (Ding et al., 2021b).

Original: Given some mild assumptions on the objective/constraint functions and policy parametrization as follows, we can show that a growth condition holds broadly for any CMDP problems.
Revised: Given some mild assumptions on the objective/constraint functions and policy parametrization as follows, we can show that a growth condition holds broadly for any CMDP problems.

Original: Assumption 2.
Revised: Assumption 2.

Original: The functions J t,i ( · ) for i = 0 , 1 , ..., p and t ∈ [ T ] and parametric policy π θ are definable in some o-minimal structure (Van den Dries & Miller, 1996).
Revised: The functions J t,i ( · ) for i = 0 , 1 , ..., p and t ∈ [ T ] and parametric policy π θ are definable in some o-minimal structure (Van den Dries & Miller, 1996).

Revised: Assumption 2 allows the optimization to have a curvature around the optima by assuming the definability of policy parametrization and objective/constraints in some o-minimal structure.

Revised: We make use of the fact that the definable sets admit the property of Whitney stratification, and that any stratifiable function enjoys a nonsmooth Kurdyka-Lojasiewicz inequality.

Revised: This provides the foundation to bound ∥ ˆ π t − π ∗ t ∥ and ∥

Original:  Theorem 3.1 (KL divergence estimation error bound) .
Revised: ν ∗ t − ˜ ν t ∥ in terms of policy parameters, i.e., ∥ θ ∗ t − ˆ θ t ∥ . For more details on o-minimal structure, preliminaries on tame geometry, and how we make use of the Assumption 2 to bound terms (A) and (C), refer Appendix F.1- F.3. Theorem 3.1 (KL divergence estimation error bound) .

Original: The following bound holds:
Revised: The following bound holds:

Original: [Equation]
Revised: [Equation]

Original: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in Proposition 1, ϵ approx
Revised: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in Proposition 1, ϵ approx

Original: ( F , H ) and ϵ opt are the approximation error and optimization error of DualDICE, defined in Equation 38 and Equation 39, respectively.
Revised: ( F , H ) and ϵ opt are the approximation error and optimization error of DualDICE, defined in Equation 38 and Equation 39, respectively.

Original: Remark 1.
Revised: Remark 1.

Original: We define cumulative inexactness E T :
Revised: We define cumulative inexactness E T :

Original: = (cid:80) Tt =1 ϵ t .
Revised: = (cid:80) Tt =1 ϵ t .

Original: This quantity decays with M at a rate of O (cid:16) h (cid:16) 1 √ M (cid:17) + 1 √ M (cid:17) up to some approximation and optimization errors ϵ approx and ϵ opt .
Revised: This quantity decays with M at a rate of O (cid:16) h (cid:16) 1 √ M (cid:17) + 1 √ M (cid:17) up to some approximation and optimization errors ϵ approx and ϵ opt .

Original: Moreover, there is a trade-off between ϵ approx and ϵ opt : if the parametrization functions F and H used to solve DualDICE optimization are chosen as neural networks, then ϵ approx can be reduced at the cost of increasing ϵ opt .
Revised: Moreover, there is a trade-off between ϵ approx and ϵ opt : if the parametrization functions F and H used to solve DualDICE optimization are chosen as neural networks, then ϵ approx can be reduced at the cost of increasing ϵ opt .

Original: If we use stochastic gradient descent (SGD) as an optimization algorithm in DualDICE with K steps, then ϵ opt decays at a rate of O (1 /K ) .
Revised: If we use stochastic gradient descent (SGD) as an optimization algorithm in DualDICE with K steps, then ϵ opt decays at a rate of O (1 /K ) .

Original: Also, note that h is a strictly increasing continuous definable function used in the Kurdyka–Łojasiewicz (KL) inequality (see, e.g., (Bolte et al., 2007, Thm. 14)).
Revised: Also, note that h is a strictly increasing continuous definable function used in the Kurdyka–Łojasiewicz (KL) inequality (see, e.g., (Bolte et al., 2007, Thm. 14)).

Original: With the above uniform bound on estimation error, our next step is to develop static regret bounds for the inexact online gradient descent, which can be used to furnish the TAOG and TACV of the proposed inexact CMDP-within-online algorithm.
Revised: With the above uniform bound on estimation error, our next step is to develop static regret bounds for the inexact online gradient descent, which can be used to furnish the TAOG and TACV of the proposed inexact CMDP-within-online algorithm.

Original: Lemma 2 (Static regret bound for inexact OGD) .
Revised: Lemma 2 (Static regret bound for inexact OGD) .

Original: Denote f t ( π t, 0 ) :
Revised: Denote f t ( π t, 0 ) :

Original: = E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] for all t ∈ [ T ] .
Revised: = E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] for all t ∈ [ T ] .

Original: For any fixed comparator π ∗ 0 = arg min
Revised: For any fixed comparator π ∗ 0 = arg min

Original: [Equation]
Revised: [Equation]

Original: (cid:80) Tt =1 f t ( π 0 ) , if OGD is run on a sequence of loss functions {
Revised: t [Equation] π 0 ) , if OGD is run on a sequence of loss functions {

Original: ˆ f t } t ∈
Revised: ˆ f t } t ∈

Original: [ T ] , where ˆ f t ( π t, 0 ) :
Revised: [ T ] , where ˆ f t ( π t, 0 ) :

Original: =
Revised: =

Original: E ˆ
Revised: E ˆ

Original: ν t [ D KL (ˆ π t
Revised: ν t [ D KL (ˆ π t

Original: | π t, 0 )] for all t ∈
Revised: | π t, 0 )] for all t ∈

Original: [ T ] with the step-size of
Revised: [ T ] with the step-size of

Original: O (1 / √ T ) , then the following bound holds for static regret:
Revised: O (1 / √ T ) , then the following bound holds for static regret:

Original: [Equation]
Revised: [Equation]

Original: where E T := (cid:80) Tt
Revised: where E T : = (cid:80) Tt

Original: =

Revised: =

Original: ϵ t is the cumulative inexactness, and ϵ t is the upper bound from Theorem 3.1.
Revised: ϵ t is the cumulative inexactness, and ϵ t is the upper bound from Theorem 3.1.

Original: Refer the appendix for the full proof.
Revised: Refer the appendix for the full proof.

Revised: Note that there is a key practical consideration by defining the static regret with respect to the initial policy, not the final learned policy.

Revised: A static regret with respect to a static final learned policy has the flaw, which reduces the meta-learning problem to a single-task problem.

Revised: However, a static regret with respect to an initial policy provides freedom for the safe RL algorithm to adapt the initial policy based on observations within the task, which is exactly the meta-learning scenario analyzed in this paper.

Original: Once the static regret for the inexact OGD is established, we can obtain the TAOG and TACV for the proposed inexact CMDP-within-online algorithm in terms of the empirical task-similarity ˆ D 
Revised: Once the static regret for the inexact OGD is established, we can obtain the TAOG and TACV for the proposed inexact CMDP-within-online algorithm in terms of the empirical task-similarity ˆ D ∗ 2 =

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: For each task t , we run CRPO for M iterations with α= O (cid:16) 1 √ M (cid:113)
Revised: For each task t , we run CRPO for M iterations with α

Original: [Equation] D
Revised: = O

Original: [Equation]
Revised: [Equation]

Original: and we obtain { ˆ ν t } Tt =1 and { ˆ π t } Tt =1 .
Revised: and we obtain { ˆ ν t } Tt =1 and { ˆ π t } Tt =1 .

Original: In addition, the initialization { π t, 0 } Tt =1 for each task t are determined by playing inexact OGD (Algorithm 2) on E
Revised: In addition, the initialization { π t, 0 } Tt =1 for each task t are determined by playing inexact OGD (Algorithm 2) on E

Original: ˆ ν t
Revised: ˆ ν t

Original: [ D KL (ˆ π t |· )] , for t = 1 , . .
Revised: [ D KL (ˆ π t |· )] , for t = 1 , . .

Original: , T .
Revised: , T .

Original: Then, the following holds for TAOG ( i = 0 ) and TACV ( i = 1 , ..., p ):
Revised: Then, the following holds for TAOG ( i = 0 ) and TACV ( i = 1 , ..., p ):

Original: [Equation]
Revised: [Equation]

Original: The benefit of task-similarity is preserved when we perform directly on the plug-in estimator E ˆ ν t
Revised: The benefit of task-similarity is preserved when we perform directly on the plug-in estimator E ˆ ν t

Original: [ D KL (ˆ π t |· )] , though we incur an additional penalty on the inexactness E T and the dependence on T is worse compared to Lemma 1 in the ideal setting.
Revised: [ D KL (ˆ π t |· )] , though we incur an additional penalty on the inexactness E T and the dependence on T is worse compared to Lemma 1 in the ideal setting.

Original: As
Revised: As

Original: E T /T diminishes when the learned policy becomes optimal across tasks, e.g., by increasing within-task steps M or when meta-initialization is chosen such that a few steps suffice to reach optimal (as observed in the experiments), we expect the inexactness to have a limited effect on the performance.
Revised: E T /T diminishes when the learned policy becomes optimal across tasks, e.g., by increasing within-task steps M or when meta-initialization is chosen such that a few steps suffice to reach optimal (as observed in the experiments), we expect the inexactness to have a limited effect on the performance.

Original: 3. D YNAMIC REGRET AND TASK - RELATEDNESS
Revised: 3. D YNAMIC REGRET AND TASK - RELATEDNESS

Original: In many settings we have a changing environment, so it is natural to study dynamic regret and compare with a sequence of potentially time-varying comparators { ψ ∗ t } Tt =1 .
Revised: In many settings we have a changing environment, so it is natural to study dynamic regret and compare with a sequence of potentially time-varying comparators { ψ ∗ t } Tt =1 .

Original: To measure task-similarity in this case, we define task-relatedness which can be measured by V 2 ψ = 1 T (cid:80)
Revised: To measure task-similarity in this case, we define task-relatedness which can be measured by V 2 ψ = 1 T (cid:80)

Original: Tt =1 E s ∼ ν ∗ t [ D KL ( π ∗ t | ψ ∗ t )] .
Revised: Tt =1 E s ∼ ν ∗ t [ D KL ( π ∗ t | ψ ∗ t )] .

Original: This notion of task-relatedness gives the measure of how far optimal policies are in each task from some time-varying comparator.
Revised: This notion of task-relatedness gives the measure of how far optimal policies are in each task from some time-varying comparator.

Original: We denote empirical task-relatedness as ˆ V 2 ψ := 1 T (cid:80) T t =1 E s ∼ ˆν t
Revised: We denote empirical task-relatedness as ˆ V 2 ψ := 1 T (cid:80) Tt =1 E s ∼ ˆ

Original:  [ D KL (ˆ π t | ψ ∗ t )] , which depends on the subpotimal policy returned by the within-task algorithm.
Revised: ν t [ D KL (ˆ π t | ψ ∗ t )] , which depends on the subpotimal policy returned by the within-task algorithm.

Original: Note that ˆ V ψ is algorithm-dependent yet V ψ is algorithm-agnostic.
Revised: Note that ˆ V ψ is algorithm-dependent yet V ψ is algorithm-agnostic.

Original: Furthermore, to measure the performance of the Meta-SRL in dynamic settings, we analyze the dynamic regret bound, i.e., U T :=
Revised: Furthermore, to measure the performance of the Meta-SRL in dynamic settings, we analyze the dynamic regret bound, i.e., U T :=

Original: (cid:80) Tt =1 f t ( ϕ t ) − (cid:80)
Revised: (cid:80) Tt =1 f t ( ϕ t ) − (cid:80)

Original: Tt =1 f t ( ψ ∗ t ) , where ψ ∗ t ∈ arg min x ∈X f t ( x ) is a sequence of minimizers for each loss, and f t ( · ) =
Revised: Tt =1 f t ( ψ ∗ t ) , where ψ ∗ t ∈ arg min x ∈X f t ( x ) is a sequence of minimizers for each loss, and f t ( · ) =

Original: E s ∼ ν ∗ t
Revised: E s ∼ ν ∗ t

Original: [ D KL ( π ∗ t |· )] .
Revised: [ D KL ( π ∗ t |· )] .

Original: By exploiting the strong convexity of the loss function (KL divergence in our case), previous studies have shown that the dynamic regret can be upper bounded by the path-length of the comparator sequence, defined as P T := (cid:80) Tt =2 ∥ ψ ∗ t − ψ ∗ t − 1 ∥ , which captures the cumulative difference between successive comparators (Zhao et al., 2020).
Revised: By exploiting the strong convexity of the loss function (KL divergence in our case), previous studies have shown that the dynamic regret can be upper bounded by the path-length of the comparator sequence, defined as P T := (cid:80) T t =2 ∥ ψ ∗ t − ψ ∗ t − 1 ∥ , which captures the cumulative difference between successive comparators (Zhao et al., 2020).

Original: The bound can be further improved for strongly convex functions as the minimum of the path-length and the squared path-length, S T :
Revised: The bound can be further improved for strongly convex functions as the minimum of the path-length and the squared path-length, S T :

Original: =
Revised: =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: ∥
Revised: ∥

Original: ψ ∗ t − ψ ∗ t − 1 ∥ 2 , which can be much smaller than the path-length (Zhang et al., 2017).
Revised: ψ ∗ t − ψ ∗ t − 1 ∥ 2 , which can be much smaller than the path-length (Zhang et al., 2017).

Original: We extend these results to the settings of inexact online gradient descent by also allowing the learner to query the inexact gradient of the function.
Revised: We extend these results to the settings of inexact online gradient descent by also allowing the learner to query the inexact gradient of the function.

Original: Lemma 3 (Dynamic regret bound for inexact OGD) .
Revised: Lemma 3 (Dynamic regret bound for inexact OGD) .

Original: Denote f t ( ϕ t ) :=
Revised: Denote f t ( ϕ t ) :=

Original: E ν ∗ t [ D KL ( π ∗ t | ϕ t )] for all t ∈
Revised: E ν ∗ t [ D KL ( π ∗ t | ϕ t )] for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: For any dynamically varying comparator{ ϕ t } Tt =1 with ψ ∗ t∈ arg min ϕ t ∈ ∆ A |S| ϱ f t ( ϕ t ) , if single-step inexact OGD is run with the step-size β ≤ 12 µ π on a sequence of loss functions { ˆ f t } t ∈ [ T ] , where ˆ f t ( ϕ t ) =
Revised: For any dynamically varying comparator ψ ∗ t , if single-step inexact OGD is run with the step-size β ≤ 12 µ π on a sequence of loss functions { ˆ f t } t ∈ [ T ] , where ˆ f t ( ϕ t ) =

Original: E ˆ ν t [ D KL (ˆ π t | ϕ t )] , then the following bound holds for dynamic regret:
Revised: E ˆ ν t [ D KL (ˆ π t | ϕ t )] , then the following bound holds for dynamic regret:

Original: [Equation]
Revised: [Equation]

Original: where P T := (cid:80) Tt =

Revised: where P T := (cid:80) Tt =

Original: ∥
Revised: ∥

Original: ψ ∗ t − ψ ∗ t − 1 ∥ is the path-length of the comparator sequence, S T := (cid:80) Tt =

Revised: ψ ∗ t − ψ ∗ t − 1 ∥ is the path-length of the comparator sequence, S T := (cid:80) Tt =

Original: ∥
Revised: ∥

Original: ψ ∗ t − ψ ∗ t − 1 ∥ 2 is the squared path-length, E T :
Revised: ψ ∗ t − ψ ∗ t − 1 ∥ 2 is the squared path-length, E T :

Original: = (cid:80) T t =

Revised: = (cid:80) Tt =

Original: ϵ t is the cumulative inexactness, ˜ E T :=
Revised: ϵ t is the cumulative inexactness, ˜ E T :=

Original: (cid:80) T t =1 √ ϵ t is the cumulative square root of inexactness, and ϵ t is the upper bound from Theorem 3.1.
Revised: (cid:80) Tt =1 √ ϵ t is the cumulative square root of inexactness, and ϵ t is the upper bound from Theorem 3.1.

Original: 3. D YNAMIC REGRET WITH ADAPTIVE LEARNING RATES
Revised: 3. D YNAMIC REGRET WITH ADAPTIVE LEARNING RATES

Original: It can be observed from the last section that to set the learning rate α t for the within-task algorithm
Revised: It can be observed from the last section that to set the learning rate α t for the within-task algorithm

Original: CRPO, knowledge of optimal/suboptimal policies from all T tasks is used.
Revised: CRPO, knowledge of optimal/suboptimal policies from all T tasks is used.

Original: This makes the algorithm less applicable in online settings where tasks are encountered sequentially.
Revised: This makes the algorithm less applicable in online settings where tasks are encountered sequentially.

Original: Moreover, when the task-environment changes dynamically, a fixed policy ψ may not be the best candidate comparator, where it is natural to study dynamic regret by competing with a potentially time-varying sequence { ψ ∗ t } Tt =1 .
Revised: Moreover, when the task-environment changes dynamically, a fixed policy ψ may not be the best candidate comparator, where it is natural to study dynamic regret by competing with a potentially time-varying sequence { ψ ∗ t } Tt =1 .

Original: Also, the tasks may share some common aspects of the optimization landscape, so adapting learning rates based on prior experience may further improve performance.
Revised: Also, the tasks may share some common aspects of the optimization landscape, so adapting learning rates based on prior experience may further improve performance.

Original: This is the direction we pursue next.
Revised: This is the direction we pursue next.

Original: To begin with, recall the regret for suboptimality and constraint violation of the CRPO:
Revised: To begin with, recall the regret for suboptimality and constraint violation of the CRPO:

Original: [Equation]
Revised: [Equation]

Original: where the constants { c ti } i =1 ,..., 4 are given in Algorithm 1.
Revised: where the constants { c ti } i =1 ,..., 4 are given in Algorithm 1.

Original: We assume that α t ∈ Λ := { α t | α t ≥ ζ } for some ζ > 0 , where Λ is a convex set.
Revised: We assume that α t ∈ Λ := { α t | α t ≥ ζ } for some ζ > 0 , where Λ is a convex set.

Original: Overall, the goal of the meta-learner is to make a sequence of decisions, collected by x t = { π t, 0 ∈
Revised: Overall, the goal of the meta-learner is to make a sequence of decisions, collected by x t = { π t, 0 ∈

Original: ∆
Revised: ∆

Original: A |S| ϱ , α t ∈ Λ } , such that TAOG and TACV are minimized.
Revised: A |S| ϱ , α t ∈ Λ } , such that TAOG and TACV are minimized.

Original: To design the adaptive algorithm, we consider the following two parallel sequences of loss functions over initial policy ϕ , f initt ( ϕ ) =
Revised: To design the adaptive algorithm, we consider the following two parallel sequences of loss functions over initial policy ϕ , f init t

Original: E ν ∗ t [ D KL ( π ∗ t | ϕ )] , and learning rate κ ,

Original: [Equation]
Revised: [Equation]

Original: f
Revised: and learning rate

Original: [Equation]
Revised: [Equation]

Original: Note that f simt ( α t ) =
Revised: Note that f simt ( α t ) =

Original: U t ( π t, 0 , α t ) matches the upper bound in 5.
Revised: U t ( π t, 0 , α t ) matches the upper bound in 5.

Original: We also denote the inexact versions ˆ f initt ( ϕ ) and ˆ f simt ( κ ) by replacing E ν ∗ t [ D KL ( π ∗ t | ϕ )] with E
Revised: We also denote the inexact versions ˆ f initt ( ϕ ) and ˆ f simt ( κ ) by replacing E ν ∗ t [ D KL ( π ∗ t | ϕ )] with E

Original: ˆ ν t [ D KL (ˆ π t | ϕ )] in the above.
Revised: ˆ ν t [ D KL (ˆ π t | ϕ )] in the above.

Original: Inspired by Khodak et al. ; Niazadeh et al. (2021), instead of running one online algorithm on U t ( π t , α t ) , we will run two online algorithms separately for the function sequences ˆ f initt and ˆ f simt by taking actions on the initial policy and learning rates, respectively, such that the overall regret can be bounded by an expression that depends on the regrets for each sequence.
Revised: Inspired by (Khodak et al., 2019; Niazadeh et al., 2021), instead of running one online algorithm on U t ( π t , α t ) , we will run two online algorithms separately for the function sequences ˆ f initt and ˆ f simt by taking actions on the initial policy and learning rates, respectively, such that the overall regret can be bounded by an expression that depends on the regrets for each sequence.

Original: Let INIT and SIM be two algorithms, such that the actions π t +1 , 0 :=
Revised: Let INIT and SIM be two algorithms, such that the actions π t +1 , 0 :=

Original: INIT( t ) are taken over ˆ f initt and the actions α t +1 :=
Revised: INIT( t ) are taken over ˆ f initt and the actions α t +1 :=

Original: SIM( t ) are taken over ˆ f simt ; these actions will then be used as policy initialization and learning rates for the next
Revised: SIM( t ) are taken over ˆ f simt ; these actions will then be used as policy initialization and learning rates for the next

Original: CMDP.
Revised: CMDP.

Original: We assume the following regret upper bounds for each algorithm:
Revised: We assume the following regret upper bounds for each algorithm:

Original: U initT ( { ψ ∗ t } Tt =1 ) : upper bound on the dynamic regret for INIT over functions { ˆ f initt } Tt = with respect to a time-varying sequence { ψ ∗ t } Tt =1 ; [Equation]
Revised: U initT ( { ψ ∗ t } Tt =1 ) : upper bound on the dynamic regret for INIT over functions { ˆ f initt } Tt = with respect to a time-varying sequence { ψ ∗ t } Tt =1 ; 2.

Original: [Equation]
Revised: U simT ( κ ) : upper bound on the static regret for SIM over functions { ˆ f simt } Tt =1 with respect to a comparator κ > 0 .

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Let each within-task CMDP t run M steps of CRPO, initialized by policy π t, 0 :=
Revised: Let each within-task CMDP t run M steps of CRPO, initialized by policy π t, 0 :=

Original: INIT( t ) and learning rates α t := SIM( t ) .
Revised: INIT( t ) and learning rates α t := SIM( t ) .

Original: Let κ ∗ :=
Revised: Let κ ∗ :=

Original: arg min L ( κ ) , where
Revised: arg min L ( κ ) , where

Original: [Equation]
Revised: [Equation]

Original: and { ψ ∗ t } Tt =1 is any comparator sequence.
Revised: and { ψ ∗ t } Tt =1 is any comparator sequence.

Original: Then, the following bounds on TAOG and TACV hold:
Revised: Then, the following bounds on TAOG and TACV hold:

Original: [Equation]
Revised: [Equation]

Original: The following corollary presents the TAOG and TACV regret bounds when INIT and SIM are inexact
Revised: The following corollary presents the TAOG and TACV regret bounds when INIT and SIM are inexact

Original: OGD over the loss functions { ˆ f initt } Tt =1 and { ˆ f simt } Tt =1 respectively.
Revised: OGD over the loss functions { ˆ f initt } Tt =1 and { ˆ f simt } Tt =1 respectively.

Original: Corollary 1.
Revised: Corollary 1.

Original: If INIT( t ) and SIM( t ) are inexact OGD, and are run over the sequences { ˆ f initt } Tt
Revised: If INIT( t ) and SIM( t ) are inexact OGD, and are run over the sequences { ˆ f initt } Tt

Original: = and { ˆ f simt } Tt =1 , then, the following bounds on TAOG and TACV hold:
Revised: = and { ˆ f simt } Tt =1 , then, the following bounds on TAOG and TACV hold:

Original: [Equation]
Revised: [Equation]

Original: for all i = 0 , . ..
Revised: for all i = 0 , . .

Original: , p .
Revised: , p .

Original: Remark 2.
Revised: Remark 2.

Original: The bounds are improved in terms of M and T due to the adaptation of the learning rates.
Revised: The bounds are improved in terms of M and T due to the adaptation of the learning rates.

Original: Specifically, the bounds diminish at a rate O (cid:16)1 M
Revised: Specifically, the bounds diminish at a rate O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: [Equation] + ˆ V 2 ψ (cid:17)(cid:17) as compared to the previous rate O (cid:16) 1 √ M (cid:16)(cid:113) E T √ T + ˆ D ∗ 2 (cid:17)(cid:17) .
Revised: as compared to the previous rate O (cid:16) 1 √ M (cid:16)(cid:113) E T √ T + ˆ D ∗ 2 (cid:17)(cid:17) .

Original: Note that ˆ V ψ is same as ˆ D ∗ in the case of a fixed comparator ψ ∗ .
Revised: Note that ˆ V ψ is same as ˆ D ∗ in the case of a fixed comparator ψ ∗ .

Original: Moreover, a practical aspect of our algorithm is that it does not require the knowledge of quantities like S T , P T and E T to decide the value of learning rate α t .
Revised: Moreover, a practical aspect of our algorithm is that it does not require the knowledge of quantities like S T , P T and E T to decide the value of learning rate α t .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we show the effectiveness of the proposed Meta-SRL method and compare with the following baselines: simple averaging (i.e., initialize with the average of learned policies from past
Revised: In this section, we show the effectiveness of the proposed Meta-SRL method and compare with the following baselines: simple averaging (i.e., initialize with the average of learned policies from past

Original: CMDPs), pre-trained (i.e., initialize test task with the suboptimal policy from another CMDP)
Revised: CMDPs), pre-trained (i.e., initialize test task with the suboptimal policy from another CMDP)

Original: , Follow the Average Leader (FAL), and random initialization strategies as done in single-task CRPO.
Revised: , Follow the Average Leader (FAL), and random initialization strategies as done in single-task CRPO.

Original: Note that simple averaging takes average of previous suboptimal policies obtained random initializations on all CMDP tasks, while FAL does this in an online-manner while tasks arrive sequentially.
Revised: Note that simple averaging takes average of previous suboptimal policies obtained random initializations on all CMDP tasks, while FAL does this in an online-manner while tasks arrive sequentially.

Original: Different
Revised: Different

Original: CMDPs are generated using a probability distribution over the parameters of CMDPs (e.g., rewards, transition dynamics), similar to the latent CMDP model Chen et al. (2021a).
Revised: CMDPs are generated using a probability distribution over the parameters of CMDPs (e.g., rewards, transition dynamics), similar to the latent CMDP model (Chen et al., 2021a).

Original: We consider the Frozen lake and acrobot  environments from the OpenAI gym Brockman et al.
Revised: We consider the Frozen lake, acrobot and Half-Cheetah environments from the OpenAI gym (Brockman et al., 2016) and

Revised: MuJoco Todorov et al.

Original: (2016) under constrained settings.
Revised: (2012) under constrained settings.

Original: More details can be found in the Appendix.
Revised: More details can be found in the Appendix.

Original: We can observe from Figure 1 that Meta-SRL achieves higher rewards and lower constraint violations more quickly than baseline initializations.
Revised: We can observe from Figure 1 that Meta-SRL achieves higher rewards and lower constraint violations more quickly than baseline initializations.

Original: The baseline FAL shich simply takes the average of previous suboptimal policies performs poorly.
Revised: The baseline FAL which simply takes the average of previous suboptimal policies performs poorly.

Original: This illustrates the benefit of incorporating stationary distribution correction estimation and adaptive learning rates.
Revised: This illustrates the benefit of incorporating stationary distribution correction estimation and adaptive learning rates.

Original: Indeed, for Frozen lake, different locations of the hole can result in different stationary distributions—it is more sensible to put higher weights on policies that frequently visit a particular state, since it implies that the corresponding strategies can have substantial impact on the case of low task-similarity conditions.
Revised: Indeed, for Frozen lake, different locations of the hole can result in different stationary distributions—it is more sensible to put higher weights on policies that frequently visit a particular state, since it implies that the corresponding strategies can have substantial impact on the case of low task-similarity conditions.

Original: We also observe similar trends for the Acrobot in Figure 2, where Meta-SRL achieves higher rewards quickly and zero constraint violations as compared to other baseline initializations under low task-relatedness settings.
Revised: We also observe similar trends for the Acrobot in Figure 2, where Meta-SRL achieves higher rewards quickly and zero constraint violations as compared to other baseline initializations under low task-relatedness [Figure]

Original:  The pre-trained baseline was able to achieve higher rewards, but did not achieve constraint satisfactions for costs 1 and 2 .
Revised: settings. The pre-trained baseline was able to achieve higher rewards, but did not achieve constraint satisfactions for costs 1 and 2 .

Original: Under high task-similarity settings, we expected all the methods (except vanilla CRPO) to perform well; however, we noticed that simple averaging does poorly even in this setting, possibly due to adverse interference among different tasks.
Revised: Under high task-similarity settings, we expected all the methods (except vanilla CRPO) to perform well; however, we noticed that simple averaging does poorly even in this setting, possibly due to adverse interference among different tasks.

Revised: We also perormed extra experiments on the HalfCheetah from Mujoco Todorov et al.

Revised: It can be observed from Figure 3 that Meta-SRL is able to achieve higher rewards and zero constraint violations quickly compared to other baseline initializations under low task-relatedness settings.

Revised: The pre-trained baseline can achieve higher rewards, but similar to other baselines, it cannot achieve constraint satisfaction within 10 steps.

Revised: It can also be observed that both simple averaging and random initialization perform poorly in reward maximization in this setting.

Revised: Close inspection indicates that there is a high variance among the policy parameters learned from each task, which may result in interference among different tasks in the relatively high dimensional state space.

Original: Further results and discussions can be found in the Appendix.
Revised: For more details, please see the experimental section and Appendix H.

Original:  C ONCLUSION AND FUTURE DIRECTIONS
Revised:  C ONCLUSION AND FUTURE DIRECTIONS

Original: This paper introduced a novel framework, Meta-SRL, for meta-learning over CMDPs.
Revised: This paper introduced a novel framework, Meta-SRL, for meta-learning over CMDPs.

Original: The proposed framework does not assume access to globally optimal policies from the training tasks, and instead performs online learning over inexact within-task bounds estimated by stationary distribution correction.
Revised: The proposed framework does not assume access to globally optimal policies from the training tasks, and instead performs online learning over inexact within-task bounds estimated by stationary distribution correction.

Original: Moreover, strategies for learning rates adaptation are designed to further exploit taskrelatedness.
Revised: Moreover, strategies for learning rates adaptation are designed to further exploit task-relatedness.

Original: One of the limitations of the proposed method is that it only considers CRPO as the within-task algorithm; nevertheless, our framework can be potentially adapted to more single-task algorithms by making the dependence of guarantees on initial policy/step sizes explicit, e.g., safe exploration Efroni et al. regularization Geist et al. , off-policy evaluation Duan et al.
Revised: One of the limitations of the proposed method is that it only considers CRPO as the within-task algorithm; nevertheless, our framework can be potentially adapted to more single-task algorithms by making the dependence of guarantees on initial policy/step sizes explicit, e g., safe exploration (Efroni et al., 2020), regularization (Geist et al., 2019), off-policy evaluation (Duan et al., 2020).

Original: Some potential future directions could be to design Meta-SRL with zero constraint violation Liu et ), improve exploration using regularization Dai al
Revised: Some potential future directions could be to design Meta-SRL with zero constraint violation (Liu et al., 2021b;a), improve exploration using regularization (Dai et al., 2018;

Original: Geist  nonstationary environments 
Revised: Geist et al., 2019), nonstationary environments (Ding & Lavaei, 2022), and multi-agent settings (De Nijs et al., 2021)

Original: A PPENDIX
Revised: A PPENDIX

Original: In this section, we start with a summary of related work in Sec.
Revised: In this section, we start with a summary of related work in Sec.

Original: A followed by a brief recapitulation of the CRPO algorithm in Sec.
Revised: A followed by a brief recapitulation of the CRPO algorithm in Sec.

Original: B. Note that CRPO will be our focus as the exemplary within-task safe-RL algorithm.
Revised: B. Note that CRPO will be our focus as the exemplary within-task safe-RL algorithm.

Original: We also introduce notations therein that will be used in later analysis.
Revised: We also introduce notations therein that will be used in later analysis.

Original: In Sec.
Revised: In Sec.

Original: C, we give the pseudo-code of our inexact CMDP-within-online algorithm, with further discussions on key aspects.
Revised: C, we give the pseudo-code of our inexact CMDP-within-online algorithm, with further discussions on key aspects.

Original: Sec.
Revised: Sec.

Original: D provides the proof for Sec.
Revised: D provides the proof for Sec.

Original: 2 of the main paper, which focuses on an elementary but yet illustrative example of CMDP-within-online approach.
Revised: 2 of the main paper, which focuses on an elementary but yet illustrative example of CMDP-within-online approach.

Original: We start by providing a simplified proof to help the reader understand the main approach of CRPO, and then demonstrate the potential improvement by exploiting inter-task-relatedness (Lemma 5).
Revised: We start by providing a simplified proof to help the reader understand the main approach of CRPO, and then demonstrate the potential improvement by exploiting inter-task-relatedness (Lemma 5).

Original: Sec.
Revised: Sec.

Original: E contains the key developments in extending approaches of online learning, specifically online gradient descent, to the case of inexact loss functions.
Revised: E contains the key developments in extending approaches of online learning, specifically online gradient descent, to the case of inexact loss functions.

Original: We start with some preliminaries on ϵ -subgradient (Sec. E.1).
Revised: We start with some preliminaries on ϵ -subgradient (Sec. E.1).

Original: Then, we conduct the analysis for static regret (Thm.
Revised: Then, we conduct the analysis for static regret (Thm.

Original: E.1) and dynamic regret (Thm. E.3).
Revised: E.1) and dynamic regret (Thm. E.3).

Original: In Sec.
Revised: In Sec.

Original: F, we provide the detailed analysis on the KL divergence estimation error bound, which contributes to one of our main contributions in understanding the key aspects of the proposed inexact CMDP-within-online framework.
Revised: F, we provide the detailed analysis on the KL divergence estimation error bound, which contributes to one of our main contributions in understanding the key aspects of the proposed inexact CMDP-within-online framework.

Original: Our development leverages the seminal results developed for tame geometry, which we briefly review in Sec.
Revised: Our development leverages the seminal results developed for tame geometry, which we briefly review in Sec.

Original: F.1.
Revised: F.1.

Original: We also briefly set up the notations and recall basic properties of subgradient flow systems F.2.
Revised: We also briefly set up the notations and recall basic properties of subgradient flow systems F.2.

Original: Through a series of bounds, the final result is obtained in Thm.
Revised: Through a series of bounds, the final result is obtained in Thm.

Original: F.1.
Revised: F.1.

Original: We then proceed with providing proofs for Sec. 3.3.
Revised: We then proceed with providing proofs for Sec. 3.3.

Original: In Sec.
Revised: In Sec.

Original: G.1, we first extend the analysis of CRPO to the case of adaptive learning rates.
Revised: G.1, we first extend the analysis of CRPO to the case of adaptive learning rates.

Original: Then, we provide the proof for Thm.
Revised: Then, we provide the proof for Thm.

Original: 3.3 and Corollary 1 in Sec.
Revised: 3.3 and Corollary 1 in Sec.

Original: G.2.
Revised: G.2.

Original: Experimental details are provided in Sec.
Revised: Experimental details are provided in Sec.

Original: H. Frequently used notations and constants are listed in Sec.
Revised: H. Frequently used notations and constants are listed in Sec.

Original: I.
Revised: J.

Original: A R ELATED WORK
Revised: A R ELATED WORK

Original: Meta-reinforcement learning: Current state-of-the-art meta-
Revised: Meta-reinforcement learning: Current state-of-the-art meta-

Original: RL includes learning the initial conditions Finn et al. (2017), hyperparameters Jaderberg et al. (2019), step directions Li et al. and stepsizes Young et al.
Revised: RL includes learning the initial conditions (Finn et al., 2017), hyperparameters (Jaderberg et al., 2019), step directions (Li et al., 2017) and stepsizes (Young et al., 2018), and training recurrent neural networks to embed previous task experience (Duan et al., 2016)

Original: (2018), and training recurrent neural networks to embed previous task experience Duan et al. (see also Chen et al.
Revised: (see also (Chen et al., 2021a) for sim-to-real transfer, and Suilen et al.

Original: (2021a) for sim-to-real transfer), with recent developments on improving meta-optimization Rothfuss et al. Liu et al. ; Song et al.
Revised: (2022) for extension to robust MDPs), with recent developments on improving meta-optimization (Rothfuss et al., 2018; Liu et al., 2019; Song et al., 2019)

Original: (2019) (see Hospedales et al. (2020) for a review).
Revised: (see (Hospedales et al., 2020) for a review).

Original: Recently, Fallah et al. ; Ji et al. (2022) provided the theoretical studies on the convergence of model-agnostic meta-RL.
Revised: Recently, (Fallah et al., 2021; Ji et al., 2022) provided the theoretical studies on the convergence of model-agnostic meta-RL.

Original: However, these works all focus on the unconstrained meta-RL and their local optimality convergence, while our work is the first to obtain provable guarantees for optimality and constraint satisfaction for CMDPs.
Revised: However, these works all focus on the unconstrained meta-RL and their local optimality convergence, while our work is the first to obtain provable guarantees for optimality and constraint satisfaction for CMDPs.

Original: Online meta-learning/learning-to-learn (LTL).
Revised: Online meta-learning/learning-to-learn (LTL).

Original: Most efforts studying initialization-based metalearning focus on the setting with decomposable within-task loss functions that are often convex Finn et al. ; Denevi et al. ; Balcan et al.
Revised: Most efforts studying initialization-based metalearning focus on the setting with decomposable within-task loss functions that are often convex (Finn et al., 2019; Denevi et al., 2019; Balcan et al., 2019); nonconvex within-task settings are studied usually for multi-task representation learning (Balcan et al., 2015; Maurer et al., 2016; Du et al., 2020;

Original: ; nonconvex within-task settings are studied usually for multi-task representation learning Balcan et al. (2015); Maurer et al.

Original: ; Du et al. Tripuraneni et al.
Revised: Tripuraneni et al., 2020).

Original: Theoretically, our work is inspired by the Average RegretUpper-Bound Analysis (ARUBA) strategy Khodak et al.
Revised: Theoretically, our work is inspired by the Average Regret-Upper-Bound

Original:  for obtaining a meta-procedure, which has been recently extended to learning nonconvex piecewise-Lipschitz functions Balcan et al. (2021); the main technical advance in our work is in providing the guarantees for CMDPs, which is challenging due to the interplay between the nonconvexity and stochasticity of the optimization and the complexity of the within-task safe-RL algorithms that involves policy update, critic learning, and the proper choice of stepsizes for reward/constraints.
Revised: Analysis (ARUBA) strategy (Khodak et al., 2019) for obtaining a meta-procedure, which has been recently extended to learning nonconvex piecewise-Lipschitz functions (Balcan et al., 2021); the main technical advance in our work is in providing the guarantees for CMDPs, which is challenging due to the interplay between the nonconvexity and stochasticity of the optimization and the complexity of the within-task safe-RL algorithms that involves policy update, critic learning, and the proper choice of stepsizes for reward/constraints.

Original: Inexact online learning.
Revised: Inexact online learning.

Original: Online learning with access to inexact loss/gradient information has been studied for stochastic zero-biased noise Cesa-Bianchi et al.

Original: ; Yang et al.

Original: ; Bedi et al.

Original: (2018); Dixit et al. (2019), deterministic error/nonzero-biased stochastic noise Bedi et al. Dixit et al (2019), and adversarial perturbation Resler & Mansour (2019).
Revised: Online learning with access to inexact loss/gradient information has been studied for stochastic zero-biased noise (Cesa-Bianchi et al., 2011; Yang et al., 2016; Bedi et al., 2018; Dixit et al., 2019), deterministic error/nonzero-biased stochastic noise (Bedi et al., 2018; Dixit et al., 2019), and adversarial perturbation (Resler & Mansour, 2019).

Original: Our analysis for static regret uses the formalism of ϵ − subgradient (Jean-Baptiste, 2010, Chap. XI); for dynamic regret, we extend the work Zhang et al. (2017) to the inexact setting allowing multiple updates per round and provide improved rates than prior results Bedi et al. Dixit et al.
Revised: Our analysis for static regret uses the formalism of ϵ − subgradient (Jean-Baptiste, 2010, Chap. XI); for dynamic regret, we extend the work (Zhang et al., 2017) to the inexact setting allowing multiple updates per round and provide improved rates than prior results (Bedi et al., 2018; Dixit et al., 2019).

Original: ( safe-RL and CMDP.
Revised: safe-RL and CMDP.

Original: Direct policy search methods have had substantial empirical successes in solving
Revised: Direct policy search methods have had substantial empirical successes in solving

Original: CMDPs Borkar (2005); Uchibe & Doya (2007); Bhatnagar & Lakshmanan (2012;?); Achiam et al.
Revised: CMDPs (Borkar, 2005; Uchibe & Doya, 2007; Bhatnagar & Lakshmanan, 2012; Achiam et al., 2017;

Original: (2017); Chow et al. (2017) (see, e.g., Garcıa & Fernández (2015) for a survey of safe-RL).
Revised: Chow et al., 2017) (see, e.g., (Garcıa & Fernández, 2015) for a survey of safe-RL).

Original: Recently, a major progress in understanding the theoretical nonasymptotic global convergence behavior of policy-based methods for CMDPs has also been achieved Chow et al.
Revised: Recently, a major progress in understanding the theoretical nonasymptotic global convergence behavior of policy-based methods for CMDPs has also been achieved (Chow et al., 2018;

Original: ; Paternain et al. Efroni et al. Ding et al. (2021a)
Revised: Paternain et al., 2022; Efroni et al., 2020; Ding et al., 2021a; Ding & Lavaei, 2022;

Original: ; Ding & Lavaei (2022); Ying et al. (2022); Yu et al. Xu et al.
Revised: Ying et al., 2022; Yu et al., 2019; Xu et al., 2021;

Original: ; Chen et al. (2021b); Liu et al. (2021b;a).
Revised: Chen et al., 2021b; Liu et al., 2021b;a).

Original: However, most of these works only study a single CMDP task and don’t seek to make the algorithm perform well on new, potentially related CMDP tasks.
Revised: However, most of these works only study a single CMDP task and don’t seek to make the algorithm perform well on new, potentially related CMDP tasks.

Original: In addition, while our work uses the constraint-rectified policy optimization (CRPO) algorithm proposed in Xu et al. (2021) as a building block, our framework can be potentially adapted to most of the existing RL literature by making the dependence of guarantees on initial policy/step sizes explicit, e.g., safe exploration Efroni et al. regularization Geist et al. off-policy evaluation Duan et al.
Revised: In addition, while our work uses the constraint-rectified policy optimization (CRPO) algorithm proposed in (Xu et al., 2021) as a building block, our framework can be potentially adapted to most of the existing RL literature by making the dependence of guarantees on initial policy/step sizes explicit, e.g., safe exploration (Efroni et al., 2020), regularization (Geist et al., 2019), off-policy evaluation (Duan et al., 2020;

Original: Tennenholtz et al. (2020), and offline RL under constraints Le et al. ; Wu et al. ; Lee et al.
Revised: Tennenholtz et al., 2020), and offline RL under constraints (Le et al., 2019; Wu et al., 2021; Lee et al., 2021; Thomas et al., 2021).

Original: ; Thomas et al.

Original: B CRPO A LGORITHM AND NOTATIONS
Revised: B CRPO A LGORITHM AND NOTATIONS

Original: We provide some preliminaries and notations for the CRPO algorithm for the sake of completeness.
Revised: We provide some preliminaries and notations for the CRPO algorithm for the sake of completeness.

Original: CRPO (Xu et al., 2021) is a primal-based CMDP algorithm, which performs policy optimization (natural gradient ascent on the reward) when constraints are not violated, or constraint minimization (natural gradient descent on the constraint function) for the corresponding violated constraint.
Revised: CRPO (Xu et al., 2021) is a primal-based CMDP algorithm, which performs policy optimization (natural gradient ascent on the reward) when constraints are not violated, or constraint minimization (natural gradient descent on the constraint function) for the corresponding violated constraint.

Original: There are three crucial components in the overall strategy to solve the CMDP problem 1:
Revised: There are three crucial components in the overall strategy to solve the CMDP problem 1:

Original: Policy evaluation: In each step m of task t , for a certain policy π t,m , the action-value functions Q it,π t,m are estimated for the reward ( i = 0 ) and constraints ( i = 1 , ..., p ).
Revised: Policy evaluation: In each step m of task t , for a certain policy π t,m , the action-value functions Q it,π t,m are estimated for the reward ( i = 0 ) and constraints ( i = 1 , ..., p ).

Original: TDlearning is employed for critic evaluation Bhandari et al.
Revised: TDlearning is employed for critic evaluation (Bhandari et al., 2018).

Original: Estimation of constraint violation: Once the Q-estimates ¯ Q it,π t,m ( s, a ) are obtained, then a weighted average is taken to estimate expected constraint violation ¯ J t,i ( π t,m ) under a given policy π t,m .
Revised: Estimation of constraint violation: Once the Q-estimates ¯ Q it,π t,m ( s, a ) are obtained, then a weighted average is taken to estimate expected constraint violation ¯ J t,i ( π t,m ) under a given policy π t,m .

Original: Policy optimization: After the constraint estimation, it is checked if the expected constraint violation ¯ J it,π t,m exceeds the given safety threshold i.e., if ¯ J t,i ( π t,m ) ≤ d t,i + η t for all i = 1 , . . .
Revised: Policy optimization: After the constraint estimation, it is checked if the expected constraint violation ¯ J it,π t,m exceeds the given safety threshold i.e., if ¯ J t,i ( π t,m ) ≤ d t,i + η t for all i = 1 , . . .

Original: , p .
Revised: , p .

Original: If none of the constraints are violated, then one step of natural policy gradient ascent is performed to maximize the objective.
Revised: If none of the constraints are violated, then one step of natural policy gradient ascent is performed to maximize the objective.

Original: If one or more constraints are violated, then one step of natural policy gradient descent is conducted to minimize one of the unsatisfied constraints.
Revised: If one or more constraints are violated, then one step of natural policy gradient descent is conducted to minimize one of the unsatisfied constraints.

Original: The set of time steps the policy optimization for reward maximization takes place is denoted by
Revised: The set of time steps the policy optimization for reward maximization takes place is denoted by

Original: N t, 0 , and the set of time steps constraint minimization takes place is denoted by N t,i .
Revised: N t, 0 , and the set of time steps constraint minimization takes place is denoted by N t,i .

Original: Thus |N t, 0 | + [Equation] pi =

Revised: Thus |N t, 0 | + (cid:80) pi =

Original: |N t,i | =
Revised: |N t,i | =

Original: M for any task t .
Revised: M for any task t .

Original: C I NEXACT CMDP- WITHIN - ONLINE A LGORITHM
Revised: C I NEXACT CMDP- WITHIN - ONLINE A LGORITHM

Original: Algorithm 1 presents the inexact-CMDP-within-online algorithm for Meta-SRL.
Revised: Algorithm 1 presents the inexact-CMDP-within-online algorithm for Meta-SRL.

Original: The first step in the algorithm is to initialize with some random actor policy ϕ 1 , and the learning rate α 1 for the first task.
Revised: The first step in the algorithm is to initialize with some random actor policy ϕ 1 , and the learning rate α 1 for the first task.

Original: Then, for each task t , a within-task algorithm (i.e., CRPO) is run for M steps to obtain a policy ˆ π t .
Revised: Then, for each task t , a within-task algorithm (i.e., CRPO) is run for M steps to obtain a policy ˆ π t .

Original: The discounted state visitation distribution ˆ ν t induced by ˆ π t is then estimated using the trajectory data collected within-task t .
Revised: The discounted state visitation distribution ˆ ν t induced by ˆ π t is then estimated using the trajectory data collected within-task t .

Original: Afterwards, an inexact-OGD method is run on the new loss functions to update the meta-initialization policy ϕ t +1 , and the learning rate α t +1 .
Revised: Afterwards, an inexact-OGD method is run on the new loss functions to update the meta-initialization policy ϕ t +1 , and the learning rate α t +1 .

Original: The online learning loop is iterated for all tasks t ∈
Revised: The online learning loop is iterated for all tasks t ∈

Original: [ T ] .
Revised: [ T ] .

Original: D P ROOF IN S ECTION
Revised: D P ROOF IN S ECTION

Original: We first present a simplified proof for the results in 2:
Revised: We first present a simplified proof for the results in Equation 2.

Revised: Next lemma also presents how the safety threshold η t can be chosen to achieve sublinear convergence rate in M .

Original: Lemma 4.
Revised: Lemma 4.

Original: For CRPO Xu et al. (2021) with the softmax parametrization and the exact critic estimation (i.e., no critic evaluation error), if we have
Revised: For CRPO (Xu et al., 2021) with the softmax parametrization and the exact critic estimation (i.e., no critic evaluation error), if we have

Original: [Equation]
Revised: [Equation]

Original: then the following holds
Revised: then the following holds

Original: N t, 0 ̸ = ∅ , i.e., ˆ π t is well-defined, 2.
Revised: N t, 0 ̸ = ∅ , i.e., ˆ π t is well-defined, 2.

Original: J t, 0 ( π ∗ t ) − J t, 0 (ˆ π t ) ≤ η t . 3.
Revised: J t, 0 ( π ∗ t ) − J t, 0 (ˆ π t ) ≤ η t . 3.

Original: J t,i ( π ∗ t ) − J t,i (ˆ π t ) ≤ η t , for i
Revised: J t,i ( π ∗ t ) − J t,i (ˆ π t ) ≤ η t , for i

Original: = 1 , . .
Revised: = 1 , . .

Original: . , p .
Revised: . , p .

Original: Proof.
Revised: Proof.

Original: The following inequality holds due to Lemma 7 in Xu et al.
Revised: The following inequality holds due to Lemma 7 in (Xu et al., 2021):

Original: [Equation]
Revised: [Equation]

Original: We first verify item 1.
Revised: We first verify item 1.

Original: If N t, 0 = ∅ , then (cid:80) pi
Revised: If N t, 0 = ∅ , then (cid:80) pi

Original: =

Revised: =

Original: |N t,i | =
Revised: |N t,i | =

Original: M , and 10 implies that
Revised: M , and 10 implies that

Original: [Equation]
Revised: [Equation]

Original: which contradicts 9.
Revised: which contradicts 9.

Original: Thus, we must have N t, 0 ̸ = ∅ .
Revised: Thus, we must have N t, 0 ̸ = ∅ .

Original: We then proceed to verify item 2.
Revised: We then proceed to verify item 2.

Original: If (cid:80) m ∈N t, 0 ( J t, 0 ( π ∗
Revised: If (cid:80) m ∈N t, 0 ( J t

Original: [Equation]
Revised: [Equation]

Original: ) − J t
Revised: 0 ( π ∗

Original: [Equation]
Revised: [Equation]

Original: ( π t
Revised: ) − J t

Original: [Equation]
Revised: [Equation]

Original: ) > η t |N t, 0 | , then 10 implies that
Revised: ( π t,m )) > η t |N t, 0 | , then 10 implies that

Original: [Equation]
Revised: [Equation]

Original: which contradicts 9.
Revised: which contradicts 9.

Original: Hence, the item 2 holds.
Revised: Hence, the item 2 holds.

Original: Finally, the item 3 holds obviously since ˆ π t is sampled from N t, 0 .
Revised: Finally, the item 3 holds obviously since ˆ π t is sampled from N t, 0 .

Original: This completes the proof.
Revised: This completes the proof.

Original: We now prove Lemma 1 in Section 2.
Revised: We now prove Lemma 1 in Section 2.

Original: Lemma 5.
Revised: Lemma 5.

Original: Assume { ν ∗ t } Tt =1 and { π ∗ t } Tt =1 are given.
Revised: Assume { ν ∗ t } Tt =1 and { π ∗ t } Tt =1 are given.

Original: For each task t , we run CRPO for M iterations with α =(1 −
Revised: For each task t , we run CRPO for M iterations with α =

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: |
Revised: D

Original: [Equation]
Revised: [Equation]

Original: |A|

Original: [Equation]
Revised: [Equation]

Original: + D ∗ 2 (cid:17) . In addition, the initialization { π t, 0 } Tt =1 are determined by playing FTRL or OGD on the functions E s ∼ ν ∗ t
Revised: In addition, the initialization { π t, 0 } Tt =1 are determined by playing FTRL or OGD on the functions E s ∼ ν ∗ t

Original: [ D KL ( π ∗ t |· )] , for t = 1 , . .
Revised: [ D KL ( π ∗ t |· )] , for t = 1 , . .

Original: , T .
Revised: , T .

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the within-task guarantee for CMDP, we know that ¯ R 0 and { ¯ R i } pi =1 are well-defined.
Revised: By the within-task guarantee for CMDP, we know that ¯ R 0 and { ¯ R i } pi =1 are well-defined.

Original: In addition, it holds that
Revised: In addition, it holds that

Original: [Equation]
Revised: [Equation]

Original: where ϕ t =
Revised: where ϕ t =

Original: π t, 0 .
Revised: π t, 0 .

Original: The first inequality follows from the choice of η t from Lemma 4.
Revised: The first inequality follows from the choice of η t from Lemma 4.

Original: The key step is the last step, which splits the total loss into the loss of the meta-update algorithm and the the loss if we had always initialized at ϕ ∗ .
Revised: The key step is the last step, which splits the total loss into the loss of the meta-update algorithm and the the loss if we had always initialized at ϕ ∗ .

Original: Since each E s ∼ ν ∗ t [ D KL ( π ∗ t |· )] is µ π -strongly convex due to Assumption 1, and each ϕ t is determined by playing OGD, we have that:
Revised: Since each E s ∼ ν ∗ t [ D KL ( π ∗ t |· )] is µ π -strongly convex due to Assumption 1, and each ϕ t is determined by playing OGD, we have that:

Original: [Equation]
Revised: [Equation]

Original: where L g is the upper bound on ∇ ϕ E
Revised: where L g is the upper bound on ∇ ϕ E

Original: s ∼ ν ∗ t [ D KL ( π ∗ t | ϕ )] , µ π is the strong convexity parameter for the KL divergence of the softmax policy.
Revised: s ∼ ν ∗ t [ D KL ( π ∗ t | ϕ )] , µ π is the strong convexity parameter for the KL divergence of the softmax policy.

Original: Since ϕ ∗ = arg min ϕ (cid:80) Tt =1 E s ∼ ν ∗ t [ D KL ( π ∗ t | ϕ )] , by the definition of D ∗ , we have E s ∼ ν ∗ t
Revised: Since ϕ ∗ = arg min ϕ (cid:80) Tt =1 E s ∼ ν ∗ t [ D KL ( π ∗ t | ϕ )] , by the definition of D ∗ , we have E s ∼ ν ∗ t

Original: [ D KL ( π ∗ t | ϕ ∗ )] ≤ D ∗ 2 .
Revised: [ D KL ( π ∗ t | ϕ ∗ )] ≤ D ∗ 2 .

Original: Thus, by substituting the definition of ϕ ∗ , it holds that
Revised: Thus, by substituting the definition of ϕ ∗ , it holds that

Original: [Equation]
Revised: [Equation]

Original: Setting the value of α = (cid:115)(cid:18) 
Revised: Setting the value of α = (cid:115)(cid:18) [Equation] √ [Equation]

Revised: |

Original: [Equation]
Revised: [Equation]

Original:  , we can obtain the TAOG ¯ R 0 as:
Revised: A| , we can obtain the TAOG ¯ R 0 as:

Original: [Equation]
Revised: [Equation]

Original: The bound for ¯
Revised: The bound for ¯

Original: R
Revised: R

Original: i can be derived similarly.
Revised: i can be derived similarly.

Revised: Note: The lemma above shows how the parameters like learning rate α and safety threshold η t can be carefully chosen to achieve decreasing TAOG and TACV in time steps M and tasks T .

Original: E I NEXACT ONLINE GRADIENT DESCENT E. B ASICS FOR ϵ - SUBGRADIENT
Revised: E I NEXACT ONLINE GRADIENT DESCENT E. B ASICS FOR ϵ - SUBGRADIENT

Original: We start with some basics for ϵ -subdifferential used in the subsequent analysis.
Revised: We start with some basics for ϵ -subdifferential used in the subsequent analysis.

Original: This material is based on (Jean-Baptiste, 2010, Chap. XI).
Revised: This material is based on (Jean-Baptiste, 2010, Chap. XI).

Original: Throughout this section, we consider a convex, closed, and proper function f : R d → R ∪ { + ∞} with domain Dom( f ) .
Revised: Throughout this section, we consider a convex, closed, and proper function f : R d → R ∪ { + ∞} with domain Dom( f ) .

Original: We always consider a positive ϵ > 0 .
Revised: We always consider a positive ϵ > 0 .

Original: Definition 2 ( ϵ -subgradient Jean-Baptiste (2010)) .
Revised: Definition 2 ( ϵ -subgradient (Jean-Baptiste, 2010)) .

Original: Given ˆ x ∈ Dom( f ) , the vector u ∈ R d is called ϵ -subgradient of f at ˆ x when the following property holds for any x ∈ R d :
Revised: Given ˆ x ∈ Dom( f ) , the vector u ∈ R d is called ϵ -subgradient of f at ˆ x when the following property holds for any x ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: The set of all ϵ -subgradients of f at ˆ x is the ϵ -subdifferential of f at ˆ x , denoted by ∂ ϵ f
Revised: The set of all ϵ -subgradients of f at ˆ x is the ϵ -subdifferential of f at ˆ x , denoted by ∂ ϵ f

Original: (ˆ x ) .
Revised: (ˆ x ) .

Original: In view of the exact subdifferential ∂f ( x ) , ∂ ϵ f (ˆ x ) can be called an approximate subdifferential, which is a set-valued function with a convex graph.
Revised: In view of the exact subdifferential ∂f ( x ) , ∂ ϵ f (ˆ x ) can be called an approximate subdifferential, which is a set-valued function with a convex graph.

Original: For practical use, ∂ ϵ f
Revised: For practical use, ∂ ϵ f

Original: (ˆ x ) can be used to characterize the ϵ -solution to a convex minimization problem.
Revised: (ˆ x ) can be used to characterize the ϵ -solution to a convex minimization problem.

Original: Lemma 6.
Revised: Lemma 6.

Original: ((Jean-Baptiste, 2010, Thm. 1.1.5))
Revised: ((Jean-Baptiste, 2010, Thm. 1.1.5))

Original: The following two properties are equivalent.
Revised: The following two properties are equivalent.

Original: [Equation]
Revised: [Equation]

Original: One useful result that stems directly from the definition is to link the ϵ -subdifferential of two uniformly close functions (e.g., an expectation of a function and its empirical version).
Revised: One useful result that stems directly from the definition is to link the ϵ -subdifferential of two uniformly close functions (e.g., an expectation of a function and its empirical version).

Original: Lemma 7.
Revised: Lemma 7.

Original: Consider two convex functions f and g , with the property that ∥
Revised: Consider two convex functions f and g , with the property that ∥

Original: f − g ∥ ∞ ≤ ϵ , where ∥ f − g
Revised: f − g ∥ ∞ ≤ ϵ , where ∥ f − g

Original: ∥
Revised: ∥

Original: ∞
Revised: ∞

Original: =
Revised: =

Original: max x | f ( x ) − g ( x ) | .
Revised: max x | f ( x ) − g ( x ) | .

Original: Then, for any x ∈ R d and u ∈ ∂f ( x ) in the subdifferential of f at x , it is also in the 2 ϵ -subdifferential of g at x , i.e., u ∈ ∂ 2 ϵ g
Revised: Then, for any x ∈ R d and u ∈ ∂f ( x ) in the subdifferential of f at x , it is also in the 2 ϵ -subdifferential of g at x , i.e., u ∈ ∂ 2 ϵ g

Original: ( x ) .
Revised: ( x ) .

Original: Proof.
Revised: Proof.

Original: The proof follows directly by convexity and the uniform condition:
Revised: The proof follows directly by convexity and the uniform condition:

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is by convexity of f , and the first and last inequalities are due to the supremum norm condition.
Revised: where the second inequality is by convexity of f , and the first and last inequalities are due to the supremum norm condition.

Original: Our next result is concerned about bounding the distance (measured in ℓ 2 norm) between the true gradient and the ϵ -subgradient of the function, assuming the function is differentiable and smooth.

Original: Input: Learning rate α , x 1 =1: for t = 1 , .., T do 2:
Revised: Input: Learning rate α , x 1 =1: for t = 1 , .., T do 2:

Original: Incur loss ℓ t ( x t ) and compute ϵ -gradient ˆ ∇ t ℓ t ( x t ) 3: x t +1 =
Revised: Incur loss ℓ t ( x t ) and compute ϵ -gradient ˆ ∇ t ℓ t ( x t ) 3: x t +1 =

Original: P X ( x t − α ˆ ∇ t ℓ t ( x t )) 4: end for
Revised: P X ( x t − α ˆ ∇ t ℓ t ( x t )) 4: end for

Revised: Our next result is concerned about bounding the distance (measured in ℓ 2 norm) between the true gradient and the ϵ -subgradient of the function, assuming the function is differentiable and smooth.

Original: Lemma 8.
Revised: Lemma 8.

Original: Suppose a function f is convex, differentiable, and L -smooth over Dom( f ) , and u ∈ ∂ ϵ f ( x ) is an ϵ -subgradient of f at x ∈ Dom( f ) .
Revised: Suppose a function f is convex, differentiable, and L -smooth over Dom( f ) , and u ∈ ∂ ϵ f ( x ) is an ϵ -subgradient of f at x ∈ Dom( f ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: for any C 1 ∈ { c ∈ (0 , 2 L ) : x + c ( u − ∇ f ( x ))
Revised: for any C 1 ∈ { c ∈ (0 , 2 L ) : x + c ( u − ∇ f ( x ))

Original: ∈ Dom( f ) } .
Revised: ∈ Dom( f ) } .

Original: In particular, if x + 1 L ( u − ∇ f ( x )) ∈
Revised: In particular, if x + 1 L ( u − ∇ f ( x )) ∈

Original: Dom( f ) , then ∥ u − ∇ f ( x ) ∥ 22 ≤ 2 ϵL .
Revised: Dom( f ) , then ∥ u − ∇ f ( x ) ∥ 22 ≤ 2 ϵL .

Original: Proof.
Revised: Proof.

Original: Since u is an ϵ -gradient, f ( y ) ≥ f
Revised: Since u is an ϵ -gradient, f ( y ) ≥ f

Original: ( x ) + ⟨ u, y − x ⟩ − ϵ for all y ∈ Dom( f ) .
Revised: ( x ) + ⟨ u, y − x ⟩ − ϵ for all y ∈ Dom( f ) .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Choose y =
Revised: Choose y =

Original: x
Revised: x

Original: + c ( u − ∇ f ( x )) for c ∈ (0 , 2 L ) such that x + c ( u − ∇ f ( x ))
Revised: + c ( u − ∇ f ( x )) for c ∈ (0 , 2 L ) such that x + c ( u − ∇ f ( x ))

Original: ∈ Dom( f ) , we have that
Revised: ∈ Dom( f ) , we have that

Original: [Equation]
Revised: [Equation]

Original: The smoothness condition in the above seems necessary, as we can construct counterexamples that drive the distance of an ϵ -subgradient and its exact counterpart arbitrarily large without the smoothness condition.
Revised: The smoothness condition in the above seems necessary, as we can construct counterexamples that drive the distance of an ϵ -subgradient and its exact counterpart arbitrarily large without the smoothness condition.

Original: In fact, it is known that the set-valued mapping ( x, ϵ ) → ∂ ϵ f ( x ) is inner semi-continuous for a Lipschitz-continuous f , which is implied by the fact that the distance (using the
Revised: In fact, it is known that the set-valued mapping ( x, ϵ ) → ∂ ϵ f ( x ) is inner semi-continuous for a Lipschitz-continuous f , which is implied by the fact that the distance (using the

Original: Hausdorff distance for sets) between any two subdifferential ∂ ϵ f ( x ) and ∂ ϵ ′ f
Revised: Hausdorff distance for sets) between any two subdifferential ∂ ϵ f ( x ) and ∂ ϵ ′ f

Original: ( x ′ ) for all x, x ′ ∈ R d and ϵ, ϵ ′ is positive, and shown to be bounded by O (cid:16)
Revised: ( x ′ ) for all x, x ′ ∈ R d and ϵ, ϵ ′ is positive, and shown to be bounded by O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: -Baptiste [Equation] 2010, Thm. 
Revised: (Jean-Baptiste, 2010, Thm. 4.1.3).

Original: While the exact gradient can be interpreted as ϵ -subgradient driving ϵ → 0 + , the existing bound provided by (Jean-Baptiste, 2010, Thm. 4.1.3) is vacuous in this case; on the other hand, the bound provided in Lemma 8 remains meaningful.
Revised: While the exact gradient can be interpreted as ϵ -subgradient driving ϵ → 0 + , the existing bound provided by (Jean-Baptiste, 2010, Thm. 4.1.3) is vacuous in this case; on the other hand, the bound provided in Lemma 8 remains meaningful.

Original: E. S TATIC REGRET FOR THE INEXACT OGD ALGORITHM
Revised: E. S TATIC REGRET FOR THE INEXACT OGD ALGORITHM

Original: In the following, we consider the online learning setup, where a sequence of loss functions { ℓ t } t ∈ [ T ] are revealed sequentially, and the performance of the OGD algorithm (see Algorithm 2) is measured against a static decision in hindsight:
Revised: In the following, we consider the online learning setup, where a sequence of loss functions { ℓ t } t ∈ [ T ] are revealed sequentially, and the performance of the OGD algorithm (see Algorithm 2) is measured against a static decision in hindsight:

Original: [Equation]
Revised: [Equation]

Original: where { x t ∈ X } t ∈ [ T ] is a sequence of actions played by the online algorithm.
Revised: where { x t ∈ X } t ∈ [ T ] is a sequence of actions played by the online algorithm.

Original: For simplicity, we assume that X belongs to the domains of ℓ t for all t ∈
Revised: For simplicity, we assume that X belongs to the domains of ℓ t for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: Furthermore, we define the following cumulative inexact error bounds:
Revised: Furthermore, we define the following cumulative inexact error bounds:

Original: [Equation]
Revised: [Equation]

Original: where ϵ t corresponds to the inexactness of the ϵ t -subgradient in each round of OGD.
Revised: where ϵ t corresponds to the inexactness of the ϵ t -subgradient in each round of OGD.

Original: Theorem E.1 (Static regret bound for the inexact OGD) .
Revised: Theorem E.1 (Static regret bound for the inexact OGD) .

Original: Assume that { ℓ t } t ∈ [ T ] are convex and
Revised: Assume that { ℓ t } t ∈ [ T ] are convex and

Original: L 2 -smooth, with bounded gradient, i.e., ∥∇ ℓ t ( x ) ∥ 2 ≤ L 1 for all t ∈
Revised: L 2 -smooth, with bounded gradient, i.e., ∥∇ ℓ t ( x ) ∥ 2 ≤ L 1 for all t ∈

Original: [ T ] and all x ∈ X .
Revised: [ T ] and all x ∈ X .

Original: Then, for any comparator x ∈ X , with the stepsize α :=
Revised: Then, for any comparator x ∈ X , with the stepsize α :=

Original: [Equation]
Revised: [Equation]

Original: 2 T , we have that
Revised: 2 T , we have that

Original: [Equation]
Revised: [Equation]

Original: where ϵ t is the amount of inexactness at each step t .
Revised: where ϵ t is the amount of inexactness at each step t .

Original: Proof.
Revised: Proof.

Original: By convexity and the fact that ˆ ∇ t is an ϵ t -subgradient of ℓ t at x t , we have that
Revised: By convexity and the fact that ˆ ∇ t is an ϵ t -subgradient of ℓ t at x t , we have that

Original: [Equation]
Revised: [Equation]

Original: Hence, summing over t = 1 , ..., T , we get
Revised: Hence, summing over t = 1 , ..., T , we get

Original: [Equation]
Revised: [Equation]

Original: To bound the RHS, observe that
Revised: To bound the RHS, observe that

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to the OGD update rule and the nonexpansiveness of the projection operator.
Revised: where the first inequality is due to the OGD update rule and the nonexpansiveness of the projection operator.

Original: Thus, rearranging the terms and exploiting the telescopic sum over t ∈ [ T ] , we have that
Revised: Thus, rearranging the terms and exploiting the telescopic sum over t ∈ [ T ] , we have that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, since ℓ t is L 2 -smooth with bounded gradient, and ˆ ∇ t is an ϵ t -gradient for any t ∈
Revised: Furthermore, since ℓ t is L 2 -smooth with bounded gradient, and ˆ ∇ t is an ϵ t -gradient for any t ∈

Original: [ T ] , by Lemma 8, the following holds:
Revised: [ T ] , by Lemma 8, the following holds:

Original: [Equation]
Revised: [Equation]

Original: where the constant c is specified by Lemma 8.
Revised: where the constant c is specified by Lemma 8.

Original: Hence, combining the above relations, we get
Revised: Hence, combining the above relations, we get

Original: [Equation]
Revised: [Equation]

Original: ϵ
Revised: ϵ

Original: [Equation]
Revised: [Equation]

Original: Let α =
Revised: Let α =

Original: ∥ x ∥ L 1 √ 2 T , then we get the RHS as
Revised: ∥ x ∥ L 1 √ 2 T , then we get the RHS as

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: We can relax the dependence of setting the stepsize on T by using a standard doubling trick (first proposed in Auer et al. (2002), see also, e.g., Balcan et al. ; Khodak et al.
Revised: We can relax the dependence of setting the stepsize on T by using a standard doubling trick (first proposed in (Auer et al., 2002), see also, e.g., (Balcan et al., 2019; Khodak et al., 2019)).

Original: After establishing the static regret for the inexact OGD, we can use this result to obtain the proof of Lemma 2, which gives the static regret if we run inexact OGD over the loss sequences E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] for all t ∈
Revised: After establishing the static regret for the inexact OGD, we can use this result to obtain the proof of Lemma 2, which gives the static regret if we run inexact OGD over the loss sequences E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: Overall, the following regret bound will eventually help us establish the proof of Theorem 3.2, where we will utilize the static regret upper bound for inexact
Revised: Overall, the following regret bound will eventually help us establish the proof of Theorem 3.2, where we will utilize the static regret upper bound for inexact

Original: OGD over the loss sequences E ν ∗ t
Revised: OGD over the loss sequences E ν ∗ t

Original: [ D KL ( π ∗ t | π t, 0 )] for all t ∈
Revised: [ D KL ( π ∗ t | π t, 0 )] for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: Also, we denote the norm of the policy with respect to the state distribution ν as ∥ π ∥ ν = (cid:80) s ∈S ν ( s ) π ( s ) .
Revised: Also, we denote the norm of the policy with respect to the state distribution ν as ∥ π ∥ ν = (cid:80) s ∈S ν ( s ) π ( s ) .

Original: Now we proceed to present the proof for Lemma 2.
Revised: Now we proceed to present the proof for Lemma 2.

Original: Lemma 9.
Revised: Lemma 9.

Original: Denote ℓ t ( π t, 0 ) :
Revised: Denote ℓ t ( π t, 0 ) :

Original: = E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] for all t ∈ [ T ] .
Revised: = E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] for all t ∈ [ T ] .

Original: For any fixed comparator π ∗ 0 = arg min
Revised: For any fixed comparator π ∗ 0 = arg min

Original: [Equation]
Revised: [Equation]

Original: ℓ t
Revised: ℓ t

Original: [Equation]
Revised: [Equation]

Original: ) , if OGD is run on a sequence of loss functions { ˆ ℓ t } t ∈
Revised: ) , if OGD is run on a sequence of loss functions { ˆ ℓ t } t ∈

Original: [ T ] , where ˆ ℓ t :=
Revised: [ T ] , where ˆ ℓ t :=

Original: E ˆ
Revised: E ˆ

Original: [Equation]
Revised: [Equation]

Original: t [ D KL (ˆ π t | π t, 0 )] with the step-size α :=
Revised: t [ D KL (ˆ π t | π t, 0 )] with the step-size α :=

Original: ∥
Revised: ∥

Original: π ∗ 

Revised: π ∗

Original: ∥
Revised: ∥

Original: L g √ 2 T , then the following bound holds for static regret:
Revised: L g √ 2 T , then the following bound holds for static regret:

Original: [Equation]
Revised: [Equation]

Original: for any C 1 ∈ { c ∈
Revised: for any C 1 ∈ { c ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: (cid:17) : π ∗ 

Revised: (cid:17) : π ∗

Original: + c ( ˆ ∇ t − ∇ t ) ∈
Revised: + c ( ˆ ∇ t − ∇ t ) ∈

Original: ∆ A | S | ϱ } where ˆ ∇ t and ∇ t are an ϵ t -subgradient and exact subgradient of E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] at π t, 0 , respectively, E T := (cid:80)
Revised: ∆ A | S | ϱ } where ˆ ∇ t and ∇ t are an ϵ t -subgradient and exact subgradient of E ν ∗ t [ D KL ( π ∗ t | π t, 0 )] at π t, 0 , respectively, E T := (cid:80)

Original: Tt
Revised: Tt

Original: =

Revised: =

Original: ϵ t is the cumulative inexactness.
Revised: ϵ t is the cumulative inexactness.

Original: Proof.
Revised: Proof.

Original: The proof follows directly after substituting c = 4 2 C 1 − C 21 L π and other appropriate constants in Theorem E.1.
Revised: The proof follows directly after substituting c = 4 2 C 1 − C 21 L π and other appropriate constants in Theorem E.1.

Original: Note that the inexactness bound ϵ t can be obtained from Theorem 3.1.
Revised: Note that the inexactness bound ϵ t can be obtained from Theorem 3.1.

Original: Establishing the above
Revised: Establishing the above

Original: Corollary, we can finally provide the proof for Theorem 3.2.
Revised: Corollary, we can finally provide the proof for Theorem 3.2.

Original: Proof of Theorem 3. 
Revised: Proof of Theorem 3.

Original: Theorem E.2.
Revised: Theorem E.2.

Original: Let ˆ D
Revised: Let ˆ D

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 1 E
Revised: 1 E

Original: [Equation]
Revised: [Equation]

Original: [ D KL (ˆ π t | ϕ )] be the empirical task-similarity, and let c 1 = √ 2 L g ∥ ϕ ∗ ∥ , and
Revised: [ D KL (ˆ π t | ϕ )] be the empirical task-similarity, and let c 1 = √ 2 L g ∥ ϕ ∗ ∥ , and

Original: c 2 = (cid:16) 2 + 4 √ 2 L g L π ∥ ϕ ∗ ∥ (2 C− C 21 L π ) √ T (cid:17) , where ϕ ∗ is the fixed optimal metainitialization for all the tasks given by ϕ ∗ = arg min
Revised: c 2 = (cid:16) 2 + 4 √ 2 L g L π ∥ ϕ ∗ ∥ (2 C− C 21 L π ) √ T (cid:17) , where ϕ ∗ is the fixed optimal metainitialization for all the tasks given by ϕ ∗ = arg min

Original: [Equation]
Revised: [Equation]

Original: [ D
Revised: [ D

Original: [Equation]
Revised: [Equation]

Original: (ˆ π t | ϕ )] .
Revised: (ˆ π t | ϕ )] .

Original: For each task t , we run CRPO for M iterations with α = (cid:113)
Revised: For each task t , we run CRPO for M iterations with α = (cid:113)

Original: [Equation]
Revised: [Equation]

Original: + ˆ D ∗ 2 (cid:17) , and we obtain { ˆ ν t } Tt =1 and { ˆ π t } Tt =1 .
Revised: + ˆ D ∗ 2 (cid:17) , and we obtain { ˆ ν t } Tt =1 and { ˆ π t } Tt =1 .

Original: In addition, the initialization { π t, 0 } Tt =1 are determined by playing OGD on the functions
Revised: In addition, the initialization { π t, 0 } Tt =1 are determined by playing OGD on the functions

Original: E s ∼ ˆ ν t
Revised: E s ∼ ˆ ν t

Original: [ D KL (ˆ π t |· )] , for t = 1 , . .
Revised: [ D KL (ˆ π t |· )] , for t = 1 , . .

Original: , T .
Revised: , T .

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We know that ¯ R 0 and { ¯ R i } pi =1 are well-defined.
Revised: We know that ¯ R 0 and { ¯ R i } pi =1 are well-defined.

Original: In addition, it holds that
Revised: In addition, it holds that

Original: [Equation]
Revised: [Equation]

Original: where ϕ t =
Revised: where ϕ t =

Original: π t, 0 .
Revised: π t, 0 .

Original: Second equality follows from the fact that the total loss can be split into the loss of the meta-update algorithm and the the loss if we had always initialized at ϕ ∗ .
Revised: Second equality follows from the fact that the total loss can be split into the loss of the meta-update algorithm and the the loss if we had always initialized at ϕ ∗ .

Original: Last inequality follows from the KL-divergence estimation error bound in Theorem 3.1.
Revised: Last inequality follows from the KL-divergence estimation error bound in Theorem 3.1.

Original: Since each E s ∼ ˆ ν t [ D KL (ˆ π t |· )] is µ π -strongly convex due to Assumption 1, and since each ϕ t is determined by playing FTL or inexact OGD, the following term can be upper bounded using Lemma 9 as follows:
Revised: Since each E s ∼ ˆ ν t [ D KL (ˆ π t |· )] is µ π -strongly convex due to Assumption 1, and since each ϕ t is determined by playing FTL or inexact OGD, the following term can be upper bounded using Lemma 9 as follows:

Original: [Equation]
Revised: [Equation]

Original: where the constants are from the Corollary 9.
Revised: where the constants are from the Corollary 9.

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we will upper bound the second term.
Revised: we will upper bound the second term.

Original: Since ϕ ∗ = arg min ϕ 1 T
Revised: Since ϕ ∗ = arg min ϕ 1 T

Original: [Equation]
Revised: [Equation]

Original: Tt =1 E s ∼ ˆ
Revised: Tt =1 E s ∼ ˆ

Original: ν t [ D KL (ˆ π t | ϕ )]
Revised: ν t [ D KL (ˆ π t | ϕ )]

Original: [Equation]
Revised: [Equation]

Original: by the definition of ˆ D ∗
Revised: by the definition of ˆ D ∗

Original: [Equation]
Revised: [Equation]

Original: we have E s ∼ ˆ ν t
Revised: we have E s ∼ ˆ ν t

Original: [ D KL (ˆ π t | ϕ )]
Revised: [ D KL (ˆ π t | ϕ )]

Original: ≤ ˆ D ∗ 2 .
Revised: ≤ ˆ D ∗ 2 .

Original: Thus, by substituting the definition of ϕ ∗ , it holds that
Revised: Thus, by substituting the definition of ϕ ∗ , it holds that

Original: [Equation]
Revised: [Equation]

Original: Setting the value of α
Revised: Setting the value of α

Original: [Equation]
Revised: [Equation]

Original: where c
Revised: where c

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: we can obtain the TAOG ¯ R 0 as follows:
Revised: we can obtain the TAOG ¯ R 0 as follows:

Original: [Equation]
Revised: [Equation]

Original: The bound for ¯
Revised: The bound for ¯

Original: R
Revised: R

Original: i can be derived similarly.
Revised: i can be derived similarly.

Original: E. D YNAMIC REGRET FOR THE INEXACT OGD ALGORITHM
Revised: E. D YNAMIC REGRET FOR THE INEXACT OGD ALGORITHM

Original: In the following, we consider a stronger notion of regret that measures the performance of the
Revised: In the following, we consider a stronger notion of regret that measures the performance of the

Original: OGD algorithm (see Algorithm 2) against a dynamically changing sequence in hindsight (see, e.g., Zinkevich (2003); Jadbabaie et al. ; Zhang et al. [Equation]
Revised: OGD algorithm (see Algorithm 2) against a dynamically changing sequence in hindsight (see, e.g., (Zinkevich, 2003; Jadbabaie et al., 2015; Zhang et al., 2017)):

Original: [Equation]
Revised: [Equation]

Original: where x ∗ t ∈ arg min x ∈ X ℓ t ( x ) is the optimal decision for the loss ℓ t .
Revised: where x ∗ t ∈ arg min x ∈ X ℓ t ( x ) is the optimal decision for the loss ℓ t .

Original: It is well-known that in the worst-case, it is impossible to achieve a sub-linear dynamic regret bound, due to the arbitrary fluctuation in the functions Zinkevich (2003); Besbes et al. ; Yang et al.
Revised: It is well-known that in the worst-case, it is impossible to achieve a sub-linear dynamic regret bound, due to the arbitrary fluctuation in the functions (Zinkevich, 2003; Besbes et al., 2015; Yang et al., 2016).

Original: Thus, it is common to upper bound the dynamic regret in terms of certain regularity of the comparator sequence.
Revised: Thus, it is common to upper bound the dynamic regret in terms of certain regularity of the comparator sequence.

Original: One possible regularity condition is the path-length of the comparator sequence Zinkevich (2003);
Revised: One possible regularity condition is the path-length of the comparator sequence (Zinkevich, 2003;

Original: Jadbabaie et al.
Revised: Jadbabaie et al., 2015):

Original: [Equation]
Revised: [Equation]

Original: (14) which captures the cumulative Euclidean norm of the difference between successive comparators (note that we will use ∥ · ∥ for the Euclidean norm, unless otherwise specified).
Revised: 14) which captures the cumulative Euclidean norm of the difference between successive comparators (note that we will use ∥ · ∥ for the Euclidean norm, unless otherwise specified).

Original: The path-length measure is also the regularity condition used in existing inexact OGD literature Bedi et al.
Revised: The path-length measure is also the regularity condition used in existing inexact OGD literature (Bedi et al., 2018;

Original: Dixit et al.
Revised: Dixit et al., 2019).

Original: However, as remarked in Zhang et al. (2017), a potentially tighter bound can be achieved by examining the squared path-length measure:
Revised: However, as remarked in (Zhang et al., 2017), a potentially tighter bound can be achieved by examining the squared path-length measure:

Original: [Equation]
Revised: [Equation]

Original: (15) which can be much smaller than P T when the local variations are small.
Revised: (15) which can be much smaller than P T when the local variations are small.

Original: For example, when ∥ x ∗ t − x ∗ t − 1 ∥ = Θ(1 / √ T ) for all t ∈ [ T ] , we have P T =
Revised: For example, when ∥ x ∗ t − x ∗ t − 1 ∥ = Θ(1 / √ T ) for all t ∈ [ T ] , we have P T =

Original: Θ( √ T ) but S T = Θ(1) .
Revised: Θ( √ T ) but S T = Θ(1) .

Original: In this section, we provide analysis with respect to both measures for strongly convex and smooth functions.
Revised: In this section, we provide analysis with respect to both measures for strongly convex and smooth functions.

Original: Furthermore, we propose to apply inexact OGD multiple times in each round, and demonstrate that the dynamic regret is reduced from O ( P T + E T ) to O (min {P T + E T , S T + ˜ E T } ) , where
Revised: Furthermore, we propose to apply inexact OGD multiple times in each round, and demonstrate that the dynamic regret is reduced from O ( P T + E T ) to O (min {P T + E T , S T + ˜ E T } ) , where

Original: [Equation]
Revised: [Equation]

Original: is the cumulative square root inexactness bounds.
Revised: is the cumulative square root inexactness bounds.

Original: Note that our results improve over existing bounds for inexact online learning Bedi et al. ; Dixit et al. and can be regarded as a generalization of Zhang et al. (2017) to the inexact settings.
Revised: Note that our results improve over existing bounds for inexact online learning (Bedi et al., 2018; Dixit et al., 2019) and can be regarded as a generalization of (Zhang et al., 2017) to the inexact settings.

Original: We start with a result that will be used in later analysis.
Revised: We start with a result that will be used in later analysis.

Original: Lemma 10.
Revised: Lemma 10.

Original: Assume that f : X → R is λ -strongly convex and L -smooth, and let x ∗ = arg min x ∈ X f ( x ) be the unique optimal solution.
Revised: Assume that f : X → R is λ -strongly convex and L -smooth, and let x ∗ = arg min x ∈ X f ( x ) be the unique optimal solution.

Original: Let v = P X
Revised: Let v = P X

Original: ( x − α ˆ ∇ f ( x )) ,
Revised: ( x − α ˆ ∇ f ( x )) ,

Original: where ˆ ∇ f ( u ) ∈ ∂ ϵ f ( u ) and α ≤ 12 L , we have that
Revised: where ˆ ∇ f ( u ) ∈ ∂ ϵ f ( u ) and α ≤ 12 L , we have that

Original: [Equation]
Revised: [Equation]

Original: where the constant c is specified in Lemma 8.
Revised: where the constant c is specified in Lemma 8.

Original: Proof.
Revised: Proof.

Original: By the update rule, we have that
Revised: By the update rule, we have that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By strong convexity of the objective above,
Revised: By strong convexity of the objective above,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Since, f ( x ) is λ -strongly convex and L -smooth, we have that
Revised: Since, f ( x ) is λ -strongly convex and L -smooth, we have that

Original: [Equation]
Revised: [Equation]

Original: 18) and
Revised: 18) and

Original: [Equation]
Revised: [Equation]

Original: Also, since ˆ ∇ f ( x ) is an ϵ -subgradient, we can write
Revised: Also, since ˆ ∇ f ( x ) is an ϵ -subgradient, we can write

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Input: Learning rate α , x 1 =1: for t = 1 , .., T do 2:
Revised: Input: Learning rate α , x 1 =1: for t = 1 , .., T do 2:

Original: Incur loss ℓ t ( x t ) 3: z 1 t = x t 4: for k = 1 , ..., K do 5: z k +1 t =
Revised: Incur loss ℓ t ( x t ) 3: z 1 t = x t 4: for k = 1 , ..., K do 5: z k +1 t =

Original: P X ( x t − α ˆ ∇ ℓ t ( z kt )) 6: end for 7: x t +1 =
Revised: P X ( x t − α ˆ ∇ ℓ t ( z kt )) 6: end for 7: x t +1 =

Original: z K +1 t 8: end for
Revised: z K +1 t 8: end for

Original: Combining 18, 19 and 20, we have that
Revised: Combining 18, 19 and 20, we have that

Original: [Equation]
Revised: [Equation]

Original: Combining the above relations, we have that
Revised: Combining the above relations, we have that

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to L -smoothness, ( i ) follows from 17, ( ii ) is due to convexity, ( iii ) is due to strong convexity, ( iv ) follows from Cauchy-Schwarz inequality, and ( v ) is due to the inequality ab ≤ 12 κ a 2 + κ 2 b 2 for a, b ≥ 0 and κ > 0 and the constant c comes from Lemma 8.
Revised: where the first inequality is due to L -smoothness, ( i ) follows from 17, ( ii ) is due to convexity, ( iii ) is due to strong convexity, ( iv ) follows from Cauchy-Schwarz inequality, and ( v ) is due to the inequality ab ≤ 12 κ a 2 + κ 2 b 2 for a, b ≥ 0 and κ > 0 and the constant c comes from Lemma 8.

Original: Choosing κ =
Revised: Choosing κ =

Original: L , α ≤ 12 L , and rearranging the above, we have then proved the claim.
Revised: L , α ≤ 12 L , and rearranging the above, we have then proved the claim.

Original: Theorem E.3 (Dynamic regret for inexact OGD with multiple updates) .
Revised: Theorem E.3 (Dynamic regret for inexact OGD with multiple updates) .

Original: Assume that ℓ t : X → R is λstrongly convex, L 1 -Lipschitz, and L 2 -smooth for all t ∈
Revised: Assume that ℓ t : X → R is λstrongly convex, L 1 -Lipschitz, and L 2 -smooth for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: By setting α ≤ 12 L 2 , K := ⌈ ln2 ln(1+ λα )
Revised: By setting α ≤ 12 L 2 , K := ⌈ ln2 ln(1+ λα )

Original: ⌉ , then, for any β > 0 , we have that
Revised: ⌉ , then, for any β > 0 , we have that

Original: [Equation]
Revised: [Equation]

Original: where C 1 = 2( L 2 + β ) , C 2 =
Revised: where C 1 = 2( L 2 + β ) , C 2 =

Original: ( L 2 + β ) 3 cα +6 αL 2 2 λαL 2 , C 3 = 

Revised: ( L 2 + β ) 3 cα +6 αL 2 2 λαL 2 , C 3 =

Original: ( L 2 + β ) , C 4 = 2 L 1 2 −√ 2 and C 5 =
Revised: ( L 2 + β ) , C 4 = 2 L 1 2 −√ 2 and C 5 =

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof has two parts, where we use different techniques to bound the dynamic regret by
Revised: The proof has two parts, where we use different techniques to bound the dynamic regret by

Original: S T and E T , as well as P T and ˜ E T .
Revised: S T and E T , as well as P T and ˜ E T .

Original: Then the final bound is obtained by taking the minimum between the two bounds.
Revised: Then the final bound is obtained by taking the minimum between the two bounds.

Original: Bounding the dynamic regret by S T and E T .
Revised: Bounding the dynamic regret by S T and E T .

Original: Since ℓ t is L 2 -smooth, we have that
Revised: Since ℓ t is L 2 -smooth, we have that

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to Cauchy–Schwartz and the third inequality is due to ab ≤ 12 β a 2 + β 2 b 2 for a, b ≥ 0 and β > 0 .
Revised: where the second inequality is due to Cauchy–Schwartz and the third inequality is due to ab ≤ 12 β a 2 + β 2 b 2 for a, b ≥ 0 and β > 0 .

Original: Now, using
Revised: Now, using

Original: [Equation]
Revised: [Equation]

Original: we can bound
Revised: we can bound

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Recall the updating rule z j +1 t − 1 =
Revised: Recall the updating rule z j +1 t − 1 =

Original: P X ( z jt − 1 − α ˆ ∇ f t − 1 ( z jt − 1 )) , j = 1 , ..., K ; then, we can write that
Revised: P X ( z jt − 1 − α ˆ ∇ f t − 1 ( z jt − 1 )) , j = 1 , ..., K ; then, we can write that

Original: [Equation]
Revised: [Equation]

Original: where we recursively apply the result from Lemma 10.
Revised: where we recursively apply the result from Lemma 10.

Original: Thus, by plugging in 25 into
Revised: Thus, by plugging in 25 into

Original: [Equation]
Revised: [Equation]

Original: and using the definitions of S T and S T
Revised: and using the definitions of S T and S T

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms, the above relation implies that
Revised: Rearranging the terms, the above relation implies that

Original: [Equation]
Revised: [Equation]

Original: Let ι = 12 and choose K
Revised: Let ι = 12 and choose K

Original: = ⌈
Revised: = ⌈

Original: [Equation]
Revised: [Equation]

Original: 1+ λα )
Revised: 1+ λα )

Original: ⌉ , we have
Revised: ⌉ , we have

Original: [Equation]
Revised: [Equation]

Original: Combine the above with 23, and summing over t ∈ [ T ] , we have that
Revised: Combine the above with 23, and summing over t ∈ [ T ] , we have that

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: which holds true for any positive β > 0 .
Revised: which holds true for any positive β > 0 .

Original: Bounding the dynamic regret by P T and ˜ E T .
Revised: Bounding the dynamic regret by P T and ˜ E T .

Original: By 26 and the choice of K = ⌈
Revised: By 26 and the choice of K = ⌈

Original: [Equation]
Revised: [Equation]

Original: log(1+ λα )
Revised: log(1+ λα )

Original: ⌉ , we have that:
Revised: ⌉ , we have that:

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: ∥ x
Revised: ∥ x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last inequlity follows from √ a + b ≤ √ a + √ b .
Revised: where the last inequlity follows from √ a + b ≤ √ a + √ b .

Original: Due to the bounded gradient assumption, we have that
Revised: Due to the bounded gradient assumption, we have that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To bound (cid:80) Tt =1 ∥ x t − x ∗ t ∥ , notice that
Revised: To bound (cid:80) Tt =1 ∥ x t − x ∗ t ∥ , notice that

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Plugging the above in 28 proves the claim.
Revised: Plugging the above in 28 proves the claim.

Original: In the above result, the number of OGD updates per round is on the order of O
Revised: In the above result, the number of OGD updates per round is on the order of O

Original: ( L 2 /α ) , where L 2 /α is the condition number of each loss function.
Revised: ( L 2 /α ) , where L 2 /α is the condition number of each loss function.

Original: Below, we also provide a dynamic regret bound for standard OGD (single update per round); as a result, we only provide the bound in terms of P T (similar to Jadbabaie et al. ; Mokhtari et al.
Revised: Below, we also provide a dynamic regret bound for standard OGD (single update per round); as a result, we only provide the bound in terms of P T (similar to (Jadbabaie et al., 2015; Mokhtari et al., 2016).

Original: After establishing the dynamic regret for the inexact OGD, we can use this result to obtain the proof of Lemma 3 in the main paper, which provides the dynamic regret of inexact OGD over the loss sequences E ν ∗ t [ D KL ( π ∗ t | ϕ t )] for all t ∈
Revised: After establishing the dynamic regret for the inexact OGD, we can use this result to obtain the proof of Lemma 3 in the main paper, which provides the dynamic regret of inexact OGD over the loss sequences E ν ∗ t [ D KL ( π ∗ t | ϕ t )] for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: Here, we present the full statement with constants for the Lemma 3.
Revised: Here, we present the full statement with constants for the Lemma 3.

Original: Lemma 11 (Dynamic regret bound for inexact OGD) .
Revised: Lemma 11 (Dynamic regret bound for inexact OGD) .

Original: Denote ℓ t ( ϕ t ) :=
Revised: Denote ℓ t ( ϕ t ) :=

Original: E ν ∗ t [ D KL ( π ∗ t | ϕ t )] for all t ∈
Revised: E ν ∗ t [ D KL ( π ∗ t | ϕ t )] for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: For any dynamically varying comparator ψ ∗ t = arg min
Revised: For any dynamically varying comparator ψ ∗ t = arg min

Original: [Equation]
Revised: [Equation]

Original: )] if OGD is run on a sequence of loss functions ˆ ℓ t ( ϕ t ) , where ˆ ℓ t ( ϕ t ) =
Revised: )] if OGD is run on a sequence of loss functions ˆ ℓ t ( ϕ t ) , where ˆ ℓ t ( ϕ t ) =

Original: E ν ∗ t [ D KL (ˆ π t | ϕ t )] for all t ∈
Revised: E ν ∗ t [ D KL (ˆ π t | ϕ t )] for all t ∈

Original: [ T ] with the step-size α ≤ 12 µ π , number of iterations K := ⌈ ln2 ln(1+ µ π α )
Revised: [ T ] with the step-size α ≤ 12 µ π , number of iterations K := ⌈ ln2 ln(1+ µ π α )

Original: ⌉ then the following bound holds for dynamic regret for any β > 0 :
Revised: ⌉ then the following bound holds for dynamic regret for any β > 0 :

Original: [Equation]
Revised: [Equation]

Original: where C 1 = 2( L π + β ) , C 2 =
Revised: where C 1 = 2( L π + β ) , C 2 =

Original: ( L π + β ) 3 C 6 α +6 αL π 2 µ π αL π , C 3 = 3( L π + β ) , C
Revised: ( L π + β ) 3 C 6 α +6 αL π 2 µ π αL π , C 3 = 3( L π + β ) , C

Original: 4 = 2 L g 2 −√ 2 , and C 5 = 2 L g 2 −√ 2 (cid:113) C 

Revised: 4 = 2 L g 2 −√ 2 , and C 5 = 2 L g 2 −√ 2 (cid:113) C

Original: α +2 L π α 2 αµ π L π , for any C 6 ∈ { c ∈ (cid:16) 0 , 2 L π (cid:17) : ψ ∗ t + c ( ˆ ∇ t −∇ t ) ∈
Revised: α +2 L π α 2 αµ π L π , for any C 6 ∈ { c ∈ (cid:16) 0 , 2 L π (cid:17) : ψ ∗ t + c ( ˆ ∇ t −∇ t ) ∈

Original: ∆ A | S | ϱ } where ˆ ∇ t and ∇ t are an ϵ t -subgradient and exact subgradient of E ν ∗ t [ D KL ( π ∗ t | ψ t )] at ψ t , respectively, E T :=
Revised: ∆ A | S | ϱ } where ˆ ∇ t and ∇ t are an ϵ t -subgradient and exact subgradient of E ν ∗ t [ D KL ( π ∗ t | ψ t )] at ψ t , respectively, E T :=

Original: (cid:80) Tt
Revised: (cid:80) Tt

Original: =

Revised: =

Original: ϵ t is the cumulative inexactness.
Revised: ϵ t is the cumulative inexactness.

Original: Proof.
Revised: Proof.

Original: The proof directly follows after plugging in the constants from Theorem E.3.
Revised: The proof directly follows after plugging in the constants from Theorem E.3.

Original: F KL DIVERGENCE ESTIMATION ERROR BOUND
Revised: F KL DIVERGENCE ESTIMATION ERROR BOUND

Original: We recall the following notations.
Revised: We recall the following notations.

Original: For each task t , the initial state distribution is denoted by ρ t , the state distribution for the optimal policy π ∗ t is given by ν ∗ t , the state distribution for the policy ˆ π t is denoted by ˜ ν t , and the state distribution estimated using the trajectory sample dataset D t is denoted as ˆ ν t .
Revised: For each task t , the initial state distribution is denoted by ρ t , the state distribution for the optimal policy π ∗ t is given by ν ∗ t , the state distribution for the policy ˆ π t is denoted by ˜ ν t , and the state distribution estimated using the trajectory sample dataset D t is denoted as ˆ ν t .

Original: In the main paper, we breakdown the KL divergence estimation error by the sources of origin:
Revised: In the main paper, we breakdown the KL divergence estimation error by the sources of origin:

Original: [Equation]
Revised: [Equation]

Original: where ( A ) accounts for the mismatch between the discounted state visitation distributions of an optimal policy π ∗ t and a suboptimal one ˆ π t , ( B ) originates from the estimation error of DualDICE, and ( C ) is due to the difference between π ∗ t and ˆ π t measured according to ˆ π t .
Revised: where ( A ) accounts for the mismatch between the discounted state visitation distributions of an optimal policy π ∗ t and a suboptimal one ˆ π t , ( B ) originates from the estimation error of DualDICE, and ( C ) is due to the difference between π ∗ t and ˆ π t measured according to ˆ π t .

Original: By triangle inequality, we can bound the total error by controlling each term separately.
Revised: By triangle inequality, we can bound the total error by controlling each term separately.

Original: To streamline the presentation, we consider the tabular setting with softmax parametrization.
Revised: To streamline the presentation, we consider the tabular setting with softmax parametrization.

Original: To bound ( A ) , we need to control the distance between ν ∗ t and ˜ ν t , which can be bounded by the distance between the inducing policy parameters as long as they are Lipschitz continuous (Xu et al., 2020, Lemma 3).
Revised: To bound ( A ) , we need to control the distance between ν ∗ t and ˜ ν t , which can be bounded by the distance between the inducing policy parameters as long as they are Lipschitz continuous (Xu et al., 2020, Lemma 3).

Original: In addition, the bound on ( C ) also depends on the distance between policies.
Revised: In addition, the bound on ( C ) also depends on the distance between policies.

Original: In general, controlling the distance between a policy to an optimal policy based on the suboptimality gap requires the optimization to have some curvatures around the optima (e.g., quadratic growth Drusvyatskiy & Lewis (2018) or Hölderian growth Johnstone & Moulin (2020)).
Revised: In general, controlling the distance between a policy to an optimal policy based on the suboptimality gap requires the optimization to have some curvatures around the optima (e.g., quadratic growth (Drusvyatskiy & Lewis, 2018) or Hölderian growth (Johnstone & Moulin, 2020)).

Original: However, to the best of knowledge of the authors, the only available results are an algorithm-dependent PL inequalities for policy gradient Mei et al. (2020) or quadratic growth with entropy regularization Ding et al. (2021b).
Revised: However, to the best of knowledge of the authors, the only available results are an algorithm-dependent PL inequalities for policy gradient (Mei et al., 2020) or quadratic growth with entropy regularization (Ding et al., 2021b).

Original: F. P RELIMINARIES ON TAME GEOMETRY
Revised: F. P RELIMINARIES ON TAME GEOMETRY

Original: For the sake of completeness, let us recall some fundamental concepts/results in tame geometry, which allows us to study the global geometry of the solution maps of a wide range of optimization problems, which will be used in bounding the estimation error for the KL divergence.
Revised: For the sake of completeness, let us recall some fundamental concepts/results in tame geometry, which allows us to study the global geometry of the solution maps of a wide range of optimization problems, which will be used in bounding the estimation error for the KL divergence.

Original: More information can be found in (Davis et al., 2020; Van den Dries & Miller, 1996).
Revised: More information can be found in (Davis et al., 2020; Van den Dries & Miller, 1996).

Original: Recall that a class of functions on a bounded set is called C p smooth when it possesses the uniformly bounded partial derivatives up to order p .
Revised: Recall that a class of functions on a bounded set is called C p smooth when it possesses the uniformly bounded partial derivatives up to order p .

Original: Definition 3 (Whitney Stratification) .
Revised: Definition 3 (Whitney Stratification) .

Original: A Whitney C k stratification of a set I is a partition of I into finitely many nonempty C k manifolds, called strata, satisfying the following compatibility conditions:
Revised: A Whitney C k stratification of a set I is a partition of I into finitely many nonempty C k manifolds, called strata, satisfying the following compatibility conditions:

Original: For any two strata I a
Revised: For any two strata I a

Original: and I b , the implication I a ∩ I b ̸
Revised: and I b , the implication I a ∩ I b ̸

Original: = ∅ implies that I a ⊂ cl I b holds, where cl I b denotes the closure of the set I b .
Revised: = ∅ implies that I a ⊂ cl I b holds, where cl I b denotes the closure of the set I b .

Original: For any sequence of points x k in a stratum I a , converging to a point x
Revised: For any sequence of points x k in a stratum I a , converging to a point x

Original: ⋆ in a stratum I b , if the corresponding normal vectors v k ∈ N I a ( x k )
Revised: ⋆ in a stratum I b , if the corresponding normal vectors v k ∈ N I a ( x k )

Original: converge to a
Revised: converge to a

Original: vector v ,
Revised: vector v ,

Original: then the inclusion
Revised: then the inclusion

Original: v
Revised: v

Original: ∈
Revised: ∈

Original: N
Revised: N

Original: I
Revised: I

Original: b (
Revised: b (

Original: x ⋆ ) holds.
Revised: x ⋆ ) holds.

Original: Here
Revised: Here

Original: N
Revised: N

Original: I
Revised: I

Original: a (
Revised: a (

Original: x k ) denotes the normal cone to I a at x k
Revised: x k ) denotes the normal cone to I a at x k

Original: Roughly speaking, stratification is a locally finite partition of a given set into differentiable manifolds, which fit together in a regular manner (property 1 in Def. 3).
Revised: Roughly speaking, stratification is a locally finite partition of a given set into differentiable manifolds, which fit together in a regular manner (property 1 in Def. 3).

Original: Whitney stratification as defined above is a special type of stratification for which the strata are such that their tangent spaces (as viewed from normal cones) also fit regularly (property 2 ).
Revised: Whitney stratification as defined above is a special type of stratification for which the strata are such that their tangent spaces (as viewed from normal cones) also fit regularly (property 2 ).

Original: There are several paths to verifying Whitney stratifiability.
Revised: There are several paths to verifying Whitney stratifiability.

Original: For instance, one can show that the function under study belongs to one of the well-known function classes, such as semialgebraic functions (Davis et al., 2020), whose members are known to be Whitney stratifiable.
Revised: For instance, one can show that the function under study belongs to one of the well-known function classes, such as semialgebraic functions (Davis et al., 2020), whose members are known to be Whitney stratifiable.

Original: However, to study the solution function of a general convex optimization problem, we need a far-reaching axiomatic extension of semialgebraic sets to classes of functions definable on “o-minimal structures,” which are very general classes and share several attractive analytic features as semialgebraic sets, including Whitney stratifiability (Davis et al., 2020; Van den Dries & Miller, 1996).
Revised: However, to study the solution function of a general convex optimization problem, we need a far-reaching axiomatic extension of semialgebraic sets to classes of functions definable on “o-minimal structures,” which are very general classes and share several attractive analytic features as semialgebraic sets, including Whitney stratifiability (Davis et al., 2020; Van den Dries & Miller, 1996).

Original: Definition 4 (o-minimal structure) .
Revised: Definition 4 (o-minimal structure) .

Original: (Van den Dries & Miller, 1996)
Revised: (Van den Dries & Miller, 1996)

Original: An o-minimal structure is defined as a sequence of Boolean algebras O v of subsets of R v , such that for each n v ∈ N , the following properties hold:
Revised: An o-minimal structure is defined as a sequence of Boolean algebras O v of subsets of R v , such that for each n v ∈ N , the following properties hold:

Original: If some set X belongs to O v , then X × R belong to O v +1 .
Revised: If some set X belongs to O v , then X × R belong to O v +1 .

Original: Let P proj : R v × R
Revised: Let P proj : R v × R

Original: → R v denote the coordinate projection operator onto R v , then for any X in O v
Revised: → R v denote the coordinate projection operator onto R v , then for any X in O v

Original: +1 , the set P proj ( X ) belongs to O v .
Revised: +1 , the set P proj ( X ) belongs to O v .

Original: O v contains all sets of the form { x ∈ R v : y ( x ) = 0 } , where y ( x ) is a polynomial in R v . 4.
Revised: O v contains all sets of the form { x ∈ R v : y ( x ) = 0 } , where y ( x ) is a polynomial in R v . 4.

Original: The elements of O 1 are exactly the finite unions of intervals (possibly infinite) and points.
Revised: The elements of O 1 are exactly the finite unions of intervals (possibly infinite) and points.

Original: Then all the sets that belong to O v are called definable in the o-minimal structure.
Revised: Then all the sets that belong to O v are called definable in the o-minimal structure.

Original: Definable sets have broader applicability than semialgebraic sets (in the sense that the latter is a special kind of definable sets) but enjoys the same, remarkable stability property: the composition of definable mappings (including sum, inf-convolution, and several other classical operations of analysis involving a finite number of definable objects) in some o-minimal structure remains in the same structure.
Revised: Definable sets have broader applicability than semialgebraic sets (in the sense that the latter is a special kind of definable sets) but enjoys the same, remarkable stability property: the composition of definable mappings (including sum, inf-convolution, and several other classical operations of analysis involving a finite number of definable objects) in some o-minimal structure remains in the same structure.

Original: We will crucially exploit these properties in the following sections.
Revised: We will crucially exploit these properties in the following sections.

Original: F. B ASIC PROPERTIES OF SUBGRADIENT FLOW SYSTEMS
Revised: F. B ASIC PROPERTIES OF SUBGRADIENT FLOW SYSTEMS

Original: We also recall some basic definitions and properties of the subgradient flow system (see, e.g., (Bolte et al., 2010, Thm. 13)).
Revised: We also recall some basic definitions and properties of the subgradient flow system (see, e.g., (Bolte et al., 2010, Thm. 13)).

Original: Let f : R d → R ∪ { + ∞} be a proper lower semicontinuous function.
Revised: Let f : R d → R ∪ { + ∞} be a proper lower semicontinuous function.

Original: Definition 5 (Subgradient flow system) .
Revised: Definition 5 (Subgradient flow system) .

Original: For every x ∈ dom( f ) , there exists a unique absolutely continuous curve (called trajectory or subgradient curve) θ ( τ ) : [0 , + ∞ )
Revised: For every x ∈ dom( f ) , there exists a unique absolutely continuous curve (called trajectory or subgradient curve) θ ( τ ) : [0 , + ∞ )

Original: → R d that satisfies
Revised: → R d that satisfies

Original: [Equation]
Revised: [Equation]

Original: Moreover, the trajectory also satisfies the following properties (Bolte et al., 2010, Thm. 13):
Revised: Moreover, the trajectory also satisfies the following properties (Bolte et al., 2010, Thm. 13):

Original: θ ( τ ) ∈ dom( ∂f ) for all τ ∈ (0 , + ∞ ) .
Revised: θ ( τ ) ∈ dom( ∂f ) for all τ ∈ (0 , + ∞ ) .

Original: For all τ > 0 , the right derivative ˙ θ ( τ + ) is well defined and equal to
Revised: For all τ > 0 , the right derivative ˙ θ ( τ + ) is well defined and equal to

Original: [Equation]
Revised: [Equation]

Original: where ∂ 0 f ( θ ) is the minimum norm subgradient in ∂f ( θ ) .
Revised: where ∂ 0 f ( θ ) is the minimum norm subgradient in ∂f ( θ ) .

Original: In particular, we have that
Revised: In particular, we have that

Original: ˙ θ ( τ ) = − ∂ 0 f
Revised: ˙ θ ( τ ) = − ∂ 0 f

Original: ( θ ( τ )) , for almost all τ .
Revised: ( θ ( τ )) , for almost all τ .

Original: [Table]
Revised: [Table]

Original: Recall Assumption 2, which requires that the objective/constraint functions and policy parametrization are definable in some o-minimal structure (Van den Dries & Miller, 1996).
Revised: Recall Assumption 2, which requires that the objective/constraint functions and policy parametrization are definable in some o-minimal structure (Van den Dries & Miller, 1996).

Original: This is a mild assumption as practically all functions from real-world applications, including deep neural networks, are definable in some o-minimal structure Davis et al. (2020); also, the composition of mappings, along with the sum, inf-convolution, and several other classical operations of analysis involving a finite number of definable objects in some o-minimal structure remains in the same structure Van den Dries & Miller (1996).
Revised: This is a mild assumption as practically all functions from real-world applications, including deep neural networks, are definable in some o-minimal structure (Davis et al., 2020); also, the composition of mappings, along with the sum, inf-convolution, and several other classical operations of analysis involving a finite number of definable objects in some o-minimal structure remains in the same structure (Van den Dries & Miller, 1996).

Original: The far reaching consequence of definability, exploited in this study, is that definable sets and functions admit, for each k ≥ 1 , a C k –Whitney stratification with finitely many strata (see, for instance, (Van den Dries & Miller, 1996, Result 4.8)).
Revised: The far reaching consequence of definability, exploited in this study, is that definable sets and functions admit, for each k ≥ 1 , a C k –Whitney stratification with finitely many strata (see, for instance, (Van den Dries & Miller, 1996, Result 4.8)).

Original: This remarkable property, combined with the result that any stratifiable functions enjoys a nonsmooth Kurdyka–Łojasiewicz inequality Bolte et al. (2007), provides the foundation to bound the distance ∥
Revised: This remarkable property, combined with the result that any stratifiable functions enjoys a nonsmooth Kurdyka–Łojasiewicz inequality (Bolte et al., 2007), provides the foundation to bound the distance ∥

Original: π ∗ t − ˆ π t ∥ by the suboptimality gap.
Revised: π ∗ t − ˆ π t ∥ by the suboptimality gap.

Original: Note that without further specifications, π ∗ t is understood as one of optimal policies that are closest to the policy ˆ π t (i.e., the projection of ˆ π t onto the optimal policy set).
Revised: Note that without further specifications, π ∗ t is understood as one of optimal policies that are closest to the policy ˆ π t (i.e., the projection of ˆ π t onto the optimal policy set).

Original: We start with the following elementary result.
Revised: We start with the following elementary result.

Original: Here and throughout the section, we use F t, ˜ d
Revised: Here and throughout the section, we use F t, ˜ d

Original: = { π t,θ : J t,i ( π t,θ ) ≤ ˜ d t,i } to denote the feasible set with upper bounds ˜ d .
Revised: = { π t,θ : J t,i ( π t,θ ) ≤ ˜ d t,i } to denote the feasible set with upper bounds ˜ d .

Original: Note that F t,d is the original feasible set.
Revised: Note that F t,d is the original feasible set.

Original: We also let I
Revised: We also let I

Original: F t, ˜ d ( · ) be the indicator function for the set F t, ˜ d .
Revised: F t, ˜ d ( · ) be the indicator function for the set F t, ˜ d .

Original: Lemma 12.
Revised: Lemma 12.

Original: The function (with variable θ )
Revised: The function (with variable θ )

Original: J t, 0 ( π t,θ )
Revised: J t, 0 ( π t,θ )

Original: + I F t, ˜ d ( π t,θ ) , where ˜ d t is any vector such that
Revised: + I F t, ˜ d ( π t,θ ) , where ˜ d t is any vector such that

Original: F t, ˜ d is non-empty, is definable.
Revised: F t, ˜ d is non-empty, is definable.

Original: Proof.
Revised: Proof.

Original: Since J t,i ( · ) is definable for i = 1 , ..., p , by the rule of composition, which is due to the definable counterpart of the Tarski-Seidenberg theorem, J t,i ( π t,θ ) − d t,i is definable for i = 1 , ..., p .
Revised: Since J t,i ( · ) is definable for i = 1 , ..., p , by the rule of composition, which is due to the definable counterpart of the Tarski-Seidenberg theorem, J t,i ( π t,θ ) − d t,i is definable for i = 1 , ..., p .

Original: Thus, F t, ˜ d is definable on the same o-minimal structure by the definition.
Revised: Thus, F t, ˜ d is definable on the same o-minimal structure by the definition.

Original: Furthermore, I F t, ˜ d ( · ) is definable as the indicator of F t, ˜ d .
Revised: Furthermore, I F t, ˜ d ( · ) is definable as the indicator of F t, ˜ d .

Original: The definability of J t, 0 ( π t,θ ) follows similarly.
Revised: The definability of J t, 0 ( π t,θ ) follows similarly.

Original: Since definability is preserved under addition, the function J t, 0 ( π t,θ )
Revised: Since definability is preserved under addition, the function J t, 0 ( π t,θ )

Original: + I F t, ˜ d ( π t,θ ) is definable.
Revised: + I F t, ˜ d ( π t,θ ) is definable.

Original: For convenience of the reader, we restate the result for non-smooth Kurdyka–Łojasiewicz (KL) inequality from (Bolte et al., 2007, Thm. 14).
Revised: For convenience of the reader, we restate the result for non-smooth Kurdyka–Łojasiewicz (KL) inequality from (Bolte et al., 2007, Thm. 14).

Original: Proposition 1 (Non-smooth Kurdyka–Łojasiewicz inequality) .
Revised: Proposition 1 (Non-smooth Kurdyka–Łojasiewicz inequality) .

Original: Let f be a lower semicontinuous definable function.
Revised: Let f be a lower semicontinuous definable function.

Original: There esists ρ > 0 , a strictly increasing continuous definable function h : [0 , ρ ] → (0 , ∞ ) which is C 1 smooth on (0 , ρ ) , with h (0) = 0 , and a continuous definable function
Revised: There esists ρ > 0 , a strictly increasing continuous definable function h : [0 , ρ ] → (0 , ∞ ) which is C 1 smooth on (0 , ρ ) , with h (0) = 0 , and a continuous definable function

Original: X : R + → (0 , ρ ) such that
Revised: X : R + → (0 , ρ ) such that

Original: [Equation]
Revised: [Equation]

Original: whenever 0 < | f ( x ) | ≤ X ( ∥ x ∥ ) .
Revised: whenever 0 < | f ( x ) | ≤ X ( ∥ x ∥ ) .

Original: Let θ and θ ∗ t denote the parameters of a policy π θ and π ∗ t , respectively.
Revised: Let θ and θ ∗ t denote the parameters of a policy π θ and π ∗ t , respectively.

Original: Directly bounding the distance between θ and θ ∗ t is difficult, because π may be infeasible (this is even true for ˆ π t , since it is only guaranteed to approximately satisfy the constraints), i.e., θ / ∈ F t,d .
Revised: Directly bounding the distance between θ and θ ∗ t is difficult, because π may be infeasible (this is even true for ˆ π t , since it is only guaranteed to approximately satisfy the constraints), i.e., θ / ∈ F t,d .

Original: Thus, the typical approach of following the subgradient flow of J t, 0 ( π θ ) + I F t,d ( π θ ) to reach θ ∗ t is not applicable.
Revised: Thus, the typical approach of following the subgradient flow of J t, 0 ( π θ ) + I F t,d ( π θ ) to reach θ ∗ t is not applicable.

Original: The idea is to enlarge the feasible set F t,d by increasing the violation threshold ˜ d t,i ≥ d t,i + δ , for any δ > 0 , such that with high probability, θ ∈ F t, ˜ d .
Revised: The idea is to enlarge the feasible set F t,d by increasing the violation threshold ˜ d t,i ≥ d t,i + δ , for any δ > 0 , such that with high probability, θ ∈ F t, ˜ d .

Original: Then by following the subgradient flow for J t, 0 ( π θ ) + I ˜ F t ( π θ ) , we can arrive at a critical point ˜ θ ∗ t (corresponding to the policy ˜ π ∗ t ), which is most likely different from θ ∗ t .
Revised: Then by following the subgradient flow for J t, 0 ( π θ ) + I ˜ F t ( π θ ) , we can arrive at a critical point ˜ θ ∗ t (corresponding to the policy ˜ π ∗ t ), which is most likely different from θ ∗ t .

Original: It then remains to bound the distance between θ ∗ t and ˜ θ ∗ t , which is possible due to the preservation of definability through inf projection.
Revised: It then remains to bound the distance between θ ∗ t and ˜ θ ∗ t , which is possible due to the preservation of definability through inf projection.

Original: This is the roadmap we will follow.
Revised: This is the roadmap we will follow.

Original: A graphical illustration of the approach is shown in Fig.
Revised: A graphical illustration of the approach is shown in Fig.

Original: Bounding the term ∥ θ ∗ t − ˜ θ ∗ t ∥ .
Revised: Bounding the term ∥ θ ∗ t − ˜ θ ∗ t ∥ .

Original: In this part, we will bound the term ∥ θ ∗ t − ˜ θ ∗ t ∥ , which will be used to bound ∥ π ∗ t − ˜ π ∗ t ∥ .
Revised: In this part, we will bound the term ∥ θ ∗ t − ˜ θ ∗ t ∥ , which will be used to bound ∥ π ∗ t − ˜ π ∗ t ∥ .

Original: Firstly, we will prove that the parameter θ t , which represents the solution map of an optimization with definable objective and constraints, is definable.
Revised: Firstly, we will prove that the parameter θ t , which represents the solution map of an optimization with definable objective and constraints, is definable.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let θ t ( d ) ∈ arg min { J t, 0 ( π t,θ ) , s.t.
Revised: Let θ t ( d ) ∈ arg min { J t, 0 ( π t,θ ) , s.t.

Original: J t,i ( π t,θ )
Revised: J t,i ( π t,θ )

Original: ≤ d t,i , ∀ i = 1 , ..., p } be the solution map of the constraint parameters d .
Revised: ≤ d t,i , ∀ i = 1 , ..., p } be the solution map of the constraint parameters d .

Original: Then, the function θ t ( d ) is continuous and definable.
Revised: Then, the function θ t ( d ) is continuous and definable.

Original: Furthermore, there exists a finite partition of the space such that the restriction of θ t ( d ) to each partition is
Revised: Furthermore, there exists a finite partition of the space such that the restriction of θ t ( d ) to each partition is

Original: C p smooth.
Revised: C p smooth.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: ⇡ t
Revised: ⇡ t

Original: [Figure]
Revised: [Figure]

Original: latexit> ˜ ⇡
Revised: latexit> ˜ ⇡

Original: ⇤ t
Revised: ⇤ t

Original: Note that for better visualization, we vertically separate the sets F t,d and F t, ˜ d , which also aims to indicate that in general the optimal solution ˜ π ∗ t has a higher objective than π ∗ t due to the relaxed constraints.
Revised: Note that for better visualization, we vertically separate the sets F t,d and F t, ˜ d , which also aims to indicate that in general the optimal solution ˜ π ∗ t has a higher objective than π ∗ t due to the relaxed constraints.

Original: Proof.
Revised: Proof.

Original: First, it can be seen that the solution map θ t ( d ) ∈ arg min J t, 0 ( π t,θ )
Revised: First, it can be seen that the solution map θ t ( d ) ∈ arg min J t, 0 ( π t,θ )

Original: + I F t,d ( π t,θ ) .
Revised: + I F t,d ( π t,θ ) .

Original: Let ϕ t ( d ) =
Revised: Let ϕ t ( d ) =

Original: min J t, 0 ( π t,θ )
Revised: min J t, 0 ( π t,θ )

Original: + I F t,d ( π t,θ ) be the optimal value function.
Revised: + I F t,d ( π t,θ ) be the optimal value function.

Original: Since J t, 0 ( π t,θ ) + I F t,d ( π t,θ ) is definable by Proposition 12, and definability is preserved under inf projection, ϕ t ( d ) is definable.
Revised: Since J t, 0 ( π t,θ ) + I F t,d ( π t,θ ) is definable by Proposition 12, and definability is preserved under inf projection, ϕ t ( d ) is definable.

Original: Since θ t ( d ) = { θ : J t, 0 ( π t,θ )
Revised: Since θ t ( d ) = { θ : J t, 0 ( π t,θ )

Original: + I F t,d ( π t,θ ) =
Revised: + I F t,d ( π t,θ ) =

Original: ϕ t ( d ) } , by the Tarski-Seidenberg Theorem, θ t ( d ) is definable.
Revised: ϕ t ( d ) } , by the Tarski-Seidenberg Theorem, θ t ( d ) is definable.

Original: The continuity property follows directly from Berge’s Maximum theorem.
Revised: The continuity property follows directly from Berge’s Maximum theorem.

Original: Following the discussion of Whitney stratifications in Bolte et al. (2007), since the graph of a definable function is Whitney stratifiable, we can construct a partition by projecting the stratification into the function domain, which will be a Whitney C p -stratification by the constant rank theorem.
Revised: Following the discussion of Whitney stratifications in (Bolte et al., 2007), since the graph of a definable function is Whitney stratifiable, we can construct a partition by projecting the stratification into the function domain, which will be a Whitney C p -stratification by the constant rank theorem.

Original: Furthermore, the restriction of the definable function to each stratum is C p -smooth.
Revised: Furthermore, the restriction of the definable function to each stratum is C p -smooth.

Original: Alternatively, we can directly use the fact that for any definable function, there exists a C p -decomposition which has a finite number of cells, and the restriction to each cell is C p -smooth Van den Dries & Miller (1996).
Revised: Alternatively, we can directly use the fact that for any definable function, there exists a C p -decomposition which has a finite number of cells, and the restriction to each cell is C p -smooth (Van den Dries & Miller, 1996).

Original: This completes the proof.
Revised: This completes the proof.

Original: Now that we have proved that the function θ t ( d ) is definable, we can obtain the bound ∥ θ ∗ t − ˜ θ ∗ t ∥ .
Revised: Now that we have proved that the function θ t ( d ) is definable, we can obtain the bound ∥ θ ∗ t − ˜ θ ∗ t ∥ .

Original: Intuitively, our proof exploits the fact that continuous and definable functions exhibit controlled behaviors along any path, even if it crosses over finite number of Whitney strata.
Revised: Intuitively, our proof exploits the fact that continuous and definable functions exhibit controlled behaviors along any path, even if it crosses over finite number of Whitney strata.

Original: Lemma 13.
Revised: Lemma 13.

Original: For any ˜ d such that F t, ˜ d is non-empty, the following holds:
Revised: For any ˜ d such that F t, ˜ d is non-empty, the following holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since every smooth function over a bounded set is Lipschitz, let us denote L d as the maximum of the Lipschitz constants for all the cells of the Whitney stratification of θ t ( d ) .
Revised: Since every smooth function over a bounded set is Lipschitz, let us denote L d as the maximum of the Lipschitz constants for all the cells of the Whitney stratification of θ t ( d ) .

Original: Let d ( λ ) =
Revised: Let d ( λ ) =

Original: λd +(1 − λ ) ˜ d , where 0 ≤ λ ≤ 1 , be the curve that connects between d and ˜ d .
Revised: λd +(1 − λ ) ˜ d , where 0 ≤ λ ≤ 1 , be the curve that connects between d and ˜ d .

Original: Also, let 0 = λ 1 ≤ ... ≤ λ n =be the partition such that θ t ( d ( λ )) belongs to one cell for all
Revised: Also, let 0 = λ 1 ≤ ... ≤ λ n =be the partition such that θ t ( d ( λ )) belongs to one cell for all

Original: λ
Revised: λ

Original: i < λ <
Revised: i < λ <

Original: λ i +1 for i = 1 , ..., n − 1 .
Revised: λ i +1 for i = 1 , ..., n − 1 .

Original: We know that n < ∞ since d ( θ t ) is Whitney stratifiable.
Revised: We know that n < ∞ since d ( θ t ) is Whitney stratifiable.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to triangle inequality, the second inequality is due to Lipschitz continuity, the third inequality is due to the definition of d ( λ ) , the first equality is due to the nondecreasing sequence of λ i .
Revised: where the first inequality is due to triangle inequality, the second inequality is due to Lipschitz continuity, the third inequality is due to the definition of d ( λ ) , the first equality is due to the nondecreasing sequence of λ i .

Original: Bounding the term ∥ ˜
Revised: Bounding the term ∥ ˜

Original: θ ∗ t − ˆ θ t ∥ .
Revised: θ ∗ t − ˆ θ t ∥ .

Original: Recall that ˆ θ t is the parameter for ˆ π t (output of within-task CRPO), and ˜ θ ∗ t is the parameter for an optimal solution with an enlarged feasible set F t, ˜ d .
Revised: Recall that ˆ θ t is the parameter for ˆ π t (output of within-task CRPO), and ˜ θ ∗ t is the parameter for an optimal solution with an enlarged feasible set F t, ˜ d .

Original: In this subsection, we will obtain the upper bound for the term ∥ ˜
Revised: In this subsection, we will obtain the upper bound for the term ∥ ˜

Original: θ ∗ t − ˆ θ t ∥ .
Revised: θ ∗ t − ˆ θ t ∥ .

Original: Let f ( θ, ˜ d ) =
Revised: Let f ( θ, ˜ d ) =

Original: J t, 0 ( π θ )+ I F t, ˜ d
Revised: J t, 0 ( π θ )+ I F t, ˜ d

Original: ( π θ ) , and choose ˜ d i = O
Revised: ( π θ ) , and choose ˜ d i = O

Original: (1 / √ M ) for i
Revised: (1 / √ M ) for i

Original: = 1 , ..., p , which coincides with the upper bound on constraint violation for within-task CRPO such that ˆ θ t ∈ F t, ˜ d with high probability.
Revised: = 1 , ..., p , which coincides with the upper bound on constraint violation for within-task CRPO such that ˆ θ t ∈ F t, ˜ d with high probability.

Original: In the next result, we will condition on this high probability event.
Revised: In the next result, we will condition on this high probability event.

Original: Lemma 14.
Revised: Lemma 14.

Original: With the choice of ˜ d
Revised: With the choice of ˜ d

Original: = d + δ , where δ =
Revised: = d + δ , where δ =

Original: O (1 / √ M ) coincides with the upper bound on constraint violation for within-task CRPO such that ˆ θ t ∈ F t, ˜ d , the following holds:
Revised: O (1 / √ M ) coincides with the upper bound on constraint violation for within-task CRPO such that ˆ θ t ∈ F t, ˜ d , the following holds:

Original: [Equation]
Revised: [Equation]

Original: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in
Revised: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in

Original: Lemma 1.
Revised: Lemma 1.

Original: Proof.
Revised: Proof.

Original: Without loss of generality, consider f ( θ )
Revised: Without loss of generality, consider f ( θ )

Original: =
Revised: =

Original: J t, 0 ( π θ ) + I F t, ˜ d
Revised: J t, 0 ( π θ ) + I F t, ˜ d

Original: ( π θ ) + c , where c := − inf J t, 0 ( π θ ) + I ˜ F t ( π θ ) , so that the minimal value of f is translated to 0 .
Revised: ( π θ ) + c , where c := − inf J t, 0 ( π θ ) + I ˜ F t ( π θ ) , so that the minimal value of f is translated to 0 .

Original: For simplicity, also assume that f (ˆ θ t ) ≤ X ( ∥ ˆ θ t ∥ ) .
Revised: For simplicity, also assume that f (ˆ θ t ) ≤ X ( ∥ ˆ θ t ∥ ) .

Original: Note that this assumption can be relaxed by using the concept of “curves of maximal slope” at the cost of slightly more complicated analysis and bounds Ioffe (2009).
Revised: Note that this assumption can be relaxed by using the concept of “curves of maximal slope” at the cost of slightly more complicated analysis and bounds (Ioffe, 2009).

Original: Now, consider a subgradient flow ˙ θ ( τ ) ∈ − ∂f
Revised: Now, consider a subgradient flow ˙ θ ( τ ) ∈ − ∂f

Original: ( θ ( τ ))
Revised: ( θ ( τ ))

Original: (see Definition 5), initialized at θ (0)
Revised: (see Definition 5), initialized at θ (0)

Original: =
Revised: =

Original: ˆ θ t then, for any 0 ≤ s ′ <
Revised: ˆ θ t then, for any 0 ≤ s ′ <

Original: s , we have that
Revised: s , we have that

Original: [Equation]
Revised: [Equation]

Original: where the second equality is due to the property of the subgradient flow (see Sec.
Revised: where the second equality is due to the property of the subgradient flow (see Sec.

Original: F.2), the first inequality is due to ∥ ∂ 0 ( hf ) (cid:0) θ ( τ ) (cid:1) ∥ ≥ 1 from Proposition 1, and the second inequality is due to the triangle inequality.
Revised: F.2), the first inequality is due to ∥ ∂ 0 ( hf ) (cid:0) θ ( τ ) (cid:1) ∥ ≥ 1 from Proposition 1, and the second inequality is due to the triangle inequality.

Original: Thus, by taking s ′ = 0 and s → ∞ , we have shown that
Revised: Thus, by taking s ′ = 0 and s → ∞ , we have shown that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to the optimality of ˜ π ∗ t , and ( i ) follows since both ˆ π t and ˜ π ∗ t are feasible for F t, ˜ d .
Revised: where the first inequality is due to the optimality of ˜ π ∗ t , and ( i ) follows since both ˆ π t and ˜ π ∗ t are feasible for F t, ˜ d .

Original: Note that the suboptimality bound can be split as
Revised: Note that the suboptimality bound can be split as

Original: [Equation]
Revised: [Equation]

Original: By CRPO, we can bound the value difference J t, 0 (ˆ π t ) − J t, 0 ( π ∗ t ) ≤ O (1 / √ M ) .
Revised: By CRPO, we can bound the value difference J t, 0 (ˆ π t ) − J t, 0 ( π ∗ t ) ≤ O (1 / √ M ) .

Original: Moreover, Since the value function is Lipschitz (Xu et al., 2020, Lemma 4), the value difference J t, 0 ( π ∗ t )
Revised: Moreover, Since the value function is Lipschitz (Xu et al., 2020, Lemma 4), the value difference J t, 0 ( π ∗ t )

Original: [Equation]
Revised: [Equation]

Original: can be bounded by the distance ∥ θ ∗ t − ˜ θ ∗ t ∥ , which is bounded again by O (1 / √ M ) according to Lemma 13.
Revised: can be bounded by the distance ∥ θ ∗ t − ˜ θ ∗ t ∥ , which is bounded again by O (1 / √ M ) according to Lemma 13.

Original: Hence, recognizing that h is strictly increasing, we have proved the claim.
Revised: Hence, recognizing that h is strictly increasing, we have proved the claim.

Original: Bounding the term ∥ θ ∗ t − ˆ θ t ∥ .
Revised: Bounding the term ∥ θ ∗ t − ˆ θ t ∥ .

Original: Finally, we are able to bound the term of our original interests.
Revised: Finally, we are able to bound the term of our original interests.

Original: Lemma 15.
Revised: Lemma 15.

Original: Under assumption 2, the following holds:
Revised: Under assumption 2, the following holds:

Original: [Equation]
Revised: [Equation]

Original: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in
Revised: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in

Original: Lemma 1.
Revised: Lemma 1.

Original: Proof.
Revised: Proof.

Original: The claim follows directly from Lemmas 13 and 14 and the triangle inequality.
Revised: The claim follows directly from Lemmas 13 and 14 and the triangle inequality.

Original: Remark 4.
Revised: Remark 4.

Original: Note that our strategy to bound ∥ θ ∗ t − ˆ θ t ∥ is algorithmic-agnostic as it only relies on the optimization landscape.
Revised: Note that our strategy to bound ∥ θ ∗ t − ˆ θ t ∥ is algorithmic-agnostic as it only relies on the optimization landscape.

Original: The only place we rely on the algorithm is to bound the suboptimality gap, which is then converted to a bound on ∥ ˜ θ ∗ t − ˆ θ t ∥ in Lemma 14.
Revised: The only place we rely on the algorithm is to bound the suboptimality gap, which is then converted to a bound on ∥ ˜ θ ∗ t − ˆ θ t ∥ in Lemma 14.

Original: Also, the enlargement of feasible set should be viewed as a proof technique and has no implications to the algorithm design.
Revised: Also, the enlargement of feasible set should be viewed as a proof technique and has no implications to the algorithm design.

Original: Indeed, the motivation for the enlargement is to properly design a subgradient flow system.
Revised: Indeed, the motivation for the enlargement is to properly design a subgradient flow system.

Original: Thus, the result of Lemma 15 is not conditioned on how the enlargement is performed.
Revised: Thus, the result of Lemma 15 is not conditioned on how the enlargement is performed.

Original: Also, note that definability is used differently in Lemmas 13 and 14.
Revised: Also, note that definability is used differently in Lemmas 13 and 14.

Original: In the former case, we exploit the Whitney stratification property to provide an upper bound, while in the latter case, we exploit the KL property to obtain a lower bound, hence they serve different purposes.
Revised: In the former case, we exploit the Whitney stratification property to provide an upper bound, while in the latter case, we exploit the KL property to obtain a lower bound, hence they serve different purposes.

Original: F.
Revised: F.

Original: [Equation]
Revised: [Equation]

Original: The result from Lemma 15 can be used directly to provide bounds for (A) and (C) .
Revised: The result from Lemma 15 can be used directly to provide bounds for (A) and (C) .

Original: We start with the term (A) .
Revised: We start with the term (A) .

Original: Lemma 16.
Revised: Lemma 16.

Original: The following bound holds:
Revised: The following bound holds:

Original: [Equation]
Revised: [Equation]

Original: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in
Revised: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in

Original: Lemma 1.
Revised: Lemma 1.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where the first equality is by definition, the first inequality is due to Assumption 1, and the second inequality is due to (Xu et al., 2020, Lem. 3), which also specifies the constant C ν , and (Levin &
Revised: where the first equality is by definition, the first inequality is due to Assumption 1, and the second inequality is due to (Xu et al., 2020, Lem. 3), which also specifies the constant C ν , and (Levin &

Original: Peres, 2017, Prop. 4.2).
Revised: Peres, 2017, Prop. 4.2).

Original: The result then follows by recalling the result from Lemma 15.
Revised: The result then follows by recalling the result from Lemma 15.

Original: F. B OUNDING TERM
Revised: F. B OUNDING TERM

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can prove the upper bound for the error term ( C ) .
Revised: Similarly, we can prove the upper bound for the error term ( C ) .

Original: Lemma 17.
Revised: Lemma 17.

Original: The following bound holds:
Revised: The following bound holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to the non-negativity of ˆ ν t ( s ) and triangle inequality, the second inequality is due to Assumption 1, the third inequality holds for any constant c ′ and is due to (Mei et al., 2020, Lem. 24), and the last inequality is due to (cid:80) s ∈S ˆ ν t ( s )
Revised: where the first inequality is due to the non-negativity of ˆ ν t ( s ) and triangle inequality, the second inequality is due to Assumption 1, the third inequality holds for any constant c ′ and is due to (Mei et al., 2020, Lem. 24), and the last inequality is due to (cid:80) s ∈S ˆ ν t ( s )

Original: = 1 and by choosing c ′ = 0 .
Revised: = 1 and by choosing c ′ = 0 .

Original: The result then follows by recalling the result from Lemma 15.
Revised: The result then follows by recalling the result from Lemma 15.

Original: F
Revised: F

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now, we will upper bound the error term ( B ) .
Revised: Now, we will upper bound the error term ( B ) .

Original: The proof follows DualDICE Nachum et al.
Revised: The proof follows DualDICE (Nachum et al., 2019).

Original: We introduce the following notations.
Revised: We introduce the following notations.

Original: Let ˆ E d D t denote an average of empirical samples where { s i , a i , r i , s ′ i } Ni =1 ∼ d D t , and ρ t be the initial state distribution for the CMDP task t .
Revised: Let ˆ E d D t denote an average of empirical samples where { s i , a i , r i , s ′ i } Ni =1 ∼ d D t , and ρ t be the initial state distribution for the CMDP task t .

Original: The number of data points N =
Revised: The number of data points N =

Original: O ( M 1+1 /σ ) , where σ is any positive number σ ∈ (0 , 1) .
Revised: O ( M 1+1 /σ ) , where σ is any positive number σ ∈ (0 , 1) .

Original: Note that the additional factor of O ( M 1 /σ ) results from the critic evaluation per policy update (see (Xu et al., 2021, Thm. 1)).
Revised: Note that the additional factor of O ( M 1 /σ ) results from the critic evaluation per policy update (see (Xu et al., 2021, Thm. 1)).

Original: We will roughly bound N =
Revised: We will roughly bound N =

Original: O ( M 2 ) in the following to simplify the presentation.
Revised: O ( M 2 ) in the following to simplify the presentation.

Original: The stationary distribution correction factor is denoted as w ˆ π t / D t ( s, a ) =
Revised: The stationary distribution correction factor is denoted as w ˆ π t / D t ( s, a ) =

Original: ˜
Revised: ˜

Original: ν t ( s,a ) d D t ( s,a ) .
Revised: ν t ( s,a ) d D t ( s,a ) .

Original: We make the following regularity assumption on the distribution d D t with respect to the target policy ˆ π t (Nachum et al., 2019, Asm. 1).
Revised: We make the following regularity assumption on the distribution d D t with respect to the target policy ˆ π t (Nachum et al., 2019, Asm. 1).

Original: Assumption 3 (Reference distribution property) .
Revised: Assumption 3 (Reference distribution property) .

Original: For any ( s, a ) , ˜ ν t ( s, a ) > 0 implies that d D t ( s, a ) > 0 .
Revised: For any ( s, a ) , ˜ ν t ( s, a ) > 0 implies that d D t ( s, a ) > 0 .

Original: Furthermore, the correction terms are bounded by some finite constant C ω : [Equation]
Revised: Furthermore, the correction terms are bounded by some finite constant C ω : ∥ ω ˜ ν t / D t ∥ ∞ ≤ C ω .

Original: For convenience, we recapitulate the key points from DualDICE, where we also omit the task dependence t (i.e., we use d D , π , and ρ in lieu of d D t , ˆ π t , and ρ t , respectively).
Revised: For convenience, we recapitulate the key points from DualDICE, where we also omit the task dependence t (i.e., we use d D , π , and ρ in lieu of d D t , ˆ π t , and ρ t , respectively).

Original: The objective function is given by
Revised: The objective function is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The objective in the form prior to introduction of ζ is denoted as J ( z ) :
Revised: The objective in the form prior to introduction of ζ is denoted as J ( z ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let ˆ J ( z, ζ ) denotes the empirical surrogate of J ( z, ζ ) with optimal solution as (ˆ z ∗ , ˆ ζ ∗ ) .
Revised: Let ˆ J ( z, ζ ) denotes the empirical surrogate of J ( z, ζ ) with optimal solution as (ˆ z ∗ , ˆ ζ ∗ ) .

Original: We denote z ∗ =
Revised: We denote z ∗ =

Original: arg min z ∈ J ( z ) and z ∗ =
Revised: arg min z ∈ J ( z ) and z ∗ =

Original: arg min z : S ×
Revised: arg min z : S ×

Original: A → R J ( z ) .
Revised: A → R J ( z ) .

Original: We denote L ( z ) =
Revised: We denote L ( z ) =

Original: max ζ ∈H J ( z, ζ ) and ˆ L ( z ) =
Revised: max ζ ∈H J ( z, ζ ) and ˆ L ( z ) =

Original: max ζ ∈H ˆ J ( z, ζ ) as the primal objectives, and ℓ ( ζ ) =
Revised: max ζ ∈H ˆ J ( z, ζ ) as the primal objectives, and ℓ ( ζ ) =

Original: min z ∈F J ( z, ζ ) , ˆ ℓ ( ζ ) =
Revised: min z ∈F J ( z, ζ ) , ˆ ℓ ( ζ ) =

Original: min z ∈F ˆ J ( z, ζ ) as the dual objectives.
Revised: min z ∈F ˆ J ( z, ζ ) as the dual objectives.

Original: We apply some optimization algorithm OPT for optimizing ˆ J ( z, ζ ) with samples { s i , a i , r i , s ′ i } Ni =1 ∼ , { s i 0 } Ni =1 ∼ β , and target actions a ′ i ∼ ( s ′ i ) , a i 0 ∼ ( s i 0 ) for i = 1 , . . .
Revised: We apply some optimization algorithm OPT for optimizing ˆ J ( z, ζ ) with samples { s i , a i , r i , s ′ i } Ni =1 ∼ , { s i 0 } Ni =1 ∼ β , and target actions a ′ i ∼ ( s ′ i ) , a i 0 ∼ ( s i 0 ) for i = 1 , . . .

Original: , N .
Revised: , N .

Original: The outputs of OPT is denoted by (ˆ z, ˆ ζ ) .
Revised: The outputs of OPT is denoted by (ˆ z, ˆ ζ ) .

Original: We also make the following definitions to
Revised: We also make the following definitions to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We also define
Revised: We also define

Original: [Equation]
Revised: [Equation]

Original: as the optimization error of OPT from DualDICE
Revised: as the optimization error of OPT from DualDICE

Original: [Equation]
Revised: [Equation]

Original: Lemma 18.
Revised: Lemma 18.

Original: By estimating ˆ ν t with DualDICE, the following bound holds:
Revised: By estimating ˆ ν t with DualDICE, the following bound holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We begin with the following decomposition:
Revised: We begin with the following decomposition:

Original: [Equation]
Revised: [Equation]

Original: We will bound each term above separately.
Revised: We will bound each term above separately.

Original: [Equation]
Revised: [Equation]

Original: where the error ϵ opt is induced by the optimization OPT.
Revised: where the error ϵ opt is induced by the optimization OPT.

Original: The error term ϵ 2 can be decomposed as
Revised: The error term ϵ 2 can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: where the equality is due to the result that z ∗ − B ˆ π t z ∗ ( s, a )
Revised: where the equality is due to the result that z ∗ − B ˆ π t z ∗ ( s, a )

Original: = ω
Revised: = ω

Original: ˆ π t / D t ( s, a ) (see (Nachum et al., 2019, Eq. 17)) and ϵ stat is the error due to the finite number error.
Revised: ˆ π t / D t ( s, a ) (see (Nachum et al., 2019, Eq. 17)) and ϵ stat is the error due to the finite number error.

Original: By (Nachum et al., 2019, Lem.
Revised: By (Nachum et al., 2019, Lem.

Original: [Equation]
Revised: [Equation]

Original: (cid:17) with probability at least 1 − δ , where we use the bound on the number of data as O ( M 2 ) .
Revised: (cid:17) with probability at least 1 − δ , where we use the bound on the number of data as O ( M 2 ) .

Original: To bound the second term, use the fact that J ( z ) as defined in 35 is 1 -strongly convex.
Revised: To bound the second term, use the fact that J ( z ) as defined in 35 is 1 -strongly convex.

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from (Nachum et al., 2019, Section D.1)
Revised: where ( i ) follows from (Nachum et al., 2019, Section D.1)

Original: ϵ approx ( F ) is the error due to the approximation with F for z ,
Revised: ϵ approx ( F ) is the error due to the approximation with F for z ,

Original: ϵ approx
Revised: ϵ approx

Original: ( H ) is the error due to the approximation with H for ζ , and ϵ est is the estimation error, and L is the Lipschitz constant for f .
Revised: ( H ) is the error due to the approximation with H for ζ , and ϵ est is the estimation error, and L is the Lipschitz constant for f .

Original: To bound the error between J (ˆ z ∗ ) and J ( z ∗ ) , we use the decomposition suggested in Nachum et al.
Revised: To bound the error between J (ˆ z ∗ ) and J ( z ∗ ) , we use the decomposition suggested in (Nachum et al., 2019):

Original: [Equation]
Revised: [Equation]

Original: where ( i ) ≤ 2 C ω 1 − γ ∥
Revised: where ( i ) ≤ 2 C ω 1 − γ ∥

Original: ζ ∗H − ζ ∗ ∥
Revised: ζ ∗H − ζ ∗ ∥

Original: D t , 1 ≤ 2 C ω 1 − γ ϵ approx ( H ) , ( ii ) =
Revised: D t , 1 ≤ 2 C ω 1 − γ ϵ approx ( H ) , ( ii ) =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: by (Nachum et al., 2019, Lem.
Revised: by (Nachum et al., 2019, Lem.

Original: 6) (by also plugging in N =
Revised: 6) (by also plugging in N =

Original: O ( M 2 ) ), ( iii ) ≤ 0 by definition, and ( iv ) =
Revised: O ( M 2 ) ), ( iii ) ≤ 0 by definition, and ( iv ) =

Original: O ( ϵ approx ( F )) .
Revised: O ( ϵ approx ( F )) .

Original: Note that we refer the reader to (Nachum et al., 2019, Sec.
Revised: Note that we refer the reader to (Nachum et al., 2019, Sec.

Original: D.1) for the above bounds.
Revised: D.1) for the above bounds.

Original: Therefore, we can bound J (ˆ z ∗ ) − J
Revised: Therefore, we can bound J (ˆ z ∗ ) − J

Original: ( z ∗ ) on the order of
Revised: ( z ∗ ) on the order of

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: Combining the above relations, while noting that 1 √ M decreases slower than 1 M in terms of M and is thus kept as the upper bound, we have shown the result.
Revised: Combining the above relations, while noting that 1 √ M decreases slower than 1 M in terms of M and is thus kept as the upper bound, we have shown the result.

Original: F. P UTTING IT TOGETHER : BOUNDING THE KL DIVERGENCE ESTIMATION ERROR
Revised: F. P UTTING IT TOGETHER : BOUNDING THE KL DIVERGENCE ESTIMATION ERROR

Original: Theorem F.1 (KL divergence estimation error bound) .
Revised: Theorem F.1 (KL divergence estimation error bound) .

Original: The following bound holds:
Revised: The following bound holds:

Original: [Equation]
Revised: [Equation]

Original: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in
Revised: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in

Original: Lemma 1, ϵ approx ( F , H ) is defined in 38, and ϵ opt is defined in 39.
Revised: Lemma 1, ϵ approx ( F , H ) is defined in 38, and ϵ opt is defined in 39.

Original: Proof.
Revised: Proof.

Original: The result follows by combining the upper bounds for the error terms ( A ) , ( B ) and ( C ) , as specified by Lemmas 16, 18, and 17.
Revised: The result follows by combining the upper bounds for the error terms ( A ) , ( B ) and ( C ) , as specified by Lemmas 16, 18, and 17.

Original: We also apply the elementary inequality √ a + b + c ≤ √ a + √ b + √ c to further simplify the bound.
Revised: We also apply the elementary inequality √ a + b + c ≤ √ a + √ b + √ c to further simplify the bound.

Original: Remark 5.
Revised: Remark 5.

Original: The bound above depends on the number of iterations M per task on different ways.
Revised: The bound above depends on the number of iterations M per task on different ways.

Original: By increasing M , we can expect to reduce the suboptimality gap, which can help reduce the distance between ˆ π t to the optimal set of policies.
Revised: By increasing M , we can expect to reduce the suboptimality gap, which can help reduce the distance between ˆ π t to the optimal set of policies.

Original: Also, increasing M results in a larger dataset used to estimate its stationary distribution offline by DualDICE, which reduces the estimation error.
Revised: Also, increasing M results in a larger dataset used to estimate its stationary distribution offline by DualDICE, which reduces the estimation error.

Original: The bound indicates that the only terms that do not vanish as we increase the number of iterations per task are those due to the inherent optimization error
Revised: The bound indicates that the only terms that do not vanish as we increase the number of iterations per task are those due to the inherent optimization error

Original: ϵ opt and function approximation error ϵ approx
Revised: ϵ opt and function approximation error ϵ approx

Original: ( F , H ) .
Revised: ( F , H ) .

Original: In the case those terms are negligible (which are possible in view of recent breakthrough in overparametrized deep learning Zhang et al.

Original: ; Neyshabur et al.

Original: ; Li & Liang (2018); Zou et al. Arora et al. (2019), see also Fan et al. (2021) for a survey), then the KL divergence estimation can be driven to arbitrary accuracy.
Revised: In the case those terms are negligible (which are possible in view of recent breakthrough in overparametrized deep learning (Zhang et al., 2021; Neyshabur et al., 2019; Li & Liang, 2018; Zou et al., 2018; Arora et al., 2019), see also (Fan et al., 2021) for a survey), then the KL divergence estimation can be driven to arbitrary accuracy.

Original: G P ROOFS FOR THE SECTION 3. G. TAOG AND TACV BOUNDS FOR CRPO WITH ADAPTIVE LEARNING RATES
Revised: G P ROOFS FOR THE SECTION 3. G. TAOG AND TACV BOUNDS FOR CRPO WITH ADAPTIVE LEARNING RATES

Original: This section presents the task-averaged regret upper bounds for the CRPO when the adaptive learning rates α t are used for each task, and the Q-estimation error is accounted for.
Revised: This section presents the task-averaged regret upper bounds for the CRPO when the adaptive learning rates α t are used for each task, and the Q-estimation error is accounted for.

Original: We also recall that d t,i is the constraint upper bound for i
Revised: We also recall that d t,i is the constraint upper bound for i

Original: = 1 , ..., p and η t is the tolerance for constraint violation (i.e., increasing the upper bound to d t,i + η t ).
Revised: = 1 , ..., p and η t is the tolerance for constraint violation (i.e., increasing the upper bound to d t,i + η t ).

Original: For a single run of CRPO in task t , we denote N t, 0 as the set of time steps the reward is maximized and N t,i as the set of time steps constraint i is minimized.
Revised: For a single run of CRPO in task t , we denote N t, 0 as the set of time steps the reward is maximized and N t,i as the set of time steps constraint i is minimized.

Original: The Q-function in the CRPO algorithm is learned through TD learning with the total number of iterations denoted by K in .
Revised: The Q-function in the CRPO algorithm is learned through TD learning with the total number of iterations denoted by K in .

Original: The Q-function of objective i for policy π t,m at time step m is denoted by
Revised: The Q-function of objective i for policy π t,m at time step m is denoted by

Original: Q it,m , and the estimated Q-function is denoted by ¯ Q it,m .
Revised: Q it,m , and the estimated Q-function is denoted by ¯ Q it,m .

Original: The maximum value for both rewards and constraints is assumed to be c max .
Revised: The maximum value for both rewards and constraints is assumed to be c max .

Original: With all notations for CRPO in place, we present the following result (Xu et al., 2021)
Revised: With all notations for CRPO in place, we present the following result (Xu et al., 2021)

Original: Lemma 19.
Revised: Lemma 19.

Original: For the CRPO algorithm in the tabular settings with learning rates α t , the following bound holds:
Revised: For the CRPO algorithm in the tabular settings with learning rates α t , the following bound holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: If m ∈ N t, 0 , by (Xu et al., 2021, Lemma 7), we have that:
Revised: If m ∈ N t, 0 , by (Xu et al., 2021, Lemma 7), we have that:

Original: [Equation]
Revised: [Equation]

Original: Similarly, if m ∈ N t,i , we can write
Revised: Similarly, if m ∈ N t,i , we can write

Original: [Equation]
Revised: [Equation]

Original: Taking the summation of 42 and 43 from m = 0 to M − 1 , we get α t
Revised: Taking the summation of 42 and 43 from m = 0 to M − 1 , we get α t

Original: [Equation]
Revised: [Equation]

Original: Since J t,i ( π t
Revised: Since J t,i ( π t

Original: [Equation]
Revised: [Equation]

Original: m ) − J t
Revised: m ) − J t

Original: [Equation]
Revised: [Equation]

Original: i ( π ∗
Revised: i ( π ∗

Original: [Equation]
Revised: [Equation]

Original: ≥ η
Revised: ≥ η

Original: [Equation]
Revised: [Equation]

Original: − ∥ Q it
Revised: − ∥ Q it

Original: [Equation]
Revised: [Equation]

Original: m − ¯ Q it,m ∥ (Xu et al., 2021, Eq. 15), by rearranging the terms above we obtain the result.
Revised: m − ¯ Q it,m ∥ (Xu et al., 2021, Eq. 15), by rearranging the terms above we obtain the result.

Original: Next, we study the condition on the maximum constraint violation threshold η t and how it affects
Revised: Next, we study the condition on the maximum constraint violation threshold η t and how it affects

Original: N t, 0 and the upper bounds for TAOG and TACV.
Revised: N t, 0 and the upper bounds for TAOG and TACV.

Original: We make the following assumption to proceed.
Revised: We make the following assumption to proceed.

Original: Assumption 4.
Revised: Assumption 4.

Original: Assume that (cid:80) m ∈N t, 

Revised: Assume that (cid:80) m ∈N t,

Original: J t, 0 ( π ∗ t ) − J t, 0 ( π t,m ) ≥ c J for some c J ∈ ( − 12 α t η t M, 0] .
Revised: J t, 0 ( π ∗ t ) − J t, 0 ( π t,m ) ≥ c J for some c J ∈ ( − 12 α t η t M, 0] .

Original: The assumption above indicates that the policies in N t, 0 do not have rewards higher than the optimal policy by more than 12 α t η t on average.
Revised: The assumption above indicates that the policies in N t, 0 do not have rewards higher than the optimal policy by more than 12 α t η t on average.

Original: Note that it is indeed possible to have rewards higher than the optimal policy if the corresponding policy does not satisfy some safety constraints (i.e., infeasible policy).
Revised: Note that it is indeed possible to have rewards higher than the optimal policy if the corresponding policy does not satisfy some safety constraints (i.e., infeasible policy).

Original: However, it is not a strong assumption since we are comparing with the optimal policy.
Revised: However, it is not a strong assumption since we are comparing with the optimal policy.

Original: The above assumption is not present in Xu et al. (2021), which invalidates one of its derivation steps (in particular, (Xu et al., 2021, Thm. 3)) , and is thus introduced to rectify the proof.
Revised: The above assumption is not present in (Xu et al., 2021), which invalidates one of its derivation steps (in particular, (Xu et al., 2021, Thm. 3)), and is thus introduced to rectify the proof.

Original: Lemma 20.
Revised: Lemma 20.

Original: Suppose that the following condition holds:
Revised: Suppose that the following condition holds:

Original: [Equation]
Revised: [Equation]