Original: Combining the above relations, while noting that 1 √ M decreases slower than 1 M in terms of M and is thus kept as the upper bound, we have shown the result.
Revised: Combining the above relations, while noting that 1 √ M decreases slower than 1 M in terms of M and is thus kept as the upper bound, we have shown the result.

Original: F. P UTTING IT TOGETHER : BOUNDING THE KL DIVERGENCE ESTIMATION ERROR
Revised: F. P UTTING IT TOGETHER : BOUNDING THE KL DIVERGENCE ESTIMATION ERROR

Original: Theorem F.1 (KL divergence estimation error bound) .
Revised: Theorem F.1 (KL divergence estimation error bound) .

Original: The following bound holds:
Revised: The following bound holds:

Original: [Equation]
Revised: [Equation]

Original: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in
Revised: where h is a strictly increasing continuous function with the property that h (0) = 0 as specified in

Original: Lemma 1, ϵ approx ( F , H ) is defined in 38, and ϵ opt is defined in 39.
Revised: Lemma 1, ϵ approx ( F , H ) is defined in 38, and ϵ opt is defined in 39.

Original: Proof.
Revised: Proof.

Original: The result follows by combining the upper bounds for the error terms ( A ) , ( B ) and ( C ) , as specified by Lemmas 16, 18, and 17.
Revised: The result follows by combining the upper bounds for the error terms ( A ) , ( B ) and ( C ) , as specified by Lemmas 16, 18, and 17.

Original: We also apply the elementary inequality √ a + b + c ≤ √ a + √ b + √ c to further simplify the bound.
Revised: We also apply the elementary inequality √ a + b + c ≤ √ a + √ b + √ c to further simplify the bound.

Original: Remark 5.
Revised: Remark 5.

Original: The bound above depends on the number of iterations M per task on different ways.
Revised: The bound above depends on the number of iterations M per task on different ways.

Original: By increasing M , we can expect to reduce the suboptimality gap, which can help reduce the distance between ˆ π t to the optimal set of policies.
Revised: By increasing M , we can expect to reduce the suboptimality gap, which can help reduce the distance between ˆ π t to the optimal set of policies.

Original: Also, increasing M results in a larger dataset used to estimate its stationary distribution offline by DualDICE, which reduces the estimation error.
Revised: Also, increasing M results in a larger dataset used to estimate its stationary distribution offline by DualDICE, which reduces the estimation error.

Original: The bound indicates that the only terms that do not vanish as we increase the number of iterations per task are those due to the inherent optimization error
Revised: The bound indicates that the only terms that do not vanish as we increase the number of iterations per task are those due to the inherent optimization error

Original: ϵ opt and function approximation error ϵ approx
Revised: ϵ opt and function approximation error ϵ approx

Original: ( F , H ) .
Revised: ( F , H ) .

Original: In the case those terms are negligible (which are possible in view of recent breakthrough in overparametrized deep learning (Zhang et al., 2021; Neyshabur et al., 2019; Li & Liang, 2018; Zou et al., 2018; Arora et al., 2019), see also (Fan et al., 2021) for a survey), then the KL divergence estimation can be driven to arbitrary accuracy.
Revised: In the case those terms are negligible (which are possible in view of recent breakthrough in overparametrized deep learning (Zhang et al., 2021; Neyshabur et al., 2019; Li & Liang, 2018; Zou et al., 2018; Arora et al., 2019), see also (Fan et al., 2021) for a survey), then the KL divergence estimation can be driven to arbitrary accuracy.

Original: G P ROOFS FOR THE SECTION 3. G. TAOG AND TACV BOUNDS FOR CRPO WITH ADAPTIVE LEARNING RATES
Revised: G P ROOFS FOR THE SECTION 3. G. TAOG AND TACV BOUNDS FOR CRPO WITH ADAPTIVE LEARNING RATES

Original: This section presents the task-averaged regret upper bounds for the CRPO when the adaptive learning rates α t are used for each task, and the Q-estimation error is accounted for.
Revised: This section presents the task-averaged regret upper bounds for the CRPO when the adaptive learning rates α t are used for each task, and the Q-estimation error is accounted for.

Original: We also recall that d t,i is the constraint upper bound for i
Revised: We also recall that d t,i is the constraint upper bound for i

Original: = 1 , ..., p and η t is the tolerance for constraint violation (i.e., increasing the upper bound to d t,i + η t ).
Revised: = 1 , ..., p and η t is the tolerance for constraint violation (i.e., increasing the upper bound to d t,i + η t ).

Original: For a single run of CRPO in task t , we denote N t, 0 as the set of time steps the reward is maximized and N t,i as the set of time steps constraint i is minimized.
Revised: For a single run of CRPO in task t , we denote N t, 0 as the set of time steps the reward is maximized and N t,i as the set of time steps constraint i is minimized.

Original: The Q-function in the CRPO algorithm is learned through TD learning with the total number of iterations denoted by K in .
Revised: The Q-function in the CRPO algorithm is learned through TD learning with the total number of iterations denoted by K in .

Original: The Q-function of objective i for policy π t,m at time step m is denoted by
Revised: The Q-function of objective i for policy π t,m at time step m is denoted by

Original: Q it,m , and the estimated Q-function is denoted by ¯ Q it,m .
Revised: Q it,m , and the estimated Q-function is denoted by ¯ Q it,m .

Original: The maximum value for both rewards and constraints is assumed to be c max .
Revised: The maximum value for both rewards and constraints is assumed to be c max .

Original: With all notations for CRPO in place, we present the following result (Xu et al., 2021)
Revised: With all notations for CRPO in place, we present the following result (Xu et al., 2021)

Original: Lemma 19.
Revised: Lemma 19.

Original: For the CRPO algorithm in the tabular settings with learning rates α t , the following bound holds:
Revised: For the CRPO algorithm in the tabular settings with learning rates α t , the following bound holds:

Original: [Equation]
Revised: [Equation]

Revised: ∥

Original: Proof.
Revised: Proof.

Original: If m ∈ N t, 0 , by (Xu et al., 2021, Lemma 7), we have that:
Revised: If m ∈ N t, 0 , by (Xu et al., 2021, Lemma 7), we have that:

Original: [Equation]
Revised: [Equation]

Original: Similarly, if m ∈ N t,i , we can write
Revised: Similarly [Equation] if m ∈ N t

Original: [Equation]
Revised: [Equation]

Original: Taking the summation of 42 and 43 from m = 0 to M − 1 , we get α t
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Since J t,i ( π t
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original:  m ) −J t
Revised: Taking the summation of 42 and 43 from m = 0 to M −

Original: [Equation]
Revised: [Equation]

Original: i ( π ∗
Revised: we get α t (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  ≥ η [Equation] − ∥Q it [Equation]
Revised: Since J t,i ( π t,m ) − J t,i ( π ∗ t ) ≥ η t − ∥

Original: m − ¯ Q it,m ∥ (Xu et al., 2021, Eq. 15), by rearranging the terms above we obtain the result.
Revised: Q it,m − ¯ Q i t,m ∥ (Xu et al., 2021, Eq. 15), by rearranging the terms above we obtain the result.

Original: Next, we study the condition on the maximum constraint violation threshold η t and how it affects
Revised: Next, we study the condition on the maximum constraint violation threshold η t and how it affects

Original: N t, 0 and the upper bounds for TAOG and TACV.
Revised: N t, 0 and the upper bounds for TAOG and TACV.

Original: We make the following assumption to proceed.
Revised: We make the following assumption to proceed.

Original: Assumption 4.
Revised: Assumption 4.

Original: Assume that (cid:80) m ∈N t, 

Revised: Assume that (cid:80) m ∈N t,

Original: J t, 0 ( π ∗ t ) − J t, 0 ( π t,m ) ≥ c J for some c J ∈ ( − 12 α t η t M, 0] .
Revised: J t, 0 ( π ∗ t ) − J t, 0 ( π t,m ) ≥ c J for some c J ∈ ( − 1 2 α t η t M, 0] .

Original: The assumption above indicates that the policies in N t, 0 do not have rewards higher than the optimal policy by more than 12 α t η t on average.
Revised: The assumption above indicates that the policies in N t, 0 do not have rewards higher than the optimal policy by more than 12 α t η t on average.

Original: Note that it is indeed possible to have rewards higher than the optimal policy if the corresponding policy does not satisfy some safety constraints (i.e., infeasible policy).
Revised: Note that it is indeed possible to have rewards higher than the optimal policy if the corresponding policy does not satisfy some safety constraints (i.e., infeasible policy).

Original: However, it is not a strong assumption since we are comparing with the optimal policy.
Revised: However, it is not a strong assumption since we are comparing with the optimal policy.

Original: The above assumption is not present in (Xu et al., 2021), which invalidates one of its derivation steps (in particular, (Xu et al., 2021, Thm. 3)), and is thus introduced to rectify the proof.
Revised: The above assumption is not present in (Xu et al., 2021), which invalidates one of its derivation steps (in particular, (Xu et al., 2021, Thm. 3)), and is thus introduced to rectify the proof.

Original: Lemma 20.
Revised: Lemma 20.

Original: Suppose that the following condition holds:
Revised: Suppose that the following condition holds:

Original: [Equation]
Revised: [Equation]

Original: Then, we have that N t, 0 ̸ = ∅ , i.e., ˆ π t is well-defined; also, one the following two statements must hold,
Revised: Then, we have that N t, 0 ̸ = ∅ , i.e., ˆ π t is well-defined; also, one the following two statements must hold,

Original: [Equation]
Revised: [Equation]

Original: Under assumption 4, we also have the following holds:
Revised: Under assumption 4, we also have the following holds:

Original: [Equation]
Revised: [Equation]

Original: for some κ ∈ (0 , 12 ) .
Revised: for some κ ∈ (0 , 12 ) .

Original: Proof.
Revised: Proof.

Original: The proof for N t, 0 ̸ = ∅ follows directly from (Xu et al., 2021, Lem. 9).
Revised: The proof for N t, 0 ̸ = ∅ follows directly from (Xu et al., 2021, Lem. 9).

Original: For the second statement, we consider the case that (cid:80) m ∈N t, 0 (cid:16) J [Equation] π
Revised: For the second statement, we consider the case that (cid:80) m ∈N t, 0 (cid:16) J t,

Original: [Equation]
Revised: [Equation]

Original:  t [Equation] − J t
Revised: ( π ∗ t ) − J t,

Original: [Equation]
Revised: [Equation]

Original: π [Equation] m ) (cid:17) > 0 .
Revised: π t,m ) (cid:17) > 0 .

Original: From Lemma 19, it implies that
Revised: From Lemma 19, it implies that

Original: [Equation]
Revised: [Equation]

Original: Suppose that |N t, 0 | < M/ 2 , then (cid:80) pi =

Revised: Suppose that |N t, 0 | < M/ 2 , then (cid:80) pi =

Original: |N t,i | > M/ 2 , we have that
Revised: |N t,i | > M/ 2 , we have that

Original: [Equation]
Revised: [Equation]

Original: which contradicts 44.
Revised: which contradicts 44.

Original: Hence, we must have |N t, 0 | ≥ M/ 2 .
Revised: Hence, we must have |N t, 0 | ≥ M/ 2 .

Original: Next, we show that |N t, 0 | ≥ (cid:0) 12 − κ (cid:1) M for some κ ∈ (0 , 12 ) .
Revised: Next, we show that |N t, 0 | ≥ (cid:0) 12 − κ (cid:1) M for some κ ∈ (0 , 12 ) .

Original: Under assumption 4 and by Lemma 19, we have that
Revised: Under assumption 4 and by Lemma 19, we have that

Original: [Equation]
Revised: [Equation]

Original: Choose κ := − c J α t η t M .
Revised: Choose κ := − c J α t η t M .

Original: Since − c J < 12 α t η t M by assumption, we have that κ ∈ (0 , 12 ) .
Revised: Since − c J < 12 α t η t M by assumption, we have that κ ∈ (0 , 12 ) .

Original: Consider the case that |N t, 0 | < (cid:0) 12 − κ (cid:1) M , which implies that (cid:80) pi =1 |N t,i | > (cid:0) 12 + κ (cid:1) M .
Revised: Consider the case that |N t, 0 | < (cid:0) 12 − κ (cid:1) M , which implies that (cid:80) pi =1 |N t,i | > (cid:0) 12 + κ (cid:1) M .

Original: This implies that 
Revised: This implies that (cid:0)

Original: [Equation]
Revised: [Equation]

Original: which again contradicts 44.
Revised: which again contradicts 44.

Original: Hence, we must have |N t, 0 | ≥ (cid:0) 1 2 − κ (cid:1) M .
Revised: Hence, we must have |N t, 0 | ≥ (cid:0) 12 − κ (cid:1) M .

Original: Now, we prove the upper bound of suboptimality and constraint violation per task.
Revised: Now, we prove the upper bound of suboptimality and constraint violation per task.

Original: Lemma 21.
Revised: Lemma 21.

Original: Let the violation tolerance be chosen as:
Revised: Let the violation tolerance be chosen as:

Original: [Equation]
Revised: [Equation]

Original: Then, the following holds
Revised: Then, the following holds

Original: [Equation]
Revised: [Equation]

Original: where c t 1 = 2 , c t 2 = 4 c 2 max |S||A| (1 − γ )

Original: [Equation]
Revised: [Equation]

Original: Proof. From Lemma [Equation] we have that
Revised: U

Original: [Equation]
Revised: [Equation]

Original: i [Equation] (cid:88) m [Equation]
Revised: ) where c t 1 = 2 , c t 2 = 4 c 2 max |S||A| (1 − γ ), c t 3 = 3+(1 − γ ) 2 ( − γ ) 2 , c t 4 = 3 c max (1 − γ ) 2 , and c t 5 = 2 √ (1 − γ ) |S||A| 1 − 2 κ .

Original: [Equation]
Revised: Proof.

Original: it
Revised: From Lemma 19, we have that

Original: [Equation]
Revised: [Equation]

Original: If (cid:80) m ∈N [Equation] (cid:16) J t, 0 ( π ∗ t ) − J t, 0 ( π t,m ) (cid:17) ≤ 0 , then J t, 0 ( π ∗ t ) − E
Revised: If (cid:80) m ∈N t,(cid:16) J t, 0 ( π ∗ t ) − J t, 0 ( π t,m ) (cid:17) ≤ 0 , then J t, 0 ( π ∗ t ) − E

Original: [ J t, 0 (ˆ π t )] ≤ 0 .
Revised: [ J t, 0 (ˆ π t )] ≤ 0 .

Original: If (cid:80) m ∈N t, 0 (cid:16) J t, 0 ( π ∗ t ) −
Revised: If (cid:80) m ∈N t, 0 (cid:16) J t, 0 ( π ∗ t ) −

Original: J t, 0 ( π t,m ) (cid:17) > 0 , then, by Lemma 20, we have |N t, 0 | ≥ M/ 2 .
Revised: J t, 0 ( π t,m ) (cid:17) > 0 , then, by Lemma 20, we have |N t, 0 | ≥ M/ 2 .

Original: Hence,
Revised: Hence,

Original: J t, 0 ( π [Equation] [Equation] E [ 
Revised: J t, 0 ( π ∗ t ) − E [ J t, 0 (ˆ π t

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ] =

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the choice of K in
Revised: where the last inequality is due to the choice of K in

Original: = Θ( M 1 /σ log 2 /σ ( |S| 2 |A| 2 M 1+2 /σ /δ )) as specified by (Xu et al., 2021, Lem.
Revised: = Θ( M 1 /σ log 2 /σ ( |S| 2 |A| 2 M 1+2 /σ /δ )) as specified by (Xu et al., 2021, Lem.

Original: 8) for critic evaluations.
Revised: 8) for critic evaluations.

Original: Here, the constants are chosen as c t 1 = 2 ,c
Revised: Here, the constants are chosen as c t 1 = 2 ,

Original: t 2 = 4 c 2 max |S||A| (1 − γ ), c t 3 = 3+(1 − γ ) 2 ( − γ ) 2 , c t 4 = 3 c max (1 − γ ) 2 .
Revised: [Equation]

Original: For constraint violation, consider any i
Revised: For constraint violation, consider any i

Original: = 1 , ..., p , we have
Revised: = 1 , ..., p , we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to triangle inequality, the second inequality is by the design of the
Revised: where the first inequality is due to triangle inequality, the second inequality is by the design of the

Original: CRPO algorithm, and the third inequality is due to Lemma 20, where κ ∈ (0 , 12 ) .
Revised: CRPO algorithm, and the third inequality is due to Lemma 20, where κ ∈ (0 , 12 ) .

Original: By the choice of
Revised: By the choice of

Original: K in , we have that (cid:80) pi =

Revised: K in , we have that (cid:80) pi =

Original: (cid:80) m ∈N [Equation]
Revised: (cid:80) m ∈N t,i ∥ Q

Original: ,i ∥
Revised: i t

Original: [Equation]
Revised: [Equation]

Original: it,π m
Revised: π

Original: [Equation]
Revised: [Equation]

Original: | [Equation] M .
Revised: m ∥ 2 ≤ (cid:112) (1 − γ ) |S||A| M .

Original: Plugging the value of η t yields the desired result.
Revised: Plugging the value of η t yields the desired result.

Original: Finally, we are able to provide the following bounds on TAOG and TACV in the case of adaptive learning rates.
Revised: Finally, we are able to provide the following bounds on TAOG and TACV in the case of adaptive learning rates.

Original: Theorem G.1 (Bounds on TAOG and TACV) .
Revised: Theorem G.1 (Bounds on TAOG and TACV) .

Original: Suppose we run CRPO algorithm for M steps per task t with learning rates α t .
Revised: Suppose we run CRPO algorithm for M steps per task t with learning rates α t .

Original: Then, after T tasks, the TAOG ¯ R 0 is given by
Revised: Then, after T tasks, the TAOG ¯ R 0 is given by

Original: [Equation]
Revised: [Equation]

Original: and the TACV ¯ R i is given by
Revised: and the TACV ¯ R i is given by

Original: [Equation]
Revised: [Equation]

Original: for i = 1 , ..., p , where c t 1 = 2 , c t 2 = 4 c 2 max |S||A| (1 − γ ), c t 3 = 3+(1 − γ 
Revised: for i = 1 , ..., p , where c t 1 = 2 , c t 2 = 4 c 2 max |S||A| (1 − γ ), c t 3 = 3+(1 − γ ) 2 (

Original: [Equation]
Revised: [Equation]

Original: 2 ( − γ ) 2 , c t 4 = 3 c max (1 − γ ) 2 , and c t 5 = 2 √ (1 − γ )
Revised: − γ ) 2 , c t [Equation] 3 c max (1 − γ ) 2 , and c t 5 = 2 √ (1 − γ )

Original: |S||A|− 2 κ .
Revised: |S||A|− 2 κ .

Original: Proof.
Revised: Proof.

Original: The proof follows directly by summing the results 45 and 46 over t = 1 , ..., T .
Revised: The proof follows directly by summing the results 45 and 46 over t = 1 , ..., T .

Original: G. A DAPTING TO THE DYNAMIC REGRET USING ADAPTIVE LEARNING RATES
Revised: G. A DAPTING TO THE DYNAMIC REGRET USING ADAPTIVE LEARNING RATES

Original: We restate the theorem below for convenience.
Revised: We restate the theorem below for convenience.

Original: Theorem G.2.
Revised: Theorem G.2.

Original: Let each within-task CMDP t run M steps of CRPO, initialized by policy π t, 0 :=
Revised: Let each within-task CMDP t run M steps of CRPO, initialized by policy π t, 0 :=

Original: INIT( t ) and learning rates { α t } pi =0 :=
Revised: INIT( t ) and learning rates { α t } pi =0 :=

Original: SIM( t ) .
Revised: SIM( t ) .

Original: Let κ ∗ :=
Revised: Let κ ∗ :=

Original: arg min L ( κ ) , where
Revised: arg min L ( κ ) , where

Original: [Equation]
Revised: [Equation]

Original: and { ψ ∗ t } Tt =1 is any comparator sequence.
Revised: and { ψ ∗ t } Tt =1 is any comparator sequence.

Original: Then, the following bounds on TAOG and TACV hold:
Revised: Then, the following bounds on TAOG and TACV hold:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The idea of the proof is to freeze the learning rates first to obtain a dynamic regret bound based on policy initialization, and then optimize over the learning rates to obtain a tighter characterization.
Revised: The idea of the proof is to freeze the learning rates first to obtain a dynamic regret bound based on policy initialization, and then optimize over the learning rates to obtain a tighter characterization.

Original: Also, since TAOG and TACV only differ by a bias term that does not depend on either the learning rates nor the initial policy, we can treat them indistinguishably.
Revised: Also, since TAOG and TACV only differ by a bias term that does not depend on either the learning rates nor the initial policy, we can treat them indistinguishably.

Original: In particular,
Revised: In particular,

Original: T

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Ψ := { ϕ t } Tt =1 .
Revised: where Ψ := { ϕ t } Tt =1 .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: and define
Revised: and define

Original: [Equation]
Revised: [Equation]

Original: Thus, plugging κ = κ ∗ in 54 results in 50.
Revised: Thus, plugging κ = κ ∗ in 54 results in 50.

Original: Now, we provide the regret upper bound for U simT ( κ ) , when SIM( t ) is OGD over the sequence ˆ f simt ( κ ) .
Revised: Now, we provide the regret upper bound for U simT ( κ ) , when SIM( t ) is OGD over the sequence ˆ f simt ( κ ) .

Original: Corollary 2.
Revised: Corollary 2.

Original: For any fixed comparator α ∗ =
Revised: For any fixed comparator α ∗ =

Original: arg min κ (cid:80) Tt
Revised: arg min κ (cid:80) Tt

Original: =

Revised: =

Original: ˆ f simt , if SIM( t ) is OGD which is run on a sequence of loss functions { ˆ f simt } t ∈ [ T ] with the step-size α ∗
Revised: ˆ f simt , if SIM( t ) is OGD which is run on a sequence of loss functions { ˆ f simt } t ∈ [ T ] with the step-size α ∗

Original: K α √ 2 T , then the following bound holds for static regret:
Revised: K α √ 2 T , then the following bound holds for static regret:

Original: [Equation]
Revised: [Equation]

Original: for any C 1 ∈ { c ∈ [Equation] ∈ Λ } where u is an ϵ -subgradient of f simt ( κ ) with respect to κ , E T := (cid:80) Tt
Revised: for any C 1 ∈ { c ∈ (cid:16) 0 , 2 L α (cid:17) : α ∗ + c ( u − ˆ ∇ t ) ∈ Λ } where u is an ϵ -subgradient of f simt ( κ ) with respect to κ , E T := (cid:80) Tt

Original: =

Revised: =

Original: ϵ t is the cumulative inexactness, ϵ t is the upper bound from Theorem 3.1.
Revised: ϵ t is the cumulative inexactness, ϵ t is the upper bound from Theorem 3.1.

Original: Proof.
Revised: Proof.

Original: The proof follows directly after substituting c = 4 2 C 1 − C
Revised: The proof follows directly after substituting c = 4 2 C 1 − C

Original: L α and other appropriate constants in Theorem E.1.
Revised: L α and other appropriate constants in Theorem E.1.

Original: Next, we provide the proof of Colrollary 1 TAOG and TACV regret bounds when INIT and SIM are either FTL or inexact OGD.
Revised: Next, we provide the proof of Colrollary 1 TAOG and TACV regret bounds when INIT and SIM are either FTL or inexact OGD.

Original: Corollary 3.
Revised: Corollary 3.

Original: For any fixed comparator α ∗ =
Revised: For any fixed comparator α ∗ =

Original: arg min κ(cid:80) Tt
Revised: arg min κ

Original: [-=
 ˆ f simt and c t 3 = 3+(1 − γ ) 2 ( − γ ) 2 .-]
Revised: [-=
 ˆ f simt and c t 3 = 3+(1 − γ ) 2 ( − γ ) 2 .-][Equation]

Original: If INIT( t ) and SIM( t ) are FTL or inexact OGD over the sequences ˆ f initt and ˆ f simt , then, the following bounds on TAOG and TACV hold:
Revised: and SIM( t ) are FTL or inexact OGD over the sequences ˆ f initt and ˆ f simt , then, the following bounds on TAOG and TACV hold:

Original: [Equation]
Revised: [Equation]

Original: for all i =0 , . . .
Revised: for all i =

Original: , p , where { ψ ∗ t } t ∈ [ T ] is any comparator sequence,

Original: c t 3 = 3+(1 − γ ) 2 (1 − γ ) 2 , and c t 2 = 4 c 2 max |S||A| (1 − γ ).

Original: Proof.
Revised: Proof.

Original: Firstly, the following inexact upper bounds hold for U initT and U simT :
Revised: Firstly, the following inexact upper bounds hold for U initT and U simT :

Original: [Equation]
Revised: [Equation]

Original: where the constants in 
Revised: where the constants in the 58 and 59 are from the Lemma 11 and Corollary 2 respectively.

Original: Plugging these in the equation 49, we can obtain the following:
Revised: Plugging these in the equation 49, we can obtain the following:

Original: [Equation]
Revised: [Equation]

Original: where the relation ˆ V 2 ψ = 1 T (cid:80) Tt =1 E ˆ ν t
Revised: where the relation ˆ V 2 ψ = 1 T (cid:80) Tt =1 E ˆ ν t

Original: [ D KL (ˆ π t | ψ ∗ t )] is used to get the second last term T ˆ V 2 ψ κ .
Revised: [ D KL (ˆ π t | ψ ∗ t )] is used to get the second last term T ˆ V 2 ψ κ .

Original: To get κ ∗ such that L ( κ ) is minimized, we proceed using the KKT optimality conditions to get dLdκ as
Revised: To get κ ∗ such that L ( κ ) is minimized [Equation] we proceed using the KKT optimality conditions to get dLdκ as

Original: [Equation]
Revised: [Equation]

Original: Therefore, we obtain κ ∗ =
Revised: Therefore, we obtain κ ∗ =

Original: [Equation]
Revised: (cid:114) U initT ( ψ )+ E T + T ˆ V 2 ψ c t 2 M + c t 4 √ M .

Original: Substituting this value of κ ∗ in G.2, and using the approximation (cid:113) [Equation] M + √ M ≊ 1 M 1 / 4 , we can obtain the final result.
Revised: Substituting this value of κ ∗ in G.2, and using the approximation (cid:113) 1 M + √ M ≊ 1 M 1 / 4 , we can obtain the final result.

Original: H A DDITIONAL EXPERIMENTS AND DETAILS
Revised: H A DDITIONAL EXPERIMENTS AND DETAILS

Original: In this section, we describe our experimental setup and other additional details for the OpenAI gym experiments under constrained settings.
Revised: In this section, we describe our experimental setup and other additional details for the OpenAI gym experiments under constrained settings.

Original: We present the performance of Meta-SRL on the Acrobot and the Frozen lake under high task-similarity conditions.
Revised: We present the performance of Meta-SRL on the Acrobot and the Frozen lake under high task-similarity conditions.

Original: We first present some details on the baselines used to avoid any confusion.
Revised: We first present some details on the baselines used to avoid any confusion.

Original: Details of baselines used:
Revised: Details of baselines used:

Original: Denote the policy initialization for task t as ϕ t .
Revised: Denote the policy initialization for task t as ϕ t .

Original: First baseline is the FAL, which initializes the policy ϕ t +1 for the test task t + 1 as ϕ t +1 =
Revised: First baseline is the FAL, which initializes the policy ϕ t +1 for the test task t + 1 as ϕ t +1 =

Original: 1 t (cid:80) ti =
Revised: 1 t (cid:80) ti =

Original: 1 ϕ i .
Revised: 1 ϕ i .

Original: This is done online as the tasks are encountered sequentially.
Revised: This is done online as the tasks are encountered sequentially.

Original: For the simple averaging baseline, we run CRPO on 10 tasks with random initialization, and evaluate the performance on a test task by initializing with the average of all the suboptimal policies from the batch of suboptimal policies, i.e., ϕ T =
Revised: For the simple averaging baseline, we run CRPO on 10 tasks with random initialization, and evaluate the performance on a test task by initializing with the average of all the suboptimal policies from the batch of suboptimal policies, i.e., ϕ T =

Original: 1 T − 1 (cid:80) T − 1 i =1 ϕ
Revised: 1 T − 1 (cid:80) T − 1 i =1 ϕ

Original: i , where T is the test task.
Revised: i , where T is the test task.

Original: This can be seen as an offline method, where suboptimal policies from T − 1 tasks is stored and averaged to get the initialization for the test task.
Revised: This can be seen as an offline method, where suboptimal policies from T − 1 tasks is stored and averaged to get the initialization for the test task.

Original: The pre-trained baseline uses suboptimal policy from an already trained task to initialize the policy on the test task, i.e., ϕ t
Revised: The pre-trained baseline uses suboptimal policy from an already trained task to initialize the policy on the test task, i.e., ϕ t

Original: = ˆ π t ′ , where ϕ t is the policy initialization for the task t , and ˆ π t ′ is the suboptimal policy returned some pre-trained task t ′ .
Revised: = ˆ π t ′ , where ϕ t is the policy initialization for the task t , and ˆ π t ′ is the suboptimal policy returned some pre-trained task t ′ .

Original: This baseline could be thought of as the
Revised: This baseline could be thought of as the

Original: Strawman initialization strategy, where the policy for the next task is initialized using the suboptimal policy from the previous task, i.e., ϕ t +1 =
Revised: Strawman initialization strategy, where the policy for the next task is initialized using the suboptimal policy from the previous task, i.e., ϕ t +1 =

Original: ˆ π t .
Revised: ˆ π t .

Original: For the Meta-SRL in the discrete state action-space (i.e., Frozen lake), we take the weighted average of the previous suboptimal policies weighted by the stationary distributions induced by each suboptimal policy over all thes states.
Revised: For the Meta-SRL in the discrete state action-space (i.e., Frozen lake), we take the weighted average of the previous suboptimal policies weighted by the stationary distributions induced by each suboptimal policy over all thes states.

Original: We also adapt the learning rates α t for the CRPO algorithm for each task t .
Revised: We also adapt the learning rates α t for the CRPO algorithm for each task t .

Original: The difference between FAL and Meta-SRL is that Meta-SRL weights the suboptimal policies higher in the states which were encountered more frequently.
Revised: The difference between FAL and Meta-SRL is that Meta-SRL weights the suboptimal policies higher in the states which were encountered more frequently.

Original: Random initialization baseline is using the random iniitalization for the within-task algorithm CRPO.
Revised: Random initialization baseline is using the random iniitalization for the within-task algorithm CRPO.

Original: Experimental setup: We run all the algorithms online where tasks are encountered sequentially, and present the results for the test-task after the policy initialization suggested by respective baselines.
Revised: Experimental setup: We run all the algorithms online where tasks are encountered sequentially, and present the results for the test-task after the policy initialization suggested by respective baselines.

Original: We do 10 runs for each baseline on the test task, and present the variance plots.
Revised: We do 10 runs for each baseline on the test task, and present the variance plots.

Original: On the test task, we train for 8 steps on the 
Revised: On the test task, we train for 8 steps on the Frozen lake and for 5 steps on the Acrobot.

Revised: In the Frozen lake, each step corresponds to 5 episodes, and the average rewards/costs are reported for each step in the perofrmance and constraint violation plots.

Original: H. F ROZEN LAKE
Revised: H. F ROZEN LAKE

Original: Frozen lake: For the Frozen lake, we randomly generate T = 10 different orientations as tasks over the probability of a state being frozen or a hole, and evaluate the performance for the scenarios with high task-similarity (low variance for the latent CMDP distribution) or low task-similarity (high variance for the latent CMDP distribution).
Revised: Frozen lake: For the Frozen lake, we randomly generate T = 10 different orientations as tasks over the probability of a state being frozen or a hole, and evaluate the performance for the scenarios with high task-similarity (low variance for the latent CMDP distribution) or low task-similarity (high variance for the latent CMDP distribution).

Original: The agent gets rewarded +2 when it reaches the goal state, and incurs a cost − 1 when it falls into a hole.
Revised: The agent gets rewarded +2 when it reaches the goal state, and incurs a cost − 1 when it falls into a hole.

Original: We choose the constraint threshold d t,i = 0 .
Revised: We choose the constraint threshold d t,i = 0 .

Original: High task-similarity: To generate tasks with high task-similarity, we start with a random frozen lake grid of 4 × 4 , where the probability of each grid being a frozen is 0 .
Revised: High task-similarity: To generate tasks with high task-similarity, we start with a random frozen lake grid of 4 × 4 , where the probability of each grid being a frozen is 0 .

Original: Then, we generate 9 different grids which differ from the first one by only one of the grids.
Revised: Then, we generate 9 different grids which differ from the first one by only one of the grids.

Original: This means the agent always encounters the new grid which is very similar to the previous task.
Revised: This means the agent always encounters the new grid which is very similar to the previous task.

Original: From Figure 4, we can observe that baselines pre-trained and FAL are competitive with the Meta-SRL in terms of reward maximization and almost zero constraint violations.
Revised: From Figure 4, we can observe that baselines pre-trained and FAL are competitive with the Meta-SRL in terms of reward maximization and almost zero constraint violations.

Original: The good performance of the pre-trained baseline can be explained using the fact that the as the test task and the training task are very similar, the policy initialization will be close to the optimal policy of the test task.
Revised: The good performance of the pre-trained baseline can be explained using the fact that the as the test task and the training task are very similar, the policy initialization will be close to the optimal policy of the test task.

Original: H. A CROBOT
Revised: H. A CROBOT

Original: Acrobot: Acrobot is a 2 link robot OpenAI gym environment which has a continuous state space.
Revised: Acrobot: Acrobot is a 2 link robot OpenAI gym environment which has a continuous state space.

Original: The agent is rewarded when it achieves certain height of the end link.
Revised: The agent is rewarded when it achieves certain height of the end link.

Original: Two constraints are introduced for two links, where − 1 cost is incurred if any link swings in the prohibited direction.
Revised: Two constraints are introduced for two links, where − 1 cost is incurred if any link swings in the prohibited direction.

Original: We randomly generate T = 50 different tasks with different mass links and center of gravity.
Revised: We randomly generate T = 50 different tasks with different mass links and center of gravity.

Original: High task-similarity: To generate tasks with high similarity for the acrobot, we considered changing the mass of the links, center of gravity (COG) and constraint threshold for each link.
Revised: High task-similarity: To generate tasks with high similarity for the acrobot, we considered changing the mass of the links, center of gravity (COG) and constraint threshold for each link.

Original: The changes in these quantities were done by adding noise to the default quantities.
Revised: The changes in these quantities were done by adding noise to the default quantities.

Original: We considered a Gaussian noise with a low variance of 0 .
Revised: We considered a Gaussian noise with a low variance of 0 .

Original: 1 to change the tasks only slightly.
Revised: 1 to change the tasks only slightly.

Original: From Figure 5, we can observe that only pre-trained and FAL baselines are competitive with the Meta-SRL in terms of reward maximization and almost zero constraint violations.
Revised: From Figure 5, we can observe that only pre-trained and FAL baselines are competitive with the Meta-SRL in terms of reward maximization and almost zero constraint violations.

Original: Low task-similarity: To generate low similar tasks, we increased the variance of the Gaussian noise to 0 .
Revised: Low task-similarity: To generate low similar tasks, we increased the variance of the Gaussian noise to 0 .

Original: We can observe from Figure 2 that the performance of the baselines was poor for the constraint satisfaction, while Meta-SRL was able to converge quickly for reward maximization and constraint satisfaction.
Revised: We can observe from Figure 2 that the performance of the baselines was poor for the constraint satisfaction, while Meta-SRL was able to converge quickly for reward maximization and constraint satisfaction.

Original: Note that, in real-world settings, tasks are likely to have low-similarity in terms of how close their optimal policies are.
Revised: Note that, in real-world settings, tasks are likely to have low-similarity in terms of how close their optimal policies are.

Original: Good performance of Meta-SRL under these settings highlights its potential to be extended to real-world settings where safety constraints are present.
Revised: Good performance of Meta-SRL under these settings highlights its potential to be extended to real-world settings where safety constraints are present.

Original: H. A DDITIONAL EXPERIMENTS
Revised: H. A DDITIONAL EXPERIMENTS

Original: In this section, we provide additional experiments on HalfCHeetah from Mujoco.
Revised: In this section, we provide additional experiments on HalfCHeetah from Mujoco.

Original: H.3. H ALF CHEETAH
Revised: H.3. H ALF CHEETAH

Original: Half-cheetah: Half-cheetah is a simulation environment for 2-dimensional robot.
Revised: Half-cheetah: Half-cheetah is a simulation environment for 2-dimensional robot.

Original: It consists of 9 links and 8 joints, where the goal for the cheetah is to run at a certain velocity.
Revised: It consists of 9 links and 8 joints, where the goal for the cheetah is to run at a certain velocity.

Original: The original
Revised: The original

Original: HalfCheetah environment does not have any constraints.
Revised: HalfCheetah environment does not have any constraints.

Original: We introduce a constraint that penalizes the deviation of the cheetah’s head from some desired height:
Revised: We introduce a constraint that penalizes the deviation of the cheetah’s head from some desired height:

Original: [Equation]
Revised: [Equation]

Original: where we specify the cumulative absolute difference between the cheetah head height and the desired height to be less than a tolerance ϵ .
Revised: where we specify the cumulative absolute difference between the cheetah head height and the desired height to be less than a tolerance ϵ .

Original: The cheetah is trained on T = 100 tasks for both high and low task-similarity settings.
Revised: The cheetah is trained on T = 100 tasks for both high and low task-similarity settings.

Original: High task-similarity: To generate tasks with high similarity, the goal velocity for each training task is uniformly sampled from a range of [0 . 35 , 0 . 65] m/s .
Revised: High task-similarity: To generate tasks with high similarity, the goal velocity for each training task is uniformly sampled from a range of [0 . 35 , 0 . 65] m/s .

Original: From Figure 6, we can observe that under high task similarity settings, Meta-SRL is able achieve high rewards and also able to keep the constraint violation of the cheetah’s head height below the threshold.
Revised: From Figure 6, we can observe that under high task similarity settings, Meta-SRL is able achieve high rewards and also able to keep the constraint violation of the cheetah’s head height below the threshold.

Original: Under high task-similarity settings, both pre-trained and Meta-SRL perform well, as expected.
Revised: Under high task-similarity settings, both pre-trained and Meta-SRL perform well, as expected.

Original: However, it can be observed that both simple averaging and random initialization perform poorly in this setting.
Revised: However, it can be observed that both simple averaging and random initialization perform poorly in this setting.

Original: Close inspection indicates that there is a high variance among the policy parameters learned from each task, which may result in interference among different tasks in the relatively high dimensional state space.[Figure]
Revised: Close inspection indicates that there is a high variance among the policy parameters learned from each task, which may result in interference among different tasks in the relatively high dimensional state space.

Original: Here, the and random perform the worst on the

Original: Low task-similarity: To generate tasks with low similarity for the half-cheetah, the goal velocity for each training task is uniformly sampled from a range of [0 . 0 , 1 . 0] m/s .
Revised: Low task-similarity: To generate tasks with low similarity for the half-cheetah, the goal velocity for each training task is uniformly sampled from a range of [0 . 0 , 1 . 0] m/s .

Original: Tasks are less similar due to the high variance of goal velocities that the cheetah is trained on.
Revised: Tasks are less similar due to the high variance of goal velocities that the cheetah is trained on.

Original: It can be observed from Figurethat Meta-SRL is able to achieve higher rewards and zero constraint violations quickly compared to other baseline initializations under low task-relatedness settings.
Revised: It can be observed from Figurethat Meta-SRL is able to achieve higher rewards and zero constraint violations quickly compared to other baseline initializations under low task-relatedness settings.

Original: The pre-trained baseline can achieve higher rewards, but similar to other baselines, it cannot achieve constraint satisfaction within 10 steps.
Revised: The pre-trained baseline can achieve higher rewards, but similar to other baselines, it cannot achieve constraint satisfaction within 10 steps.

Original: It can also be observed that both simple averaging and random initialization perform poorly in reward maximization in this setting.
Revised: It can also be observed that both simple averaging and random initialization perform poorly in reward maximization in this setting.

Original: Close inspection indicates that there is a high variance among the policy  parameters learned from each task, which may result in interference among different tasks in the relatively high dimensional state space.
Revised: Close inspection indicates that there is a high variance among the policy [Figure] parameters learned from each task, which may result in interference among different tasks in the relatively high dimensional state space.

Original: [Figure]
Revised: [Figure]

Original: I R ELATION OF M ETA -SRL TO HARDNESS RESULTS IN (K WON ET AL ., 2021)
Revised: I R ELATION OF M ETA -SRL TO HARDNESS RESULTS IN (K WON ET AL ., 2021)

Original: There is a key difference in the problem setting of meta-learning in our study and the latent MDP setting in (Kwon et al., 2021).
Revised: There is a key difference in the problem setting of meta-learning in our study and the latent MDP setting in (Kwon et al., 2021).

Original: The latent MDP setting is more challenging in the sense that there is no clear boundary between tasks.
Revised: The latent MDP setting is more challenging in the sense that there is no clear boundary between tasks.

Original: In the latent MDP setting, each episode may come from an unknown
Revised: In the latent MDP setting, each episode may come from an unknown

Original: MDP drawn from a distribution (as a special case of POMDP); in the meta-learning setting, the agent knows when a new task has arrived and is allowed to interact with the MDP over a set of episodes (the number is linear with respect to M in our paper).
Revised: MDP drawn from a distribution (as a special case of POMDP); in the meta-learning setting, the agent knows when a new task has arrived and is allowed to interact with the MDP over a set of episodes (the number is linear with respect to M in our paper).

Original: Due to the above difference, the worst-case lower bound of requiring an exponential number of episodes to learn an ϵ -optimal policy in (Kwon et al., 2021) does not hold in our case.
Revised: Due to the above difference, the worst-case lower bound of requiring an exponential number of episodes to learn an ϵ -optimal policy in (Kwon et al., 2021) does not hold in our case.

Original: Indeed, if the identity (referred to as “context” in latent MDP) is revealed or can be inferred, (Kwon et al., 2021) is able to achieve a regret that is polynomial in the number of episodes (Thms. 3.3 and 3.4 from (Kwon et al., 2021)).
Revised: Indeed, if the identity (referred to as “context” in latent MDP) is revealed or can be inferred, (Kwon et al., 2021) is able to achieve a regret that is polynomial in the number of episodes (Thms. 3.3 and 3.4 from (Kwon et al., 2021)).

Original: Furthermore, close examination of the bounds provided by (Kwon et al., 2021) also reveals some differences from our result.
Revised: Furthermore, close examination of the bounds provided by (Kwon et al., 2021) also reveals some differences from our result.

Original: In particular, let K be the number of contexts in a latent MDP and N be the total number of episodes ( N is on the order of TM in our case as we encounter T tasks, each with M episodes).
Revised: In particular, let K be the number of contexts in a latent MDP and N be the total number of episodes ( N is on the order of TM in our case as we encounter T tasks, each with M episodes).

Original: Then (Kwon et al., 2021) is able to bound the regret (without dividing by the number of episodes N ) as O ( √ KN ) .
Revised: Then (Kwon et al., 2021) is able to bound the regret (without dividing by the number of episodes N ) as O ( √ KN ) .

Original: To compare their bound with ours, we consider each task in the meta-learning setting as a context, so
Revised: To compare their bound with ours, we consider each task in the meta-learning setting as a context, so

Original: T =
Revised: T =

Original: O ( K ) .
Revised: O ( K ) .

Original: Therefore, their upper bound (after dividing by the number of episodes N = TM ) becomes O (1 / √ M ) , which does not diminish with the number of tasks T .
Revised: Therefore, their upper bound (after dividing by the number of episodes N = TM ) becomes O (1 / √ M ) , which does not diminish with the number of tasks T .

Original: Note that our bound (see the comment after Corollary 1) is O (cid:16)
Revised: Note that our bound (see the comment after Corollary 1) is O (cid:16)

Original: [Table]
Revised: [Table]

Original: (after dividing by the number of episodes N = TM ), where for simplicity we have assumed E T = 0 , i.e., exact access to the loss function.
Revised: [Table] dividing by the number of episodes N = TM ), where for simplicity we have assumed E T = 0 , i.e., exact access to the loss function.

Original: Note that ˆ V ψ is a measure of task-relatedness (a smaller value indicates more relatedness among tasks).
Revised: Note that ˆ V ψ is a measure of task-relatedness (a smaller value indicates more relatedness among tasks).

Original: It can be seen that while we have a worse order dependence on M , our bound scales with task relatedness ˆ V ψ and diminishes with respect to the increasing number of tasks
Revised: It can be seen that while we have a worse order dependence on M , our bound scales with task relatedness ˆ V ψ and diminishes with respect to the increasing number of tasks

Original: T .
Revised: T .

Original: This is expected, as we leverage the relatedness among contexts (in fact, the result “of (Kwon et al., 2021)” would hold when tasks are sufficiently different from each other to infer the contexts with spectral methods).
Revised: This is expected, as we leverage the relatedness among contexts (in fact, the result “of (Kwon et al., 2021)” would hold when tasks are sufficiently different from each other to infer the contexts with spectral methods).

Original: In summary, we refer to (Kwon et al., 2021) as an example that achieving regret diminishing in the number of tasks T is hard, even with the assumption of observing the task identities (contexts).
Revised: In summary, we refer to (Kwon et al., 2021) as an example that achieving regret diminishing in the number of tasks T is hard, even with the assumption of observing the task identities (contexts).

Original: J N OTATIONS AND CONSTANTS
Revised: J B ROADER I MPACT S TATEMENTS

Revised: There is an increasing need to address fairness as a constraint in learning settings.

Revised: Existing works that aim to achieve zero-shot generalization without any task-specific adaptation have limited capability to adapt to shifting environments.

Revised: While online meta-learning is a principled technique to learn good priors over model parameters for fast adaptation in a sequential setting, existing methods often do not address constraints and thus have limited applications in fairness-aware learning.

Revised: The proposed CMDP-within-online framework can potentially be adapted to reinforcement learning tasks with fairness constraints in a nonstationary environment .

Revised: In practice, this can provide a strategy that learns priors over policy parameters not only to master the current fairness-aware task but also to become proficient with quick adaptation at learning newly arrived tasks.

Revised: Our theoretical analysis can be leveraged to provide sublinear bound on the “task-averaged fairness violation” regret.

Revised: Similar ideas have been explored by (Zhao et al., 2021) in the supervised learning setting, while we are not aware of any work on the reinforcement learning counterpart.

Revised: Thus, it can be an extension for future work to explore the extent to which our method can address this important problem.

Revised: Nevertheless, fairness constraints present a unique challenge for meta-safe RL settings, as fairness constraints should rarely be violated in a real-world setting due to the implicated discrimination or bias.

Revised: Additional efforts, such as incorporating pessimism Bai et al.

Revised: or developing offline methods, may be entailed to reduce fairness violations during initial deployment.

Revised: K N OTATIONS AND CONSTANTS

Original: [Table] [Table]
Revised: [Table] [Table]