Original: B OOSTED C URRICULUM R EINFORCEMENT L EARNING
Revised: B OOSTED C URRICULUM R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difﬁculty.
Revised: Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difﬁculty.

Original: However, ﬁnding an exact way of reusing action-values in this setting is still a poorly understood problem.
Revised: However, ﬁnding an exact way of reusing action-values in this setting is still a poorly understood problem.

Original: In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task.
Revised: In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task.

Original: This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the beneﬁt of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented.
Revised: This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the beneﬁt of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented.

Original: This procedure allows reusing previous action-values while promoting expressiveness of the action-value function.
Revised: This procedure allows reusing previous action-values while promoting expressiveness of the action-value function.

Original: We theoretically study BCRL as an approximate value iteration algorithm, proving its advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function.
Revised: We theoretically study BCRL as an approximate value iteration algorithm, proving its advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function.

Original: Finally, we provide detailed empirical evidence of the beneﬁts of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.
Revised: Finally, we provide detailed empirical evidence of the beneﬁts of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The combination of reinforcement learning (RL) (Sutton & Barto, 2018) algorithms with powerful function approximators, i.e., deep RL (Franc¸ois-Lavet et al., 2018; Mnih et al., 2015), is a major breakthrough towards solving complex decision making and control problems that were impractical for previous shallow RL methods.
Revised: The combination of reinforcement learning (RL) (Sutton & Barto, 2018) algorithms with powerful function approximators, i.e., deep RL (Franc¸ois-Lavet et al., 2018; Mnih et al., 2015), is a major breakthrough towards solving complex decision making and control problems that were impractical for previous shallow RL methods.

Original: However, the outstanding performance of deep RL techniques is obtained at the cost of a massive amount of data needed from the interaction with the environment, hindering the practicality of deep RL methods.
Revised: However, the outstanding performance of deep RL techniques is obtained at the cost of a massive amount of data needed from the interaction with the environment, hindering the practicality of deep RL methods.

Original: Curriculum RL is a biologically inspired approach that frames the problem of learning a complex target task into learning a sequence of simpliﬁed, increasingly difﬁcult, versions of it (Florensa et al., 2017;
Revised: Curriculum RL is a biologically inspired approach that frames the problem of learning a complex target task into learning a sequence of simpliﬁed, increasingly difﬁcult, versions of it (Florensa et al., 2017;

Original: Shao et al., 2018; Ivanovic et al., 2019;
Revised: Shao et al., 2018; Ivanovic et al., 2019;

Original: Narvekar et al., 2020).
Revised: Narvekar et al., 2020).

Original: Commonly, establishing the complexity of a task in RL is not trivial, as an indisputable notion of complexity is still missing.
Revised: Commonly, establishing the complexity of a task in RL is not trivial, as an indisputable notion of complexity is still missing.

Original: However, from a practical point of view, we can consider a task complex if it requires targeted exploration in the environment or complex policies to be solved.
Revised: However, from a practical point of view, we can consider a task complex if it requires targeted exploration in the environment or complex policies to be solved.

Original: In curriculum RL, the notion of task complexity is key, and it is assumed that tailored tasks in a sequence of increasing complexity are presented to the learning agent.
Revised: In curriculum RL, the notion of task complexity is key, and it is assumed that tailored tasks in a sequence of increasing complexity are presented to the learning agent.

Original: Such an appropriate design is, however, a difﬁcult problem, requiring either human expertise (Narvekar et al., 2020) or solutions to automatize the selection of tasks following the learning of the agent (Jiang et al., 2015; Svetlik et al., 2017; Klink et al., 2020).
Revised: Such an appropriate design is, however, a difﬁcult problem, requiring either human expertise (Narvekar et al., 2020) or solutions to automatize the selection of tasks following the learning of the agent (Jiang et al., 2015; Svetlik et al., 2017; Klink et al., 2020).

Original: Regardless of using a handcrafted or an automatically generated curriculum, a common approach adopted by most value-based curriculum RL methods is to use the same function approximator throughout all the tasks (Narvekar et al., 2020): given a task, an approximation of the action-value function is learned, and used as an initialization for learning the action-value function of the next task in the curriculum.
Revised: Regardless of using a handcrafted or an automatically generated curriculum, a common approach adopted by most value-based curriculum RL methods is to use the same function approximator throughout all the tasks (Narvekar et al., 2020): given a task, an approximation of the action-value function is learned, and used as an initialization for learning the action-value function of the next task in the curriculum.

Original: For the success of this procedure, the function approximator needs to be powerful enough to handle the complexity of the target task.
Revised: For the success of this procedure, the function approximator needs to be powerful enough to handle the complexity of the target task.

Original: However, this requirement raises the key issue of adequately designing the function approximator.
Revised: However, this requirement raises the key issue of adequately designing the function approximator.

Original: A complex function approximator is prone to overﬁtting, and sensitive to hyper-parameters, being detrimental for learning simple tasks in the curriculum; conversely, using an overly simple function approximator can hinder the learning of the target task.
Revised: A complex function approximator is prone to overﬁtting, and sensitive to hyper-parameters, being detrimental for learning simple tasks in the curriculum; conversely, using an overly simple function approximator can hinder the learning of the target task.

Original: Moreover, choosing an adequate function approximator a-priori is not trivial, as the target task is only visited at the end, and it is commonly unsolvable by non-curriculum RL methods.
Revised: Moreover, choosing an adequate function approximator a-priori is not trivial, as the target task is only visited at the end, and it is commonly unsolvable by non-curriculum RL methods.

Original: In this paper, we introduce a novel approach that models the action-value function of a task as a sum of residuals learned on the previous tasks in the curriculum.
Revised: In this paper, we introduce a novel approach that models the action-value function of a task as a sum of residuals learned on the previous tasks in the curriculum.

Original: Our method increases the representativeness of the function approximator as new tasks are presented to the agent, leading to a procedure that increases the size of the functional space as the curriculum proceeds.
Revised: Our method increases the representativeness of the function approximator as new tasks are presented to the agent, leading to a procedure that increases the size of the functional space as the curriculum proceeds.

Original: This results in a learning procedure that increases the power of the function approximator according to the complexity of the task at hand, using small models in simple problems to avoid overﬁtting, and using large models (resulting from the sum of small ones) to handle the dimensionality of complex problems.
Revised: This results in a learning procedure that increases the power of the function approximator according to the complexity of the task at hand, using small models in simple problems to avoid overﬁtting, and using large models (resulting from the sum of small ones) to handle the dimensionality of complex problems.

Original: Notably, if the curriculum is sufﬁciently ﬁne-grained, the residuals become smoother functions, close to zero, that are easy to learn, even if the respective tasks are harder than the previous ones.
Revised: Notably, if the curriculum is sufﬁciently ﬁne-grained, the residuals become smoother functions, close to zero, that are easy to learn, even if the respective tasks are harder than the previous ones.

Original: We call our method boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting technique in supervised learning (Freund, 1995).
Revised: We call our method boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting technique in supervised learning (Freund, 1995).

Original: We provide a theoretical analysis of our approach under the lens of the approximate value iteration (AVI) framework (Farahmand, 2011).
Revised: We provide a theoretical analysis of our approach under the lens of the approximate value iteration (AVI) framework (Farahmand, 2011).

Original: In AVI, an estimate of the optimal action-value function is obtained as an iterative process that starts from an arbitrary estimate and applies the optimal Bellman operator until convergence.
Revised: In AVI, an estimate of the optimal action-value function is obtained as an iterative process that starts from an arbitrary estimate and applies the optimal Bellman operator until convergence.

Original: It is shown that two sources of error exist in AVI: (i) computation of the optimal Bellman operator, (ii) representation of the action-value function (Farahmand, 2011).
Revised: It is shown that two sources of error exist in AVI: (i) computation of the optimal Bellman operator, (ii) representation of the action-value function (Farahmand, 2011).

Original: While the former is due to the need of using samples to approximate the unknown optimal Bellman operator, the latter depends on the representativeness of the functional space chosen to approximate the action-value function.
Revised: While the former is due to the need of using samples to approximate the unknown optimal Bellman operator, the latter depends on the representativeness of the functional space chosen to approximate the action-value function.

Original: In this work, we formalize the curriculum RL problem in the AVI setting to show that the representation of the action-value function obtained by BCRL results in a tighter bound on the approximation error compared to regular curriculum RL, consequently reducing the performance loss.
Revised: In this work, we formalize the curriculum RL problem in the AVI setting to show that the representation of the action-value function obtained by BCRL results in a tighter bound on the approximation error compared to regular curriculum RL, consequently reducing the performance loss.

Original: We complement this analysis with an empirical evaluation of BCRL in AVI and RL settings, resorting to the ﬁtted Q -iteration( FQI ) algorithm (Ernst et al., 2005), least-squares policy iteration( LSPI ) (Lagoudakis & Parr, 2003) and deep
Revised: We complement this analysis with an empirical evaluation of BCRL in AVI and RL settings, resorting to the ﬁtted Q -iteration algorithm (Ernst et al., 2005), least-squares policy iteration (Lagoudakis & Parr, 2003) and deep

Original: Q -networks (Mnih et al., 2015), where we conﬁrm the advantages of BCRL over regular curriculum 
Revised: Q -networks (Mnih et al., 2015), conﬁrming the advantages of BCRL over regular curriculum RL.

Original: RL and a related baseline (Tosatto et al., 2017).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Learning under a dynamic set of training experience has been investigated in different ﬁelds throughout decades (Saul, 1941; Elman, 1993; Asada et al., 1996; Krueger & Dayan, 2009).
Revised: Learning under a dynamic set of training experience has been investigated in different ﬁelds throughout decades (Saul, 1941; Elman, 1993; Asada et al., 1996; Krueger & Dayan, 2009).

Original: A recent paper by Bengio et al.
Revised: A recent paper by Bengio et al.

Original: (2009) has established the term curriculum learning for the concept of organizing training data of a neural network in a beneﬁcial way for performance and generalization.
Revised: (2009) has established the term curriculum learning for the concept of organizing training data of a neural network in a beneﬁcial way for performance and generalization.

Original: This term carried over to the domain of RL, where curricula over training tasks have been shown to stabilize the highly challenging optimization problem of RL (Narvekar et al., 2020).
Revised: This term carried over to the domain of RL, where curricula over training tasks have been shown to stabilize the highly challenging optimization problem of RL (Narvekar et al., 2020).

Original: While a large body of research in the domain of curriculum reinforcement learning investigates the interesting problem of optimally selecting and scheduling the training tasks for an RL agent to optimize performance on a set of target tasks (Florensa et al., 2017; Ivanovic et al., 2019;
Revised: While a large body of research in the domain of curriculum reinforcement learning investigates the interesting problem of optimally selecting and scheduling the training tasks for an RL agent to optimize performance on a set of target tasks (Florensa et al., 2017; Ivanovic et al., 2019;

Original: Shao et al., 2018; Klink et al., 2020), the architecture of the learning agent is  assumed ﬁxed and the transfer of agent behavior between subsequent tasks in the curriculum is realized by this shared ﬁxed architecture.
Revised: Shao et al., 2018; Klink et al., 2020), the architecture of the learning agent is often assumed ﬁxed and the transfer of agent behavior between subsequent tasks in the curriculum is realized by this shared ﬁxed architecture.

Original: However, aside from investigations into the beneﬁt of an adequately chosen evolution of training experience, investigations of an optimal learning architecture (Elsken et al., 2019) and the evolution of this architecture during learning (Aoki & Siekevitz, 1988;
Revised: However, aside from investigations into the beneﬁt of an adequately chosen evolution of training experience, investigations of an optimal learning architecture (Elsken et al., 2019) and the evolution of this architecture during learning (Aoki & Siekevitz, 1988;

Original: Rumelhart et al., 1988; Ash, 1989;
Revised: Rumelhart et al., 1988; Ash, 1989;

Original: Fahlman, 1990; Lahnaj ¨ arvi et al., 2002; Nitanda & Suzuki, 2018) have been conducted in supervised learning scenarios.
Revised: Fahlman, 1990; Lahnaj ¨ arvi et al., 2002; Nitanda & Suzuki, 2018) have been conducted in supervised learning scenarios.

Original: The adaptation of these methods to the setting of RL has been limited to on-policy temporal-difference learning (Rivest & Precup, 2003; Vamplew & Ollington, 2005b;a) and it has not been further pursued in recent years.
Revised: The adaptation of these methods to the setting of RL has been limited to on-policy temporal-difference learning (Rivest & Precup, 2003; Vamplew & Ollington, 2005b;a) and it has not been further pursued in recent years.

Original: Our work aims to improve the information transfer between subsequent tasks in curriculum value-based reinforcement learning via a boosting procedure (Freund, 1995).
Revised: Our work aims to improve the information transfer between subsequent tasks in curriculum value-based reinforcement learning via a boosting procedure (Freund, 1995).

Original: The beneﬁt of the boosting framework is that it is well understood from a theoretical perspective (Tosatto et al., 2017), allowing for the derivation of error bounds, as also presented in this paper.
Revised: The beneﬁt of the boosting framework is that it is well understood from a theoretical perspective (Tosatto et al., 2017), allowing for the derivation of error bounds, as also presented in this paper.

Original: Further, it is agnostic to any value-based RL algorithm, avoiding the need to adjust the RL algorithm of choice to be suited for the evolution of the function approximator (Rivest & Precup, 2003; Vamplew & Ollington, 2005a;b).
Revised: Further, it is agnostic to any value-based RL algorithm, avoiding the need to adjust the RL algorithm of choice to be suited for the evolution of the function approximator (Rivest & Precup, 2003; Vamplew & Ollington, 2005a;b).

Revised: Aside from improving knowledge transfer via the learning architecture of the agent, the ﬁeld of transfer learning investigates various approaches for transferring knowledge, which we shortly review in Appendix A.

Original:  P RELIMINARIES 3. D EFINITIONS AND NOTATION
Revised:  P RELIMINARIES 3. D EFINITIONS AND NOTATION

Original: We recall the notions about Markov Decision Processes (MDPs) (Puterman, 1990), and the notation used for approximate value iteration (AVI) (Farahmand, 2011), that are relevant to our work.
Revised: We recall the notions about Markov Decision Processes (MDPs) (Puterman, 1990), and the notation used for approximate value iteration (AVI) (Farahmand, 2011), that are relevant to our work.

Original: For a space Σ , we deﬁne M (Σ) the set of probability measures over the σ -algebra σ Σ .
Revised: For a space Σ , we deﬁne M (Σ) the set of probability measures over the σ -algebra σ Σ .

Original: We denote
Revised: We denote

Original: B (Σ , B ) as the space of bounded measurable functions w.r.t.
Revised: B (Σ , B ) as the space of bounded measurable functions w.r.t.

Original: σ Σ with bound 0 <B< ∞ .
Revised: σ Σ with bound 0 <B< ∞ .

Original: A ﬁniteaction discounted MDP is deﬁned as a tuple T
Revised: A ﬁniteaction discounted MDP is deﬁned as a tuple T

Original: = (cid:104)S , A , P , R , γ (cid:105) , where S is a measurable state space,
Revised: = (cid:104)S , A , P , R , γ (cid:105) , where S is a measurable state space,

Original: A is a ﬁnite set of actions, P : S ×
Revised: A is a ﬁnite set of actions, P : S ×

Original: A → M ( S ) is a transition kernel, R : S ×
Revised: A → M ( S ) is a transition kernel, R : S ×

Original: A × S → R is a reward function, and γ ∈ [0 , 1) is a discount factor.
Revised: A × S → R is a reward function, and γ ∈ [0 , 1) is a discount factor.

Original: A policy π is a mapping from the state space S to a probability distribution over the action space A .
Revised: A policy π is a mapping from the state space S to a probability distribution over the action space A .

Original: A policy induces an action-value function Q π ( s, a ) (cid:44) E [Σ ∞ t =

Revised: A policy induces an action-value function Q π ( s, a ) (cid:44) E [Σ ∞ t =

Original: γ t R t | S 

Revised: γ t R t | S

Original: = s,
Revised: = s,

Original: A 

Revised: A

Original: = a ] , corresponding to the expected cumulative discounted reward obtained performing action a in state s and following the policy π thereafter.
Revised: = a ] , corresponding to the expected cumulative discounted reward obtained performing action a in state s and following the policy π thereafter.

Original: RL aims at solving an MDP by ﬁnding an optimal policy π ∗ which induces an optimal action-value function
Revised: RL aims at solving an MDP by ﬁnding an optimal policy π ∗ which induces an optimal action-value function

Original: Q ∗ ( s, a ) =
Revised: Q ∗ ( s, a ) =

Original: sup π Q π ( s, a ) .
Revised: sup π Q π ( s, a ) .

Original: For immediate rewards uniformly bounded by R max , the action-value function is bounded by Q max
Revised: For immediate rewards uniformly bounded by R max , the action-value function is bounded by Q max

Original: = R max / 1 − γ .
Revised: = R max / 1 − γ .

Original: For a probability measure µ ∈ M ( S × A ) , we deﬁne the
Revised: For a probability measure µ ∈ M ( S × A ) , we deﬁne the

Original: L p ( µ ) -norm of a measurable function Q ∈ B ( S × A ) as (cid:107) Q (cid:107) p,µ (cid:44) [ (cid:82) S×A | Q ( s, a ) | p dµ ( s, a )] 1 / p .
Revised: L p ( µ ) -norm of a measurable function Q ∈ B ( S × A ) as (cid:107) Q (cid:107) p,µ (cid:44) [ (cid:82) S×A | Q ( s, a ) | p dµ ( s, a )] 1 / p .

Original: Given a sequence of values x 1: n =( x 1 , . . .
Revised: Given a sequence of values x 1: n =( x 1 , . . .

Original: , x n ) for some space X , the empirical L p -norm of a function f : X → R is (cid:107) f (cid:107) p,x 1: n (cid:44) 1 n (cid:80) ni
Revised: , x n ) for some space X , the empirical L p -norm of a function f : X → R is (cid:107) f (cid:107) p,x 1: n (cid:44) 1 n (cid:80) ni

Original: =1 | f ( x i ) | p .
Revised: =1 | f ( x i ) | p .

Original: Conveniently, when X i ∼ µ , we have E [ (cid:107) f (cid:107) p,x 1: n ] = (cid:107) f (cid:107) p,µ .
Revised: Conveniently, when X i ∼ µ , we have E [ (cid:107) f (cid:107) p,x 1: n ] = (cid:107) f (cid:107) p,µ .

Original: From now on, we consider the L 2 -norm when the subscript p is omitted.
Revised: From now on, we consider the L 2 -norm when the subscript p is omitted.

Original: Operators.
Revised: Operators.

Original: We deﬁne the operator P π as ( P π Q )( s, a ) (cid:44) (cid:82) S P ( ds (cid:48) | s, a ) (cid:80) a (cid:48)
Revised: We deﬁne the operator P π as ( P π Q )( s, a ) (cid:44) (cid:82) S P ( ds (cid:48) | s, a ) (cid:80) a (cid:48)

Original: ∈A π ( a (cid:48) | s (cid:48) )
Revised: ∈A π ( a (cid:48) | s (cid:48) )

Original: Q ( s (cid:48) , a (cid:48) ) , for any given policy π .
Revised: Q ( s (cid:48) , a (cid:48) ) , for any given policy π .

Original: Then, the Bellman operator T π : B ( S × A )
Revised: Then, the Bellman operator T π : B ( S × A )

Original: → B ( S × A ) is ( T π Q )( s, a ) (cid:44) r
Revised: → B ( S × A ) is ( T π Q )( s, a ) (cid:44) r

Original: ( s, a )
Revised: ( s, a )

Original: + γ ( P π Q )( s, a ) .
Revised: + γ ( P π Q )( s, a ) .

Original: The optimal Bellman operator T ∗ : B ( S × A )
Revised: The optimal Bellman operator T ∗ : B ( S × A )

Original: → B ( S × A ) is ( T ∗ Q )( s, a ) (cid:44) r
Revised: → B ( S × A ) is ( T ∗ Q )( s, a ) (cid:44) r

Original: ( s, a ) + γ (cid:82) S max a (cid:48) P ( ds (cid:48) | s, a ) Q ( s (cid:48) , a (cid:48) )) .
Revised: ( s, a ) + γ (cid:82) S max a (cid:48) P ( ds (cid:48) | s, a ) Q ( s (cid:48) , a (cid:48) )) .

Original: Finally, the ﬁxed point of the Bellman operator is ( T π Q π ) =
Revised: Finally, the ﬁxed point of the Bellman operator is ( T π Q π ) =

Original: Q π , and the ﬁxed point of the optimal Bellman operator is ( T ∗ Q ∗ ) =
Revised: Q π , and the ﬁxed point of the optimal Bellman operator is ( T ∗ Q ∗ ) =

Original: Q ∗ .
Revised: Q ∗ .

Original: As in Gy ¨ orﬁ et al.
Revised: As in Gy ¨ orﬁ et al.

Original: (2006), we deﬁne the truncation operator β B : B ( S × A )
Revised: (2006), we deﬁne the truncation operator β B : B ( S × A )

Original: → B ( S × A , B ) such that, given a function f ∈ B ( S × A ) , we have β B f ( s, a ) ∈ [ − B, B ] , ∀ ( s, a ) ∈ S × A .
Revised: → B ( S × A , B ) such that, given a function f ∈ B ( S × A ) , we have β B f ( s, a ) ∈ [ − B, B ] , ∀ ( s, a ) ∈ S × A .

Original: 3. A PPROXIMATE VALUE ITERATION
Revised: 3. A PPROXIMATE VALUE ITERATION

Original: The family of approximate value iteration (AVI) algorithms (Sutton & Barto, 2018; Farahmand, 2011; Munos, 2005) aims at approximating the solution of RL problems with large state space where ﬁnding an optimal solution is impractical or impossible.
Revised: The family of approximate value iteration (AVI) algorithms (Sutton & Barto, 2018; Farahmand, 2011; Munos, 2005) aims at approximating the solution of RL problems with large state space where ﬁnding an optimal solution is impractical or impossible.

Original: Starting from an arbitrary estimate
Revised: Starting from an arbitrary estimate

Original: Q 0 , AVI algorithms iteratively approximate the application of the optimal Bellman operator in a suitable functional space, i.e., Q k +1 ≈ T ∗ Q k .
Revised: Q 0 , AVI algorithms iteratively approximate the application of the optimal Bellman operator in a suitable functional space, i.e., Q k +1 ≈ T ∗ Q k .

Original: Intuitively, the approximation error at each iteration plays a central role in terms of convergence to the ﬁxed point Q ∗ .
Revised: Intuitively, the approximation error at each iteration plays a central role in terms of convergence to the ﬁxed point Q ∗ .

Original: The approximation error results from two key aspects: (i) the unknown optimal Bellman operator, and (ii) the representation of the action-value function in a suitable functional space F via the application of a nonlinear operator
Revised: The approximation error results from two key aspects: (i) the unknown optimal Bellman operator, and (ii) the representation of the action-value function in a suitable functional space F via the application of a nonlinear operator

Original: S : B ( S × A ) → F deﬁned as Sy = arg inf f ∈F (cid:107) f − y (cid:107) 2 µ for any y ∈ B ( S × A ) . 
Revised: S : B ( S × A ) → F deﬁned as Sy = arg inf f ∈F (cid:107) f − y (cid:107) 2 µ for any y ∈ B ( S × A ) . Then, we deﬁne Q k +1 =

Original: We denote the resulting approximation error by
Revised: S ( T ∗ Q k ) , and the resulting approximation error by

Original: [Equation]
Revised: [Equation]

Original: The role of the approximation error in AVI is evinced by the following theorem that bounds the performance loss, w.r.t.
Revised: The role of the approximation error in AVI is evinced by the following theorem that bounds the performance loss, w.r.t.

Original: the optimal performance, at iteration K 1 .
Revised: the optimal performance, at iteration K 1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: ((Farahmand, 2011)).
Revised: ((Farahmand, 2011)).

Original: Let K be a positive integer, ρ an initial state distribution, and
Revised: Let K be a positive integer, ρ an initial state distribution, and

Original: Q max ≤ R max 1 − γ .
Revised: Q max ≤ R max 1 − γ .

Original: Then, for any sequence ( Q k ) Kk =

Revised: Then, for any sequence ( Q k ) Kk =

Original: ⊂ B ( X × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 deﬁned in (1), we have[Equation] where
Revised: ⊂ B ( X × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 , we have

Original: [Equation]
Revised: [Equation]

Original: − r ) k (cid:34) (cid:88) m ≥γ m (cid:16) c VI 1 ,ρ,µ ( m, K − k ; π (cid:48) K )+ c VI 2 ,ρ,ν ( m + 1; π (cid:48) k +1 , . . . , π (cid:48) K ) (cid:17)(cid:35)and E ( ε 0 , . ..
Revised: where E ( ε 0 , . .

Original:  , ε K − 1 ; r ) =
Revised: . , ε K − 1 ; r ) =

Original: (cid:80) K − [-1 k =
-]
Revised: (cid:80) K − [-1 k =
-][Equation] =

Original: α 2 rk (cid:107) ε k (cid:107) 2 µ .
Revised: [Equation] (cid:107) [Equation] µ .

Original: We refer to Farahmand (2011) for the deﬁnition of c VI 1 , c VI2 , and α i .
Revised: We refer to Farahmand (2011) for the deﬁnition of the concentrability coefﬁcients C VI and α i .

Revised: Notably, Theorem 1 shows that the magnitude of the difference between the optimal action-value function Q ∗ and the estimate at the K -th iteration Q K , is proportional to the magnitude of the approximation errors ε k as deﬁned in Equation 1.

Original:  C URRICULUM REINFORCEMENT LEARNING WITH B ELLMAN RESIDUALS
Revised:  C URRICULUM REINFORCEMENT LEARNING WITH B ELLMAN RESIDUALS

Original: We consider the classic curriculum RL scenario where an agent aims at learning a complex target task by solving a predeﬁned sequence of easier, but increasingly challenging, versions of it (Narvekar et al., 2020).
Revised: We consider the classic curriculum RL scenario where an agent aims at learning a complex target task by solving a predeﬁned sequence of easier, but increasingly challenging, versions of it (Narvekar et al., 2020).

Original: To avoid confusion on the debatable notion of difﬁculty of a task, in this work we adopt a practical point of view where the difﬁculty of a task is related to well-known aspects of RL, e.g., the complexity of the policies for solving them and challenging exploration (Florensa et al., 2017; Ivanovic et al., 2019; Klink et al., 2020).
Revised: To avoid confusion on the debatable notion of difﬁculty of a task, in this work we adopt a practical point of view where the difﬁculty of a task is related to well-known aspects of RL, e.g., the complexity of the policies for solving them and challenging exploration (Florensa et al., 2017; Ivanovic et al., 2019; Klink et al., 2020).

Original: 4. A PPROXIMATE VALUE ITERATION IN CURRICULUM REINFORCEMENT LEARNING
Revised: 4. A PPROXIMATE VALUE ITERATION IN CURRICULUM REINFORCEMENT LEARNING

Original: Let T 1: n = {T 1 , . .
Revised: Let T 1: n = {T 1 , . .

Original: . , T n } be a sequence of increasingly complex MDPs sharing the same state and action spaces, i.e., T i
Revised: . , T n } be a sequence of increasingly complex MDPs sharing the same state and action spaces, i.e., T i

Original: =
Revised: =

Original: (cid:104)S , A , P i , R i , γ (cid:105) , and T n be the target task.
Revised: (cid:104)S , A , P i , R i , γ (cid:105) , and T n be the target task.

Original: The use of AVI algorithms in this curriculum RL setting consists in starting from an arbitrary estimate Q 01 (where the subscript indicates the position of the task in the curriculum) and approximating the application of the taskspeciﬁc optimal Bellman operator for the current task Q k +1 t ≈ T ∗ t Q kt, or when the task at hand changes Q k +1 t +1 ≈ T ∗ t +1 Q kt .
Revised: The use of AVI algorithms in this curriculum RL setting consists in starting from an arbitrary estimate Q 01 (where the subscript indicates the position of the task in the curriculum) and approximating the application of the taskspeciﬁc optimal Bellman operator for the current task Q k +1 t ≈ T ∗ t Q kt .

Original: We assume that the task changes when the estimate of the action-value of the current task converges.
Revised: We assume that the task changes when the estimate of the action-value of the current task converges.

Original: For simplicity, and without loss of generality, we denote K t the number of iterations needed to converge for each task T t , and denote Q 0 t =
Revised: For simplicity, and without loss of generality, we denote K t the number of iterations needed to converge for each task T t , and denote Q 0 t =

Original: Q K t − 1 t − 1 =
Revised: Q K t − 1 t − 1 =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let K be the current iteration.
Revised: Let K be the current iteration.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: This result can be veriﬁed applying the triangle inequality under the assumption that the algorithm switches task when the stopping condition of the algorithm is veriﬁed, e.g., when reaching a chosen number of iterations K t that allows the action-value function estimate to converge, or using heuristic methods checking the magnitude of the update (Lagoudakis & Parr, 2003; Ernst et al., 2005).
Revised: This result can be veriﬁed applying the triangle inequality under the assumption that the algorithm switches task when the stopping condition of the algorithm is veriﬁed, e.g., when reaching a chosen number of iterations K t that allows the action-value function estimate to converge, or using heuristic methods checking the magnitude of the update (Lagoudakis & Parr, 2003; Ernst et al., 2005).

Original: Note that each component of the sum in (2) can be bounded by the result in Theorem 1.
Revised: Note that each component of the sum in (2) can be bounded by the result in Theorem 1.

Original: We denote the k -step Bellman residual for task t by
Revised: We denote the k -step Bellman residual for task t by

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Given an initial Q 01 ( s, a )=0 for any ( s, a ) ∈ S × A , and considering that
Revised: Given an initial Q 01 ( s, a )=0 for any ( s, a ) ∈ S × A , and considering that

Original: Q ∗ t = 
Revised: Q ∗ t = Q K t t

Original: [Equation] ) K t Q ∗ t − 1 , we obtain 
Revised: ≈ ( T ∗ t ) K t Q ∗ t − 1 , we obtain (cid:37) kn + n − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: This result shows that, approximately, the action-value function of the target task is equivalent to the sum of k -step Bellman residuals.
Revised: This result shows that, approximately, the action-value function of the target task is equivalent to the sum of k -step Bellman residuals.

Original: Notably, especially if the curriculum is sufﬁciently ﬁne-grained, residuals becomes progressively smoother and smaller.
Revised: Notably, especially if the curriculum is sufﬁciently ﬁne-grained, residuals becomes progressively smoother and smaller.

Original: In the following, we describe how to compute the estimate shown in (4).
Revised: In the following, we describe how to compute the estimate shown in (4).

Original: 4. B OOSTED CURRICULUM REINFORCEMENT LEARNING
Revised: 4. B OOSTED CURRICULUM REINFORCEMENT LEARNING

Original: It is well-known from supervised learning that: (i) small function approximators are easier to train, and thus are desirable to solve simple tasks; (ii) large function approximators can overﬁt or suffer slow training speed in simple tasks, but they are more suitable to handle complex problems.
Revised: It is well-known from supervised learning that: (i) small function approximators are easier to train, and thus are desirable to solve simple tasks; (ii) large function approximators can overﬁt or suffer slow training speed in simple tasks, but they are more suitable to handle complex problems.

Original: Ideally, we want to increase the representativeness of the function approximator with the increase of task complexity.
Revised: Ideally, we want to increase the representativeness of the function approximator with the increase of task complexity.

Original: Recalling (4), we propose to use a separate function approximator for each residual (cid:37) t , i.e. a separate approximator for each task.
Revised: Recalling (4), we propose to use a separate function approximator for each residual (cid:37) t , i.e. a separate approximator for each task.

Original: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni =1 for each task, we denote the empirical Bellman operator by ( ˆ T ∗ t Q )( s i t , a it ) (cid:44) r
Revised: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni =1 for each task, we denote the empirical Bellman operator by ( ˆ T ∗ t Q )( s i t , a i t ) (cid:44) r

Original: i t +
Revised: i t +

Original: γ max a (cid:48) Q ( s (cid:48)
Revised: γ max a (cid:48) Q ( s (cid:48)

Original: i t , a (cid:48) ) , and deﬁne the k -step empirical residual as ˜ (cid:37) kt (cid:44) ( ˆ T ∗ t ) k Q 0 t − Q 0 t .
Revised: i t , a (cid:48) ) , and deﬁne the k -step empirical residual as ˜ (cid:37) kt (cid:44) ( ˆ T ∗ t ) k Q 0 t − Q 0 t .

Original: At iteration k , for a current task t , and deﬁning the nonlinear operator ˆ S  as the empirical version of the operator S , we compute an approximation
Revised: At iteration k , for a current task t , and deﬁning the nonlinear operator ˆ S ˜ (cid:37) kt = arg inf f ∈F (cid:107) f − ˜ (cid:37) kt (cid:107) 2 D t as the empirical version of the operator S , we compute an approximation

Original: [Equation]
Revised: [Equation]

Original: Input: Tasks T 1: n , t = 1 , dataset D , Q 01 = 0 ; while t < n + 1 do for k = 0 , . . , K t do ˜ (cid:37) kt ← ˆ T ∗ t Q kt − Q 0 t ; Q k +1 t ← Q 0 t + β B k arg inf f ∈F (cid:107) f − ˜ (cid:37) kt (cid:107) µ ; end
Revised: where we deﬁned ˆ (cid:37) kt = β B k ˆ S ˜ (cid:37) kt for compactness.

Original: Q 0 t +1 ← Q K t t ; t ← t + 1 ; end Output: Q K n n = (cid:80) nt = 1 ˆ (cid:37) K t t . where we deﬁned ˆ (cid:37) kt = β B k ˆ S ˜ (cid:37) kt for compactness.
Revised: For ease of notation, we use Q kt also to denote the Q -function obtained by applying the empirical Bellman operator.

Original: We call this approach boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting procedure in supervised learning (Schapire, 1990; Freund, 1995), and we show its pseudocode in Algorithm 1.
Revised: We call this approach boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting procedure in supervised learning (Schapire, 1990; Freund, 1995), and we show its pseudocode in Algorithm 1.

Original: Remarks.
Revised: Remarks.

Original: Our approach is based on the key insight that the strength of the function approximator should increase according to the complexity of the task at hand.
Revised: Our approach is based on the key insight that the strength of the function approximator should increase according to the complexity of the task at hand.

Original: In BCRL , we accomplish this estimating an action-value function through a sum of action-value functions of the previous tasks in the curriculum.
Revised: In BCRL , we accomplish this estimating an action-value function through a sum of action-value functions of the previous tasks in the curriculum.

Original: Thus, contrary to methods using the same functional space for all tasks, our approach has the advantage of acting in an augmented functional space resulting from the sum of the functional space of each approximator.
Revised: Thus, contrary to methods using the same functional space for all tasks, our approach has the advantage of acting in an augmented functional space resulting from the sum of the functional space of each approximator.

Original: In supervised learning literature, the function approximators in the sum are commonly known as weak regressors, meaning that boosting has the advantage of estimating complex functions as a sum of simple regressors (Geurts et al., 2006).
Revised: In supervised learning literature, the function approximators in the sum are commonly known as weak regressors, meaning that boosting has the advantage of estimating complex functions as a sum of simple regressors (Geurts et al., 2006).

Original: Similarly, our
Revised: Similarly, our

Original: BCRL procedure can use simple function approximators for each task in the curriculum, with the advantages that (i) easy tasks can be learned with easy function approximators curbing the risk of overﬁtting, (ii) the complexity of the function approximator can be chosen according to the ﬁrst tasks in the curriculum, contrarily to regular curriculum RL where the function approximator needs to be suitable for the target task, and (iii) the training time to ﬁt the model on each task is reduced due to the smoothness of the residuals.
Revised: BCRL procedure can use simple function approximators for each task in the curriculum, with the advantages that (i) easy tasks can be learned with easy function approximators curbing the risk of overﬁtting, (ii) the complexity of the function approximator can be chosen according to the ﬁrst tasks in the curriculum, contrarily to regular curriculum RL where the function approximator needs to be suitable for the target task, and (iii) the training time to ﬁt the model on each task is reduced due to the smoothness of the residuals.

Original: Conversely, at inference time, our method requires a forward pass for each approximator in the ensemble.
Revised: Conversely, at inference time, our method requires a forward pass for each approximator in the ensemble.

Original: Nevertheless, the small size of the regressors results in negligible increase of computation time.
Revised: Nevertheless, the small size of the regressors results in negligible increase of computation time.

Original: Finally, we clarify that the soundness of this procedure is guaranteed by the fact that if the functional space of each weak regressor is Glivenko-Cantelli, i.e., ”the error due to the empirical process goes to zero at least asymptotically”, then, due to the preservation theorem, also the space resulting from the sum of weak regressors is Glivenko-Cantelli (Van
Revised: Finally, we clarify that the soundness of this procedure is guaranteed by the fact that if the functional space of each weak regressor is Glivenko-Cantelli, i.e., ”the error due to the empirical process goes to zero at least asymptotically”, then, due to the preservation theorem, also the space resulting from the sum of weak regressors is Glivenko-Cantelli (Van

Original: Der Vaart & Wellner, 2000; Tosatto et al., 2017).
Revised: Der Vaart & Wellner, 2000; Tosatto et al., 2017).

Original:  T HEORETICAL ANALYSIS
Revised:  T HEORETICAL ANALYSIS

Original: In this section, we provide a theoretical analysis of BCRL in terms of convergence guarantees and ﬁnite-sample approximation error analysis.
Revised: We provide a theoretical analysis of BCRL in terms of convergence guarantees and ﬁnite-sample approximation error analysis.

Original: For simplicity and ease of notation, we assume the tasks in the curriculum have the same transition kernels, and different reward functions.
Revised: We assume the tasks in the curriculum have the same transition kernels, and different reward functions.

Original: Nevertheless, our results can be straightforwardly extended to the general case.
Revised: Nevertheless, our results can be straightforwardly extended to the general case.

Original: We refer to the well-known results in AVI (Farahmand, 2011; Farahmand et al., 2010), and the analysis provided in Tosatto et al. (2017), with which our analysis share some similarities.
Revised: We refer to the well-known results in AVI (Farahmand, 2011; Farahmand et al., 2010), and the analysis provided in Tosatto et al. (2017), with which our analysis share some similarities.

Original: 5. C ONVERGENCE GUARANTEES
Revised: 5. C ONVERGENCE GUARANTEES

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q i t ) k
Revised: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q i t ) k

Original: i =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =
Revised: i =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.
Revised: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.

Original: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )
Revised: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )

Original: y (cid:107) µ ≤ λ (cid:107) y
Revised: y (cid:107) µ ≤ λ (cid:107) y

Original: (cid:107) µ for all y ∈ B ( S × A ) .
Revised: (cid:107) µ for all y ∈ B ( S × A ) .

Original: Then, (cid:107) Q k t − Q ∗ t (cid:107) µ ≤ ((1 + L k ) λ + L k ) (cid:107) Q 0 t − Q ∗ t (cid:107) µ .
Revised: Then, [Equation] Note that the assumption on the operator S stated by Theorem 2 dictates that the approximation power of the used functional space should be good enough to learn the target function y ; more speciﬁcally, the upper bound of the approximation error is proportional to the magnitude of the function to approximate. This assumption holds for reasonable choices of function approximators and it is present in related works on the derivation of ﬁnite-time bounds for policy iteration (Munos, 2003), and value iteration (Munos, 2005; Tosatto et al., 2017).

Original: We observe that the boosting procedure adds the (1 + L k ) λ term in the regular AVI bound, that is obtained for λ = 0 .
Revised: We observe that the boosting procedure adds the (1 + L k ) λ term in the regular AVI bound, that is obtained for λ = 0 .

Original: This result shows that for each task, for appropriate values of λ and L , the boosting procedure converges to the optimal action-value.
Revised: This result shows that for each task, for appropriate values of λ and L , the boosting procedure converges to the optimal action-value.

Original: Corollary 1.
Revised: Corollary 1.

Original: Given the settings of Theorem 2, the sequence of ( Q it ) ki =0 converges to Q ∗ t whenλ < 1 −
Revised: Given the settings of Theorem 2, the sequence of ( Q it ) ki =0 converges to Q ∗ t when

Original: L k 1 + L k and L k < 1 .
Revised: [Equation]

Original: 5. F INITE - SAMPLE APPROXIMATION ERROR ANALYSIS
Revised: 5. F INITE - SAMPLE APPROXIMATION ERROR ANALYSIS

Original: In Lemma 1, we show how to derive the AVI performance loss analysis (Farahmand, 2011) for curriculum RL.
Revised: In Lemma 1, we show how to derive the AVI performance loss analysis (Farahmand, 2011) for curriculum RL.

Original: Now, we want to characterize it for our BCRL method.
Revised: Now, we want to characterize it for our BCRL method.

Original: Instead of using the regular
Revised: Instead of using the regular

Original: AVI approximation error deﬁned in (1), we consider the approximation error for each task t in the curriculum deﬁned as
Revised: AVI approximation error deﬁned in (1), we consider the approximation error for each task t in the curriculum deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.

Original: Denoting the regular AVI residual as ¯ (cid:37) kt
Revised: Denoting the regular AVI residual as ¯ (cid:37) kt

Original: = ( T ∗ t ) k . . .
Revised: = ( T ∗ t ) k . . .

Original: ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t − 1 ) K t − 1 . . .
Revised: ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t − 1 ) K t − 1 . . .

Original: ( T ∗ 1 ) K 1 Q 01 , we have
Revised: ( T ∗ 1 ) K 1 Q 01 , we have

Original: [Equation]
Revised: [Equation]

Original: This bound shows the relation between the approximation error of the residual for task t at step k , the AVI Bellman Residual ¯ (cid:37) kt , and the approximation error for the residuals of the previous tasks.
Revised: This bound shows the relation between the approximation error of the residual for task t at step k , the AVI Bellman Residual ¯ (cid:37) kt , and the approximation error for the residuals of the previous tasks.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: We use (9) and (10) to ﬁnally bound the approximation error of the residual as follows.
Revised: We use (9) and (10) to ﬁnally bound the approximation error of the residual as follows.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let B tk = max( (cid:107) ˜(cid:37)
Revised: Let B t k = max( (cid:107) ˜

Original: kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).
Revised: (cid:37) k t (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).

Original: For given dataset D t = { ( s i t , a i t , r i t , s (cid:48) i t ) } N i = for each task t , and a given sequence of action-value functions ( Q it ) 
Revised: For given dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni = for each task t , and a given sequence of action-value functions ( Q it ) ki =0 generated by our BCRL , we have

Original: ki =0 generated by our BCRL , we have
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This result bounds the error of BCRL by a sum of three components: (i) the approximation error of the residual ¯ (cid:37) k t , (ii) the propagation error due to approximation errors at previous iterations, (iii) the estimation error due to the regression problem with ﬁnite amount of samples.
Revised: This result bounds the error of BCRL by a sum of three components: (i) the approximation error of the residual ¯ (cid:37) k t , (ii) the propagation error due to approximation errors at previous iterations, (iii) the estimation error due to the regression problem with ﬁnite amount of samples.

Original: The advantages of
Revised: The advantages of

Original: BCRL are motivated by the use of residuals ¯ (cid:37) kt and ˜ (cid:37) kt , whose range of values become smaller as the curriculum proceeds, resulting in tighter approximation and estimation error bounds compared to regular curriculum RL.
Revised: BCRL are motivated by the use of residuals ¯ (cid:37) kt and ˜ (cid:37) kt , whose range of values become smaller as the curriculum proceeds, resulting in tighter approximation and estimation error bounds compared to regular curriculum RL.

Original: Moreover, BCRL enjoys advantages in terms of sample-efﬁciency, as fewer samples than regular curriculum RL can be used to achieve comparable estimation errors, due to the smaller range and smoother shape of the target (cid:107) ˜ (cid:37) kt (cid:107) µ compared to Q max t .
Revised: Moreover, BCRL enjoys advantages in terms of sample-efﬁciency, as fewer samples than regular curriculum RL can be used to achieve comparable estimation errors, due to the smaller range and smoother shape of the target (cid:107) ˜ (cid:37) kt (cid:107) µ compared to Q max t .

Original: Iteration 0 .0 .0 .

Revised: Iteration 0 .0 .0 .

Original: .k
Revised: .k

Original: Q k t − Q ∗ t k 1 , µ
Revised: Q k t − Q ∗ t k 1 , µ

Original: T
Revised: T

Original: T

Original: T
Revised: T

Original:  BC-FQI B-FQI
Revised: T 3 BC-FQI B-FQI

Original: C-FQI FQI
Revised: C-FQI FQI

Original: Iteration − 0 .0 .0 .0 .
Revised: Iteration − 0 .0 .0 .0 .

Original: C u m .
Revised: C u m .

Original: D i s c .
Revised: D i s c .

Original: R e t u r n
Revised: R e t u r n

Original: T T
Revised: T T

Original: T
Revised: T

Original: BC-FQI B-FQI
Revised: BC-FQI B-FQI

Original: C-FQI FQI
Revised: C-FQI FQI

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section serves to provide empirical evidence to the insights obtained in the previous section.
Revised: This section serves to provide empirical evidence to the insights obtained in the previous section.

Original: More precisely, we want to provide evidence that boosted curricula can indeed improve the residual in an AVI setting and with that improve the achieved agent performance.
Revised: More precisely, we want to provide evidence that boosted curricula can indeed improve the residual in an AVI setting and with that improve the achieved agent performance.

Original: Further, we show that the additional ﬂexibility of boosting w.r.t.
Revised: Further, we show that the additional ﬂexibility of boosting w.r.t.

Original: the individual function approximators can be leveraged to choose the best-suited learning approach for each learning task.
Revised: the individual function approximators can be leveraged to choose the best-suited learning approach for each learning task.

Original: We conduct experiments in multiple environments and with two different RL algorithms using the
Revised: We conduct experiments in multiple environments and with two different RL algorithms using the

Original: MushroomRL library (D’Eramo et al., 2021).
Revised: MushroomRL library (D’Eramo et al., 2021).

Original: 2 To evaluate the beneﬁt of curricula in boosted AVI settings, we turn towards the ﬁtted Q -iteration ( FQI ) algorithm in the car-on-hill environment (Ernst et al., 2005), as the low-dimensionality allows for uniform sampling of the state-action space assumed in the previous section and the ground truth optimal Q -function can be computed via bruteforce methods (Ernst et al., 2005).
Revised: 2 To evaluate the beneﬁt of curricula in boosted AVI settings, we turn towards the ﬁtted Q -iteration ( FQI ) algorithm in the car-on-hill environment (Ernst et al., 2005), as the low-dimensionality allows for uniform sampling of the state-action space assumed in the previous section and the ground truth optimal Q -function can be computed via bruteforce methods (Ernst et al., 2005).

Original: Next, we evaluate the FQI algorithm in a maze environment in which exploration is guided by the currently estimated optimal policy.
Revised: Next, we evaluate the FQI algorithm in a maze environment in which exploration is guided by the currently estimated optimal policy.

Original: Finally, we investigate boosting in a control task with DQN (Mnih et al., 2015), in which a point-mass needs to be steered over a narrow pathway subject to external perturbations.
Revised: Finally, we investigate boosting in a control task with DQN (Mnih et al., 2015), in which a point-mass needs to be steered over a narrow pathway subject to external perturbations.

Original: To disentangle the interplay between boosting and the use of a curriculum, we will compare different methods in the experiments: the algorithm under investigation ( FQI , DQN ), a version using both a curriculum with boosting ( BC - FQI , BC - DQN ) as well as two ablations of the second method - one without boosting ( C - FQI , C - DQN ) and one without a curriculum ( B - FQI, B - DQN ).
Revised: To disentangle the interplay between boosting and the use of a curriculum, we will compare different methods in the experiments: the algorithm under investigation ( FQI , DQN ), a version using both a curriculum with boosting ( BC - FQI , BC - DQN ) as well as two ablations of the second method - one without boosting ( C - FQI , C - DQN ) and one without a curriculum ( B - FQI ).

Original: The boosted methods that do not use a curriculum will introduce the additional approximators at the same iteration at which the curricula switches between tasks.
Revised: The boosted methods that do not use a curriculum will introduce the additional approximators at the same iteration at which the curricula switches between tasks.

Original: 6. C AR - ON - HILL
Revised: 6. C AR - ON - HILL

Original: The three car-on-hill tasks T 1 , T 2 and T 3 considered for curriculum learning in this experiment differ in the mass of the car, taking values of 0 .
Revised: The three car-on-hill tasks T 1 , T 2 and T 3 considered for curriculum learning in this experiment differ in the mass of the car, taking values of 0 .

Original: 8 , 1 . 0 and 1 . 2 respectively.
Revised: 8 , 1 . 0 and 1 . 2 respectively.

Original: The default mass of the environment is 1 . 

Revised: The default mass of the environment is 1 .

Original: which means we are considering a version of this environment with more challenging exploration, as the car needs more energy to escape the valley and receive a reward signal.
Revised: which means we are considering a version of this environment with more challenging exploration, as the car needs more energy to escape the valley and receive a reward signal.

Original: The dataset for learning the Q -function in each task is generated by executing 1000 trajectories with a random policy.
Revised: The dataset for learning the Q -function in each task is generated by executing 1000 trajectories with a random policy.

Original: To not make the results in this section dependent on the parametric form of the function approximator, we resort to the tree-based regression investigated in Ernst et al.
Revised: To not make the results in this section dependent on the parametric form of the function approximator, we resort to the tree-based regression investigated in Ernst et al.

Original: The results are visualized in Figure 1.
Revised: The results are visualized in Figure 1.

Original: We can see that the combination of boosting and a curriculum ( BC - FQI ) outperforms all other methods both in terms of ﬁnal approximation error achieved on Tbut also in terms of the performance of the greedy policy derived from the estimated Q -function.
Revised: We can see that the combination of boosting and a curriculum ( BC - FQI ) outperforms all other methods both in terms of ﬁnal approximation error achieved on Tbut also in terms of the performance of the greedy policy derived from the estimated Q -function.

Original: Comparing the residual of BC - FQI and C - FQI , we can see that C - FQI seems to be challenged with adapting the Q -function from T 1 to T 2 , as the jump in residual shows.
Revised: Comparing the residual of BC - FQI and C - FQI , we can see that C - FQI seems to be challenged with adapting the Q -function from T 1 to T 2 , as the jump in residual shows.

Original: BC - FQI does not exhibit such a jump in residual when moving from T 1 to T 2 .
Revised: BC - FQI does not exhibit such a jump in residual when moving from T 1 to T 2 .

Original: Furthermore, BC - FQI reduces the increased residual after switching from T 2 to T 3 while the residual of C - FQI steadily increases once arriving at the target task.
Revised: Furthermore, BC - FQI reduces the increased residual after switching from T 2 to T 3 while the residual of C - FQI steadily increases once arriving at the target task.

Original: We can particularly see that the pre-training in T 1 and T 2 leads to a signiﬁcantly smaller
Revised: We can particularly see that the pre-training in T 1 and T 2 leads to a signiﬁcantly smaller

Revised: Q -function error of BC - FQI compared B - FQI at iteration 40 , which is in line with the discussion in the previous sections.

Original: [Figure]
Revised: [Figure]

Original: k Q k t − Q ∗ t k 1 , µ ... T 5 T
Revised: [Figure] Q k t − Q ∗ t k 1 , µ ... T 5 T

Original: T
Revised: T

Original: T T
Revised: T T

Original: T 10 BC-FQI
Revised: T 10 BC-FQI

Original: B-FQI
Revised: B-FQI

Original: C-FQI FQI     

Revised: C-FQI FQI

Original: Iteration −−−−−
Revised: Iteration −−−−−

Original: C u m .
Revised: C u m .

Original: [Figure]
Revised: [Figure]

Original: i s c .
Revised: i s c .

Original: R e t u r n ... T 5 T 6 T
Revised: R e t u r n ... T 5 T 6 T

Original: T
Revised: T

Original: T T
Revised: T T

Original: [Figure]
Revised: [Figure]

Original: Q -function error of BC - FQI compared B - FQI at iteration 40 , which is in line with the discussion in the previous sections.

Original: 6. M AZE
Revised: 6. M AZE

Original: We design a maze environment, in which an agent needs to reach a target position by navigating around two large obstacles.
Revised: We design a maze environment, in which an agent needs to reach a target position by navigating around two large obstacles.

Original: To reach the goal, the agent can choose between 4 discrete actions that move the agent up, down, left, or right.
Revised: To reach the goal, the agent can choose between 4 discrete actions that move the agent up, down, left, or right.

Original: It obtains a dense reward based on the distance to the target and a reward of +5 upon reaching the goal.
Revised: It obtains a dense reward based on the distance to the target and a reward of +5 upon reaching the goal.

Original: We design a curriculum composed oftasks, in which the size of the walls gradually increases (see Figure 3a for a visualization of tasks contained in the curriculum).
Revised: We design a curriculum composed oftasks, in which the size of the walls gradually increases (see Figure 3a for a visualization of tasks contained in the curriculum).

Original: The reward function and the target task are designed to induce highly challenging exploration, requiring the use of curricula to solve the problem.
Revised: The reward function and the target task are designed to induce highly challenging exploration, requiring the use of curricula to solve the problem.

Original: As done for the caron-hill experiment, we use FQI with tree-based regression (Ernst et al., 2005).
Revised: As done for the caron-hill experiment, we use FQI with tree-based regression (Ernst et al., 2005).

Original: For all methods, we collect a dataset for each task running 500 episodes using ε -greedy exploration, decreasing the strength of exploration as the curriculum proceeds.
Revised: For all methods, we collect a dataset for each task running 500 episodes using ε -greedy exploration, decreasing the strength of exploration as the curriculum proceeds.

Original: As can be seen in Figures 2 and 3, regular learning on the target task does not allow ﬁnd a good solution, regardless of whether boosting is used or not ( B - FQI and FQI ).
Revised: As can be seen in Figures 2 and 3, regular learning on the target task does not allow ﬁnd a good solution, regardless of whether boosting is used or not ( B - FQI and FQI ).

Original: This is due to the highly challenging exploration problem that needs to be solved in this environment.
Revised: This is due to the highly challenging exploration problem that needs to be solved in this environment.

Original: The initial random exploration is insufﬁcient to maneuver around the second wall in the target task.
Revised: The initial random exploration is insufﬁcient to maneuver around the second wall in the target task.

Original: Consequently, the agent is not aware of the beneﬁt of this detour, as obvious in the learned Q -function (Figure 3b).
Revised: Consequently, the agent is not aware of the beneﬁt of this detour, as obvious in the learned Q -function (Figure 3b).

Original: Subsequently reducing the exploration does not lead to any new evidence and hence the agent quickly converges to the sub-optimal solution (Figure 2).
Revised: Subsequently reducing the exploration does not lead to any new evidence and hence the agent quickly converges to the sub-optimal solution (Figure 2).

Original: Looking at the results of C - FQI , we see that the curriculum alleviates the aforementioned exploration problem.
Revised: Looking at the results of C - FQI , we see that the curriculum alleviates the aforementioned exploration problem.

Original: However, the estimated Q -function deteriorates before approaching the target task, leading to unreliable policies.
Revised: However, the estimated Q -function deteriorates before approaching the target task, leading to unreliable policies.

Original: By introducing additional capacity as learning proceeds, BC - FQI allows to learn an optimal policy.
Revised: By introducing additional capacity as learning proceeds, BC - FQI allows to learn an optimal policy.

Original: Indeed, we show in the appendix that the residuals of BC - FQI decrease in later iterations, easing the function approximation problem when progressing to the target task. 
Revised: Indeed, we show in the appendix that the residuals of BC - FQI decrease in later iterations, easing the function approximation problem when progressing to the target task. 6. L INEAR SYSTEM CONTROL

Revised: The last task aims at highlighting a more subtle beneﬁt of BCRL :

Revised: Choosing the architecture of the function approximators appropriately for each task in the curriculum.

Revised: We show that this ﬂexibility can improve sample efﬁciency in a control task, in which a point mass needs to be steered towards

Original: [Figure]
Revised: [Figure]

Original: 4 Steps × 10− 1 .− 1 .− 1 .− 1 .
Revised: 4 Steps × 10− 1 .− 1 .− 1 .− 1 .

Original: C u m .
Revised: C u m .

Original: D i s c .
Revised: D i s c .

Original: R e t u r n × [-
-] BC-DQN C-DQN DQN  −−−−−0 ×10(a)
Revised: R e t u r n × [-
-]10 2 BC-DQN C-DQN DQN RPL PPR −−−−−0 ×

Revised: 10(a)

Original: Agent Performance −  −−
Revised: Agent performance − 5 −−

Original: LSPI −
Revised: LSPI −

Original: DQN[Figure] L INEAR SYSTEM CONTROL The last task aims at highlighting a more subtle beneﬁt of BCRL :
Revised: DQN a goal position over a narrow pathway along which it is subject to a drift moving it away from the target position (Figure 4b).

Original: Choosing the architecture of  the function approximators appropriately for each task in the curriculum. We show that this ﬂexibility can improve sample efﬁciency in a control task, in which  a point mass needs to be steered towards a goal position over a narrow pathway (Figure 4b). Along this pathway, the point mass is subject to a drift moving it away from the target position.
Revised: The agent receives a reward of − 1 until reaching the goal, which issues a reward of 0 and ends the episode.

Original: If the agent falls off the pathway, it is reset to the initial position.
Revised: If the agent falls off the pathway, it is reset to the initial position.

Original: The agent receives a reward of − 1 until reaching the goal.

Original: Upon reaching the goal, a reward of 0 is achieved and the episode terminates.

Original: We learn a policy for solving this target task using DQN . We introduce a curriculum for learning this task by ﬁrst training in an LQR system to induce a bias of moving towards the target.
Revised: We learn a policy for solving this target task using DQN and introduce a curriculum that ﬁrst trains in an easier system to induce a bias of moving towards the target.

Original: In this LQR system, the point massbehaves according to the same dynamics as in the target environment and needs to reach the same target position, but does not faceany external perturbations, obtains a reward equal to the negative squared distance to the target position and does not need to stay on a particular pathway.
Revised: In this easier system, the point mass does not face external perturbations, obtains a reward equal to the negative squared distance to the target position and does not need to stay on a particular pathway.

Original: A beneﬁt of environments with the aforementioned characteristics is that the Q -function can be shown to be a quadratic function ofaction and state  (Bradtke, 1993).
Revised: Since the easier environment is a linear quadratic (LQ) problem (see appendix C.3), the Q -function is known to be a quadratic function of state and action (Bradtke, 1993).

Original: Since the boosting framework does not prescribe the architecture of the individual employed function approximators, we can exploit the aforementioned structure of the ﬁrst(LQR) learning task by employing least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003) with quadratic features.
Revised: The boosting framework allows to exploit this knowledge by using quadratic features in the ﬁrst learning task and only rely on a neural network to realize the Q -function in the target task. With quadratic features, the Q -function can be efﬁciently learned from experience with least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003). In the boosted curriculum evaluated in this task ( BC - DQN ), we hence ﬁrst learn a Q -function using LSPI in the LQ task and then adjust this Q -function in the target task with a residual using DQN .

Original: Figure 4a shows that this incorporation of additional structure into the curriculum leads to improved performance over both directly learning on the target task ( DQN ) and not adjusting the structure of the agent to the learning task in the curriculum ( C - DQN ).
Revised: Figure 4a shows that this incorporation of additional structure into the curriculum leads to improved performance over both directly learning on the target task ( DQN ) and not adjusting the structure of the agent to the learning task in the curriculum ( C - DQN ).

Original: It is particularly surprising that C - DQN is performing even worse than DQN .
Revised: It is particularly surprising that C - DQN is performing even worse than DQN .

Original: There are multiple potential causes for this poor performance.
Revised: There are multiple potential causes for this poor performance.

Original: One is that the DQN algorithm takes signiﬁcantly longer to learn in the LQR task compared to LSPI ( 10 . 000 vs. 2 . 000 steps).
Revised: One is that the DQN algorithm takes signiﬁcantly longer to learn in the LQ problem compared to LSPI ( 10 . 000 vs. 2 . 000 steps), diminishing the beneﬁt of bootstrapping the policy in the easier learning task.

Original: Further, the learned Q -functionand policy is not as accurate as the one learned  with  LSPI , as can be seen in Figure 4b.
Revised: We further see no accelerated learning of C - DQN on the target task compared to DQN . This may be caused by the less accurate Q -function learned in the LQ problem with DQN compared to LSPI (Figure 4b), which does not perform as targeted movements to the goal. We also evaluate residual policy learning ( RPL , Silver et al.

Original: Consequently, the policy is not as stable.
Revised: and probabilistic policy reuse ( PPR , Fern ´ andez et al.

Original: Finally, due to the different reward structures of the two learning tasks, the Q -function needs to adjust during learning on the target task, potentially unlearning the stabilizing behavior induced by the initial task.
Revised: leveraging the policy learned in the LQ problem with LSPI (details in C.3). We see that

Original: BC - DQN avoids this risk by not needing to adjust the Q -function learned in the previous task.
Revised: RPL increases learning speed compared to C - DQN , but both RPL and PPR do not improve upon DQN .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced boosted curriculum reinforcement learning ( BCRL ), a new method for action-value function estimation in curriculum RL.
Revised: We introduced boosted curriculum reinforcement learning ( BCRL ), a new method for action-value function estimation in curriculum RL.

Original: For each task, BCRL approximates the respective action-value function as a sum of residuals of the previous tasks in the curriculum. This procedure increases the representativeness of the functional space as the complexity of the tasks in the curriculum increases, instead of using a single approximator through all the learning.
Revised: For each task, BCRL approximates the respective action-value function as a sum of residuals of the previous tasks in the curriculum, increasing the representativeness of the functional space alongside the complexity of the tasks.

Original: We proved thatour BCRLmethod enjoys advantageous theoretical properties in terms of convergence to the optimal actionvalue function of the target task.
Revised: We proved that BCRL enjoys advantageous theoretical properties in terms of convergence to the optimal action-value function of the target task.

Original: Moreover, we empirically validated BCRL across different problems and algorithms, showing itsoverall advantages over regular curriculum RL.
Revised: Moreover, we empirically validated BCRL across different problems and algorithms, showing its advantages over regular curriculum RL.

Revised: Our work is a step towards understanding the complex interplay between curricula and the structure of the learning agent.

Revised: We assume that the curricula are so-called sequence curricula, while recent progress in automated curriculum generation deﬁnes curricula as a distribution over tasks (Narvekar et al., 2020).

Revised: Adapting our theoretical framework to these recent advances would allow to couple them with BCRL .

Original: A P ROOFS
Revised: Further, the complexity of the network, and hence the required computation, increases with each task. Finding ways to decay the inﬂuence of older Q -functions may allow for pruning, avoiding linear growth of the network.  R EPRODUCIBILITY S TATEMENT We aim to guarantee the reproducibility of our work by providing the proofs of the main paper theorems as well as additional details of the conducted experiments in the appendix.

Revised: Further, we provide the code for running the experiments alongside the submission and will make this code publicly available upon acceptance

Revised: A D ISCUSSION OF RELATED T RANSFER L EARNING M ETHODS

Revised: Given our focus on knowledge transfer via Q -functions in curriculum RL, our investigations are closely related to the ﬁeld of transfer learning and we hence want to situate our method in this ﬁeld by this short review.

Revised: Transfer learning is a well-established concept in reinforcement learning and multiple surveys have been conducted in the single- and multi-agent RL setting (Taylor & Stone, 2009; Lazaric, 2012; Da Silva et al., 2018; Da Silva & Costa, 2019).

Revised: Apart from detecting a change in the current task (Hernandez-Leal et al., 2017), determining from which task to transfer knowledge (Fern´andez & Veloso, 2006; Barrett & Stone, 2015; Hernandez-Leal & Kaisers, 2017) to the current task is an important problem in transfer learning.

Revised: For an adequately chosen curriculum, the aforementioned problems are less severe as a) the agent is assumed to be aware of the task change and b) the knowledge from the most recent task in the curriculum can be assumed to be the most relevant for solving the current one.

Revised: The transfer of knowledge between tasks can then happen in various forms, e.g., via samples (Lazaric et al., 2008; Lazaric & Restelli, 2011), policies (Fern´andez & Veloso, 2006; Fern´andez et al., 2010; Silver et al., 2018), reward functions (Brys et al., 2015) or demonstrations/teachers (Chernova & Veloso, 2009; Billard et al., 2016;

Revised: Koert et al., 2020).

Revised: When sharing value- or Q -functions, which resembles the boosting scenario investigated in this paper, incorporating abstract concepts like objects, classes or relations (Koga et al., 2013; 2014; Da Silva &

Revised: Costa, 2017; Silva & Costa, 2018) can drastically improve the generalization of learned Q - or valuefunctions between tasks at the expense of additional user-speciﬁed information.

Revised: When not assuming any additional information of the aforementioned forms in a value- or Q -function transfer setting, methods that extend upon the naive idea of initializing the Q -function of the next task with the one learned in the previous one, as e.g. done in (Narvekar et al., 2016), are rather sparse.

Revised: Rusu et al.

Revised: (2016) present a particular neural network architecture that aims to improve transfer between tasks by adding so called columns for each encountered task.

Revised: Those additional columns are connected to the columns of the previous task by learnable weight matrices.

Revised: Boutsioukis et al.

Revised: (2011) investigate the idea of sparsely initializing the Q -function in the new task with a bias value for the optimal actions of the previous task in discrete state-action spaces.

Revised: We compare to residual policy learning (RPL, Silver et al.

Revised: in Section 6.3, as it follows a similar idea of learning a residual, however in the form of a policy, and furthermore avoids the complication of the overall curriculum by e.g. introducing a secondary RL objective (Brys et al., 2015) to realize the knowledge transfer between tasks.

Revised: For the environment in Section 6.3, the idea of a residual policy is well-suited since thediscrete actions in the target task (see appendix C.3) allows to deﬁne reasonable ways to combine the actions of the previous and the current policy.

Revised: For the tasks in sections 6.1 and 6.2, combining actions is problematic due to the action spaces of the tasks.

Revised: For the car-on-hill task, combining the two opposing actions of moving left or right will either always ignore the actions of the previous policy or ignore the actions of the current policy.

Revised: The same problem occurs for the maze task with its four actions left , right , up and down .

Revised: B P ROOFS

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q it )
Revised: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =
Revised: =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.
Revised: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.

Original: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S ) 
Revised: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S ) y (cid:107) µ ≤ λ (cid:107) y (cid:107) µ for all y ∈ B ( S × A ) . Then, (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (1 + L

Original: [Equation]
Revised: [Equation]

Original: λ (cid:107)y (cid:107) µ for all y ∈ B ( S × A ) . Then,
Revised: L [Equation] (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (13) considers the assumption on the operator ( I − S ) and (14) follows noting that
Revised: where (13) considers the assumption on the operator ( I − S ) and (14) follows noting that

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.

Original: Denoting the regular AVI residual as ¯ (cid:37) kt
Revised: Denoting the regular AVI residual as ¯ (cid:37) kt

Original: = ( T ∗t ) k . . .
Revised: = ( T ∗

Original: ( T ∗ 1 ) K 1 Q 01 −

Original: [Equation]
Revised: [Equation]

Original: T ∗ t − 1 ) K t − 1 . . . 
Revised: k . . . ( T ∗ [Equation] [Equation]

Original: ( T ∗ 1 ) K 1 Q01 , we have
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: ¯ (cid:37) [Equation]
Revised: −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: 1 ) K t

Original: [Equation]
Revised: [Equation]

Original: ε
Revised: ( T ∗ 1 ) K 1 Q 01 , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:107) µ (cid:17) .

Original: Proof.
Revised: Proof.

Original: For an arbitrary function f ∈ F and task t , we have (cid:107) f − (cid:37) kt
Revised: For an arbitrary function f ∈ F and task t , we have (cid:107) f − (cid:37) kt

Original: (cid:107)µ ≤ (cid:107) f
Revised: (cid:107)

Original: − ¯ (cid:37) kt (cid:107) µ + (cid:107) ¯ (cid:37) kt − (cid:37) kt (cid:107) µ . Then, (cid:13)(cid:13)(cid:13) ¯ (cid:37) kt − (cid:37) kt (cid:13)(cid:13)(cid:13) µ ≤ (cid:13)(cid:13)(cid:13) ( T ∗ t ) k . . . ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t ) k Q 0 t (cid:13)(cid:13)(cid:13) µ + (cid:13)(cid:13)(cid:13) ( T ∗ t − 1 ) K t − 1 . . . ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ ≤ L k (cid:13) (cid:13)(cid:13)
Revised: [Equation]

Original: ( T ∗ t − 1 ) K t − 1 . . . ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ + (cid:13) (cid:13)(cid:13) ( T ∗ t − 1 ) K t − 1 . . . ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ = (
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: +
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)(cid:13) ( T

Original: [Equation]
Revised: [Table]

Original: − 1 ) k t − 1 . . . ( T ∗ 1 ) K 1 Q 01 − Q 0 t
Revised: Then,

Original: [Table]
Revised: [Table]

Original: Theorem 4.
Revised: Theorem 4.

Original: Let B tk = max( (cid:107) ˜ (cid:37)
Revised: Let B tk = max( (cid:107) ˜ (cid:37)

Original: kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).
Revised: kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).

Original: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni = for each task t , and a sequence of action-value functions ( Q it ) ki =0 generated by our BCRL, we have 
Revised: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni = for each task t , and a sequence of action-value functions ( Q i t ) k i =0 generated by our BCRL, we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Theorem 5.11 from Gy¨orﬁ et al.
Revised: Using Theorem 11.5 from Gy¨orﬁ et al.

Original: (2006), we obtain
Revised: (2006), we obtain

Original: [Equation]
Revised: [Equation]

Original: We complete the proof using (15) and the Cauchy-Schwartz inequality to bound the ﬁrst term.
Revised: We complete the proof using (15) and the Cauchy-Schwartz inequality to bound the ﬁrst term.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation] 
Revised: [Equation] [Figure]

Revised: Iteration −−−

Revised: P a % t ( s 0 , a ) # A ... T 5 T

Revised: T

Revised: T

Revised: T

Revised: T

Revised: BC-FQI

Original: Proof.
Revised: Proof.

Original: First, we bound the k -step Bellman residual.
Revised: First, we bound the k -step Bellman residual.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: [Equation]

Revised: ) − ε k − 1 t ( s (cid:48) , a (cid:48) ) (cid:1) − Q 0 t ( s, a ) (cid:12)(cid:12)(cid:12) ≤ γ

Revised: [Equation]

Revised: ε

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: a )

Revised: [Equation]

Revised: max

Revised: [Equation]

Revised: (cid:48)

Revised: [Equation]

Revised: [Equation]

Revised: s (cid:48)

Revised: [Equation]

Revised: (cid:48) )

Revised: [Equation]

Revised: max

Revised: [Equation]

Revised: (cid:48)(cid:48) ∈A Q 0 t ( s (cid:48) , a (cid:48)(cid:48) )

Revised: (cid:19) (cid:12)(cid:12)(cid:12) ≤ γ

Revised: [Equation]

Revised: ε k

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: (cid:90)

Revised: [Equation]

Revised: (cid:48)

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: a ) (cid:18) max a (cid:48)

Revised: [Equation]

Revised: (cid:12)(cid:12) (

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ( s (cid:48) ,

Revised: [Equation]

Revised: (cid:48) )

Revised: [Equation]

Revised: Q 0 t ( s (cid:48) , a (cid:48) )

Revised: (cid:12)(cid:12)(cid:19) (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) (cid:37) 1 t (cid:107) ∞
Revised: Since (cid:107) (cid:37) 1 t (cid:107) ∞

Original: ≤ R max t , solving the recursion proves the lemma.
Revised: ≤ R max t , solving the recursion proves the lemma.

Original: We are now able to derive the bound for the k -step empirical Bellman residual.
Revised: We are now able to derive the bound for the k -step empirical Bellman residual.

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: Iteration −−− P a % t ( s 0 , a ) # A ... T 5 T T T
Revised: The bound is proved using (18)

Original: T T BC-FQI
Revised: [Equation]

Original: B E XPERIMENTAL DETAILS
Revised: C E XPERIMENTAL DETAILS

Original: This section serves to present additional details on the experiments that did not ﬁt in the main paper, such as algorithm hyperparameters and additional details of the learning environments.
Revised: This section serves to present additional details on the experiments that did not ﬁt in the main paper, such as algorithm hyperparameters and additional details of the learning environments.

Original: These additional details will be provided for each experiment individually in the following subsections.
Revised: These additional details will be provided for each experiment individually in the following subsections.

Original: The computations were executed on a desktop computer with a Geforce RTX 2080, 64 GB memory and an AMD Ryzen 9 3900X processor.
Revised: The computations were executed on a desktop computer with a Geforce RTX 2080, 64 GB memory and an AMD Ryzen 9 3900X processor.

Original: All experiments were run in roughly a week of computation time on this single machine.
Revised: All experiments were run in roughly a week of computation time on this single machine.

Original: The employed MushroomRL library is available under the MIT license.
Revised: The employed MushroomRL library is available under the MIT license.

Original: B. C AR - ON - HILL
Revised: C. C AR - ON - HILL

Original: As mentioned in the main text, we used the FQI algorithm with extra-trees (Ernst et al., 2005) implemented in MushroomRL (D’Eramo et al., 2021) to conduct this experiment.
Revised: As mentioned in the main text, we used the FQI algorithm with extra-trees (Ernst et al., 2005) implemented in MushroomRL (D’Eramo et al., 2021) to conduct this experiment.

Original: The number of trees in the extra-trees implementation was set to 50 , the minimum number of samples required to split an internal node to 5 and the minimum number of samples required to be a leaf node to 2 .
Revised: The number of trees in the extra-trees implementation was set to 50 , the minimum number of samples required to split an internal node to 5 and the minimum number of samples required to be a leaf node to 2 .

Original: All other parameters were left to the implementation defaults of MushroomRL.
Revised: All other parameters were left to the implementation defaults of MushroomRL.

Original: Based on this implementation we created a boosted version that runs the original FQI algorithm on the residuals of the previously learned models.
Revised: Based on this implementation we created a boosted version that runs the original FQI algorithm on the residuals of the previously learned models.

Original: The hyperparameters for those extra-trees models were the same as for the regular FQI .
Revised: The hyperparameters for those extra-trees models were the same as for the regular FQI .

Original: The implementation can be found in the accompanying code.
Revised: The implementation can be found in the accompanying code.

Original: The car-on-hill environment used for this experiment is also implemented in MushroomRL.
Revised: The car-on-hill environment used for this experiment is also implemented in MushroomRL.

Original: B. M AZE
Revised: C. M AZE

Original: The state space of the maze environments consists of the two-dimensional coordinates s =[ s x , s y ] ∈ [0 , 1] 2 .
Revised: The state space of the maze environments consists of the two-dimensional coordinates s =[ s x , s y ] ∈ [0 , 1] 2 .

Original: The initial state is s 0 =[0 , 0] , and the goal state is g 0 =[0 .
Revised: The initial state is s 0 =[0 , 0] , and the goal state is g 0 =[0 .

Original: The four actions displace the agent in x - or y -position by a value of 0 .
Revised: The four actions displace the agent in x - or y -position by a value of 0 .

Original: The discount factor is set to γ =0 .
Revised: The discount factor is set to γ =0 .

Original: 99 and the horizon length is 200 .
Revised: 99 and the horizon length is 200 .

Original: We use a dense reward function R ( · , · , s (cid:48) )
Revised: We use a dense reward function R ( · , · , s (cid:48) )

Original: =
Revised: =

Original: −| g − s (cid:48) | when | g − s (cid:48)
Revised: −| g − s (cid:48) | when | g − s (cid:48)

Original: | > = 0 .
Revised: | > = 0 .

Original: 1 , otherwise R ( · , · , s (cid:48) )
Revised: 1 , otherwise R ( · , · , s (cid:48) )

Original: =
Revised: =

Original: +5 and the episode terminates.
Revised: +5 and the episode terminates.

Original: There are two walls centered at x = 0 .
Revised: There are two walls centered at x = 0 .

Original: 2 and x = 0 .
Revised: 2 and x = 0 .

Original: 6 , both with width w =0 .
Revised: 6 , both with width w =0 .

Original: 05 , and height h t
Revised: 05 , and height h t

Original: depending on the task t .
Revised: depending on the task t .

Original: We design a curriculum composed of 10 tasks, with h 1 ,..., 9 = { 0 . 05 , 0 . 15 , . . .
Revised: We design a curriculum composed of 10 tasks, with h 1 ,..., 9 = { 0 . 05 , 0 . 15 , . . .

Original: , 0 . 85 } and h 10 =0 .
Revised: , 0 . 85 } and h 10 =0 .

Original: Hence in the ﬁnal task T 10 , there is only a very narrow passage through which the walls can be passed.
Revised: Hence in the ﬁnal task T 10 , there is only a very narrow passage through which the walls can be passed.

Original: We again make use of the FQI algorithm with extra-trees implemented in MushroomRL to conduct this experiment, again using 50 trees, a minimum number of 5 samples to split an internal node, a minimum number of 2 samples to be a leaf node and everything else left to the implementation defaults of MushroomRL.
Revised: We again make use of the FQI algorithm with extra-trees implemented in MushroomRL to conduct this experiment, again using 50 trees, a minimum number of 5 samples to split an internal node, a minimum number of 2 samples to be a leaf node and everything else left to the implementation defaults of MushroomRL.

Original: The main conceptual different to the car-on-hill experiment is that exploration is guided by the currently estimated Q -function, as we are employing an εgreedy exploration strategy to generate the samples for learning the Q -functions of the learning tasks.
Revised: The main conceptual different to the car-on-hill experiment is that exploration is guided by the currently estimated Q -function, as we are employing an εgreedy exploration strategy to generate the samples for learning the Q -functions of the learning tasks.

Original: The exploration percentage ε is annealed in later environments.
Revised: The exploration percentage ε is annealed in later environments.

Original: More precisely, we set ε 1 ,..., 10 = { 1 ., 1 ., 1 ., 1 ., .
Revised: More precisely, we set ε 1 ,..., 10 = { 1 ., 1 ., 1 ., 1 ., .

Original: 25 } for the tasks T 1 , . .
Revised: 25 } for the tasks T 1 , . .

Original: , T 10 .
Revised: , T 10 .

Original: Figure 5 visualizes the residuals of the starting state s 0 that the Q -functions learned in the individual tasks, i.e., the additional values the Q -functions needed to predict in addition to the Q -value predicted from the ensemble of previously learned Q -functions.
Revised: Figure 5 visualizes the residuals of the starting state s 0 that the Q -functions learned in the individual tasks, i.e., the additional values the Q -functions needed to predict in addition to the Q -value predicted from the ensemble of previously learned Q -functions.

Original: We see that the magnitude of residuals continuously decreases from task T 1 to task T 6 .
Revised: We see that the magnitude of residuals continuously decreases from task T 1 to task T 6 .

Original: Switching to task T 7 increases the magnitude of the residuals after which it then continuously declines again.
Revised: Switching to task T 7 increases the magnitude of the residuals after which it then continuously declines again.

Original: The reason for this jump is that, while the optimal policy is largely unaffected by the presence of the small walls in the early learning tasks, a more signiﬁcant detour is required starting from task T 7 .
Revised: The reason for this jump is that, while the optimal policy is largely unaffected by the presence of the small walls in the early learning tasks, a more signiﬁcant detour is required starting from task T 7 .

Original: B. L INEAR SYSTEM CONTROL
Revised: C. L INEAR SYSTEM CONTROL

Original: The linear system that needs to be controlled is given by
Revised: The linear system that needs to be controlled is given by

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: 0 .

Original:    x

Original: [Equation]

Original: y 

Original: [Equation]

Original:   0100  (cid:20) a x a y (cid:21) =

Original: As

Original: [Equation]
Revised: [Equation]

Original: Ba , representing a simple, fully-actuated point-mass with a slight amount of friction.
Revised: representing a simple, fully-actuated point-mass with a slight amount of friction.

Original: For the LQR task, to which the curriculum learning agents are exposed ﬁrst, the reward function is given by[Equation] ( s , a , s [Equation]
Revised: For the linear quadratic (LQ) task, to which the curriculum learning agents are exposed ﬁrst, the reward function is given by

Original: [Equation]

Original:  3 0 00 1 00 0 30 0 0

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]

Original: (cid:20) 0 .

Original: 0 .(cid:21)

Original: a and the initial state is given by s 0 =
Revised: and the initial state is given by s 0 =

Original: [ − 4 v x, 0 0 v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .
Revised: [ − 4 v x, 0 0 v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .

Original: In the target task, the agent receives the following reward signal
Revised: In the target task, the agent receives the following reward signal

Original: [Equation]
Revised: [Equation]

Original: 0 , else and its initial position is given by s 0 =
Revised: 0 , else and its initial position is given by s 0 =

Original: [ − 4 v x, 0 y v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .
Revised: [ − 4 v x, 0 y v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .

Original: 1 , 0 . 1] and y within [ − 0 . 4 , 0 . 4] .
Revised: 1 , 0 . 1] and y within [ − 0 . 4 , 0 . 4] .

Original: Furthermore, the agent needs to keep its y -position within [ − 0 . 5 , 0 . 5] in order to stay on the pathway indicated in Figure 4b and it faces an additional drift moving it away from the target, particularly[Equation]
Revised: Furthermore, the agent needs to keep its y -position within [ − 0 . 5 , 0 . 5] in order to stay on the pathway indicated in Figure 4b and it faces an additional drift moving it away from the target, particularly

Original: [Equation]
Revised: [Equation]

Original: Consequently, the policies pre-trained in the LQR do not quite reach the target but need to explore that they need to counteract this external force while staying on the pathway.
Revised: Consequently, the policies pre-trained in the LQ task do not quite reach the target but need to explore that they need to counteract this external force while staying on the pathway.

Original: When learning with LSPI in the LQR system, we represent the policy by a PD-controller with explorative noise 
Revised: To learn with LSPI in the LQ task, we represent the policy by a PD-controller with explorative noise a ∼ N ( Ks , 0 . 1 I ) as this ensures the quadratic form of the optimal Q -function mentioned in the main text. More precisely, it can be shown that

Original: [Equation]
Revised: [Equation]

Original: as this ensures the quadratic form of the optimal Q -function mentioned in the main text (please see Bradtke (1993) for additional details).
Revised: where φ ( s , a ) computes linear and quadratic terms of s and a (please see Bradtke (1993) for additional details).

Original: When learning with DQN in both the LQR and target task, the agent can choose between applying 8 discrete actions
Revised: When learning with DQN in both the LQ and target task, the agent can choose between applying 8 discrete actions

Original: [Equation]
Revised: [Equation]

Revised: , a 5 = (cid:20) ..

Revised: (cid:21) , a 6 = (cid:20) .− .(cid:21) , a 7 = (cid:20) − ..(cid:21) , a 8 = (cid:20) − .− .(cid:21) .

Original: Consequently, we predict these discrete Q -values with the function approximator learned in the LQR task with LSPI when training on the target task.
Revised: Consequently, we predict these discrete Q -values with the function approximator learned in the LQ task with LSPI when training on the target task.

Original: The Q -networks employed in DQN consist of two hidden layers of 128 neurons and ReLU activations.
Revised: The Q -networks employed in DQN consist of two hidden layers of 128 neurons and ReLU activations.

Original: During training, we realized that the update frequency of the target network in the DQN algorithm had a strong impact on the ﬁnal agent performance.
Revised: During training, we realized that the update frequency of the target network in the DQN algorithm had a strong impact on the ﬁnal agent performance.

Original: Hence, we conducted a grid search testing values in [25 , 50 , 100 , 200 , 500 , 1000 , 2000] .
Revised: Hence, we conducted a grid search testing values in [25 , 50 , 100 , 200 , 500 , 1000 , 2000] .

Original: For regular DQN on the target task, an update frequency of 200 performed best, while BC - DQN yielded best results with an update frequency of 2000 .
Revised: For regular DQN on the target task, an update frequency of 200 performed best, while BC - DQN yielded best results with an update frequency of 2000 .

Original: For C - DQN , best results where achieved when using an update frequency of 1000 in the LQR task and an update frequency of 25 in the target task.
Revised: For C - DQN , best results where achieved when using an update frequency of 1000 in the LQ task and an update frequency of 25 in the target task.

Original: We used Adam for optimization of the network with a learning rate of 10 − 4 and ( β 1 , β 2 ) =
Revised: We used Adam for optimization of the network with a learning rate of 10 − 4 and ( β 1 , β 2 ) =

Original: To exploit the previously learned policy in the target task for the curriculum methods, we used a ﬁxed value of ε = 0 .
Revised: To exploit the previously learned policy in the target task for the curriculum methods, we used a ﬁxed value of ε = 0 .

Original: 1 in the target task.
Revised: 1 in the target task.

Original: For default DQN , we annealed ε from 1 to 0 .
Revised: For default DQN , we annealed ε from 1 to 0 .

Original: 1 over the ﬁrstenvironment steps.
Revised: 1 over the ﬁrstenvironment steps.

Revised: As mentioned in the main paper, we additionally investigate residual policy learning ( RPL ) and probabilistic policy reuse ( PPR ) in this environment.

Revised: Both approaches exploit a policy π source learned in a source task (for our experiment this is the LQ task) to accelerate learning in the target task.

Revised: As the methods are agnostic to the structure of the source task policy, we make use of the policy learned with LSPI , as this policy exhibits a more structured movement towards the target (see Figure 4b) and further requires less environment interactions to be learned (see Figure 4a).

Revised: To implement RPL (Silver et al., 2018), we need to specify how to combine the actions a i and a j of the source task policy π source and the residual policy to be learned in the target task.

Revised: In our experiment, we can simply add the actions a i and a j and then select the discrete action that is more similar to the combined action in terms of euclidean distance.

Revised: If there are multiple actions that are within the same minimum distance to the combined action, we choose the action that is more similar to the action of the residual policy, i.e. more similar to a j .

Revised: This allows the residual policy to still move opposite to the previous policy if necessary, while preserving the bias induced by the previous policy if the residual policy does not produce an action exactly opposing the one from the previous policy.

Revised: RPL has an offset parameter that controls for how many steps the residual policy does not interfere with the source task policy.

Revised: For the described way of combining actions, not interfering means simply choosing the same actions as the source task policy.

Revised: We tried values of 0 , 2000 and 4000 steps for this offset value, where 2000 performed best.

Revised: PPR requires the user to choose a decay factor ν to anneal the likelihood of executing the source task policy via p ( π source ) =

Revised: ν t , where t is the number of past interactions (note that we follow the interpretation of PPR by Brys et al.

Revised: (2015) for transfer from a single source task).

Revised: As for the other baselines, we do a grid search over the target network update frequency, choosing the best.

Revised: For RPL , a value of 25 performed best.

Revised: For PPR , a value ofperformed slightly better than 25 or 50 .