Original: B OOSTED C URRICULUM R EINFORCEMENT L EARNING
Revised: B OOSTED C URRICULUM R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difﬁculty.
Revised: Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difﬁculty.

Original: However, ﬁnding an exact way of reusing action-values in this setting is still a poorly understood problem.
Revised: However, ﬁnding an exact way of reusing action-values in this setting is still a poorly understood problem.

Original: In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task.
Revised: In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task.

Original: This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the beneﬁt of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented.
Revised: This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the beneﬁt of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented.

Original: This procedure allows reusing previous action-values while promoting expressiveness of the action-value function.
Revised: This procedure allows reusing previous action-values while promoting expressiveness of the action-value function.

Original: We theoretically study BCRL as an approximate value iteration algorithm, proving its advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function.
Revised: We theoretically study BCRL as an approximate value iteration algorithm, proving its advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function.

Original: Finally, we provide detailed empirical evidence of the beneﬁts of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.
Revised: Finally, we provide detailed empirical evidence of the beneﬁts of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The combination of reinforcement learning (RL) (Sutton & Barto, 2018) algorithms with powerful function approximators, i.e., deep RL (Franc¸ois-Lavet et al., 2018; Mnih et al., 2015), is a major breakthrough towards solving complex decision making and control problems that were impractical for previous shallow RL methods.
Revised: The combination of reinforcement learning (RL) (Sutton & Barto, 2018) algorithms with powerful function approximators, i.e., deep RL (Franc¸ois-Lavet et al., 2018; Mnih et al., 2015), is a major breakthrough towards solving complex decision making and control problems that were impractical for previous shallow RL methods.

Original: However, the outstanding performance of deep RL techniques is obtained at the cost of a massive amount of data needed from the interaction with the environment, hindering the practicality of deep RL methods.
Revised: However, the outstanding performance of deep RL techniques is obtained at the cost of a massive amount of data needed from the interaction with the environment, hindering the practicality of deep RL methods.

Original: Curriculum RL is a biologically inspired approach that frames the problem of learning a complex target task into learning a sequence of simpliﬁed, increasingly difﬁcult, versions of it (Florensa et al., 2017;
Revised: Curriculum RL is a biologically inspired approach that frames the problem of learning a complex target task into learning a sequence of simpliﬁed, increasingly difﬁcult, versions of it (Florensa et al., 2017;

Original: Shao et al., 2018; Ivanovic et al., 2019;
Revised: Shao et al., 2018; Ivanovic et al., 2019;

Original: Narvekar et al., 2020).
Revised: Narvekar et al., 2020).

Original: Commonly, establishing the complexity of a task in RL is not trivial, as an indisputable notion of complexity is still missing.
Revised: Commonly, establishing the complexity of a task in RL is not trivial, as an indisputable notion of complexity is still missing.

Original: However, from a practical point of view, we can consider a task complex if it requires targeted exploration in the environment or complex policies to be solved.
Revised: However, from a practical point of view, we can consider a task complex if it requires targeted exploration in the environment or complex policies to be solved.

Original: In curriculum RL, the notion of task complexity is key, and it is assumed that tailored tasks in a sequence of increasing complexity are presented to the learning agent.
Revised: In curriculum RL, the notion of task complexity is key, and it is assumed that tailored tasks in a sequence of increasing complexity are presented to the learning agent.

Original: Such an appropriate design is, however, a difﬁcult problem, requiring either human expertise (Narvekar et al., 2020) or solutions to automatize the selection of tasks following the learning of the agent (Jiang et al., 2015; Svetlik et al., 2017; Klink et al., 2020).
Revised: Such an appropriate design is, however, a difﬁcult problem, requiring either human expertise (Narvekar et al., 2020) or solutions to automatize the selection of tasks following the learning of the agent (Jiang et al., 2015; Svetlik et al., 2017; Klink et al., 2020).

Original: Regardless of using a handcrafted or an automatically generated curriculum, a common approach adopted by most value-based curriculum RL methods is to use the same function approximator throughout all the tasks (Narvekar et al., 2020): given a task, an approximation of the action-value function is learned, and used as an initialization for learning the action-value function of the next task in the curriculum.
Revised: Regardless of using a handcrafted or an automatically generated curriculum, a common approach adopted by most value-based curriculum RL methods is to use the same function approximator throughout all the tasks (Narvekar et al., 2020): given a task, an approximation of the action-value function is learned, and used as an initialization for learning the action-value function of the next task in the curriculum.

Original: For the success of this procedure, the function approximator needs to be powerful enough to handle the complexity of the target task.
Revised: For the success of this procedure, the function approximator needs to be powerful enough to handle the complexity of the target task.

Original: However, this requirement raises the key issue of adequately designing the function approximator.
Revised: However, this requirement raises the key issue of adequately designing the function approximator.

Original: A complex function approximator is prone to overﬁtting, and sensitive to hyper-parameters, being detrimental for learning simple tasks in the curriculum; conversely, using an overly simple function approximator can hinder the learning of the target task.
Revised: A complex function approximator is prone to overﬁtting, and sensitive to hyper-parameters, being detrimental for learning simple tasks in the curriculum; conversely, using an overly simple function approximator can hinder the learning of the target task.

Original: Moreover, choosing an adequate function approximator a-priori is not trivial, as the target task is only visited at the end, and it is commonly unsolvable by non-curriculum RL methods.
Revised: Moreover, choosing an adequate function approximator a-priori is not trivial, as the target task is only visited at the end, and it is commonly unsolvable by non-curriculum RL methods.

Original: In this paper, we introduce a novel approach that models the action-value function of a task as a sum of residuals learned on the previous tasks in the curriculum.
Revised: In this paper, we introduce a novel approach that models the action-value function of a task as a sum of residuals learned on the previous tasks in the curriculum.

Original: Our method increases the representativeness of the function approximator as new tasks are presented to the agent, leading to a procedure that increases the size of the functional space as the curriculum proceeds.
Revised: Our method increases the representativeness of the function approximator as new tasks are presented to the agent, leading to a procedure that increases the size of the functional space as the curriculum proceeds.

Original: This results in a learning procedure that increases the power of the function approximator according to the complexity of the task at hand, using small models in simple problems to avoid overﬁtting, and using large models (resulting from the sum of small ones) to handle the dimensionality of complex problems.
Revised: This results in a learning procedure that increases the power of the function approximator according to the complexity of the task at hand, using small models in simple problems to avoid overﬁtting, and using large models (resulting from the sum of small ones) to handle the dimensionality of complex problems.

Original: Notably, if the curriculum is sufﬁciently ﬁne-grained, the residuals become smoother functions, close to zero, that are easy to learn, even if the respective tasks are harder than the previous ones.
Revised: Notably, if the curriculum is sufﬁciently ﬁne-grained, the residuals become smoother functions, close to zero, that are easy to learn, even if the respective tasks are harder than the previous ones.

Original: We call our method boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting technique in supervised learning (Freund, 1995).
Revised: We call our method boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting technique in supervised learning (Freund, 1995).

Original: We provide a theoretical analysis of our approach under the lens of the approximate value iteration (AVI) framework (Farahmand, 2011).
Revised: We provide a theoretical analysis of our approach under the lens of the approximate value iteration (AVI) framework (Farahmand, 2011).

Original: In AVI, an estimate of the optimal action-value function is obtained as an iterative process that starts from an arbitrary estimate and applies the optimal Bellman operator until convergence.
Revised: In AVI, an estimate of the optimal action-value function is obtained as an iterative process that starts from an arbitrary estimate and applies the optimal Bellman operator until convergence.

Original: It is shown that two sources of error exist in AVI: (i) computation of the optimal Bellman operator, (ii) representation of the action-value function (Farahmand, 2011).
Revised: It is shown that two sources of error exist in AVI: (i) computation of the optimal Bellman operator, (ii) representation of the action-value function (Farahmand, 2011).

Original: While the former is due to the need of using samples to approximate the unknown optimal Bellman operator, the latter depends on the representativeness of the functional space chosen to approximate the action-value function.
Revised: While the former is due to the need of using samples to approximate the unknown optimal Bellman operator, the latter depends on the representativeness of the functional space chosen to approximate the action-value function.

Original: In this work, we formalize the curriculum RL problem in the AVI setting to show that the representation of the action-value function obtained by BCRL results in a tighter bound on the approximation error compared to regular curriculum RL, consequently reducing the performance loss.
Revised: In this work, we formalize the curriculum RL problem in the AVI setting to show that the representation of the action-value function obtained by BCRL results in a tighter bound on the approximation error compared to regular curriculum RL, consequently reducing the performance loss.

Original: We complement this analysis with an empirical evaluation of BCRL in AVI and RL settings, resorting to the ﬁtted Q -iteration ( FQI ) algorithm (Ernst et al., 2005), least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003) and deep
Revised: We complement this analysis with an empirical evaluation of BCRL in AVI and RL settings, resorting to the ﬁtted Q -iteration ( FQI ) algorithm (Ernst et al., 2005), least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003) and deep

Original: Q -networks (Mnih et al., 2015), where we conﬁrm the advantages of BCRL over regular curriculum
Revised: Q -networks (Mnih et al., 2015), where we conﬁrm the advantages of BCRL over regular curriculum

Original: RL and a related baseline (Tosatto et al., 2017).
Revised: RL and a related baseline (Tosatto et al., 2017).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Learning under a dynamic set of training experience has been investigated in different ﬁelds throughout decades (Saul, 1941; Elman, 1993; Asada et al., 1996; Krueger & Dayan, 2009).
Revised: Learning under a dynamic set of training experience has been investigated in different ﬁelds throughout decades (Saul, 1941; Elman, 1993; Asada et al., 1996; Krueger & Dayan, 2009).

Original: A recent paper by Bengio et al.
Revised: A recent paper by Bengio et al.

Original: (2009) has established the term curriculum learning for the concept of organizing training data of a neural network in a beneﬁcial way for performance and generalization.
Revised: (2009) has established the term curriculum learning for the concept of organizing training data of a neural network in a beneﬁcial way for performance and generalization.

Original: This term carried over to the domain of RL, where curricula over training tasks have been shown to stabilize the highly challenging optimization problem of RL (Narvekar et al., 2020).
Revised: This term carried over to the domain of RL, where curricula over training tasks have been shown to stabilize the highly challenging optimization problem of RL (Narvekar et al., 2020).

Original: While a large body of research in the domain of curriculum reinforcement learning investigates the interesting problem of optimally selecting and scheduling the training tasks for an RL agent to optimize performance on a set of target tasks (Florensa et al., 2017; Ivanovic et al., 2019;
Revised: While a large body of research in the domain of curriculum reinforcement learning investigates the interesting problem of optimally selecting and scheduling the training tasks for an RL agent to optimize performance on a set of target tasks (Florensa et al., 2017; Ivanovic et al., 2019;

Original: Shao et al., 2018; Klink et al., 2020), the architecture of the learning agent is assumed ﬁxed and given.
Revised: Shao et al., 2018; Klink et al., 2020), the architecture of the learning agent is assumed ﬁxed and the transfer of agent behavior between subsequent tasks in the curriculum is realized by this shared ﬁxed architecture.

Original: However, aside from investigations into the beneﬁt of an adequately chosen evolution of training experience, investigations of an optimal learning architecture (Elsken et al., 2019) and the evolution of this architecture during learning (Aoki & Siekevitz, 1988;
Revised: However, aside from investigations into the beneﬁt of an adequately chosen evolution of training experience, investigations of an optimal learning architecture (Elsken et al., 2019) and the evolution of this architecture during learning (Aoki & Siekevitz, 1988;

Original: Rumelhart et al., 1988; Ash, 1989;Fahlman, 1990;
Revised: Rumelhart et al., 1988; Ash, 1989;

Original: Lahnaj¨arvi et al., 2002; Nitanda & Suzuki, 2018) have been conducted in supervised learning scenarios.
Revised: Fahlman, 1990; Lahnaj ¨ arvi et al., 2002; Nitanda & Suzuki, 2018) have been conducted in supervised learning scenarios.

Original: These works are particularly motivating from our perspective, as they not only indicate the beneﬁt of learning from an appropriate curriculum of experience, but also the advantages of changing the function approximator capacity as training progresses.

Original: However, the adaptation of these methods to the setting of RL has been limited to on-policy temporal-difference learning (Rivest & Precup, 2003; Vamplew & Ollington, 2005b;a) and it has not been further pursued in recent years.
Revised: The adaptation of these methods to the setting of RL has been limited to on-policy temporal-difference learning (Rivest & Precup, 2003; Vamplew & Ollington, 2005b;a) and it has not been further pursued in recent years.

Original: Contrary to these works, we grow the capacity of our function approximator during training via a boosting procedure (Freund, 1995).
Revised: Our work aims to improve the information transfer between subsequent tasks in curriculum value-based reinforcement learning via a boosting procedure (Freund, 1995).

Original: The beneﬁt of the boosting framework is that it is well understood from a theoretical perspective (Tosatto et al., 2017), allowing for the derivation of error bounds, as also presented in this paper.
Revised: The beneﬁt of the boosting framework is that it is well understood from a theoretical perspective (Tosatto et al., 2017), allowing for the derivation of error bounds, as also presented in this paper.

Original: Further, it is agnostic to any value-based RL algorithm, avoiding the need to adjust the RL algorithm of choice to be suited for the evolution of the function approximator (Rivest & Precup, 2003; Vamplew & Ollington, 2005a;b).
Revised: Further, it is agnostic to any value-based RL algorithm, avoiding the need to adjust the RL algorithm of choice to be suited for the evolution of the function approximator (Rivest & Precup, 2003; Vamplew & Ollington, 2005a;b).

Original:  P RELIMINARIES 3. D EFINITIONS AND NOTATION
Revised:  P RELIMINARIES 3. D EFINITIONS AND NOTATION

Original: We recall the notions about Markov Decision Processes (MDPs) (Puterman, 1990), and the notation used for approximate value iteration (AVI) (Farahmand, 2011), that are relevant to our work.
Revised: We recall the notions about Markov Decision Processes (MDPs) (Puterman, 1990), and the notation used for approximate value iteration (AVI) (Farahmand, 2011), that are relevant to our work.

Original: For a space Σ , we deﬁne M (Σ) the set of probability measures over the σ -algebra σ Σ .
Revised: For a space Σ , we deﬁne M (Σ) the set of probability measures over the σ -algebra σ Σ .

Original: We denote
Revised: We denote

Original: B (Σ , B ) as the space of bounded measurable functions w.r.t.
Revised: B (Σ , B ) as the space of bounded measurable functions w.r.t.

Original: σ Σ with bound 0 <B< ∞ .
Revised: σ Σ with bound 0 <B< ∞ .

Original: A ﬁniteaction discounted MDP is deﬁned as a tuple T
Revised: A ﬁniteaction discounted MDP is deﬁned as a tuple T

Original: = (cid:104)S , A , P , R , γ (cid:105) , where S is a measurable state space,
Revised: = (cid:104)S , A , P , R , γ (cid:105) , where S is a measurable state space,

Original: A is a ﬁnite set of actions, P : S ×
Revised: A is a ﬁnite set of actions, P : S ×

Original: A → M ( S ) is a transition kernel, R : S ×
Revised: A → M ( S ) is a transition kernel, R : S ×

Original: A × S → R is a reward function, and γ ∈ [0 , 1) is a discount factor.
Revised: A × S → R is a reward function, and γ ∈ [0 , 1) is a discount factor.

Original: A policy π is a mapping from the state space S to a probability distribution over the action space A .
Revised: A policy π is a mapping from the state space S to a probability distribution over the action space A .

Original: A policy induces an action-value function Q π ( s, a ) (cid:44) E [Σ ∞ t =

Revised: A policy induces an action-value function Q π ( s, a ) (cid:44) E [Σ ∞ t =

Original: γ t R t | S 

Revised: γ t R t | S

Original: = s,
Revised: = s,

Original: A 

Revised: A

Original: = a ] , corresponding to the expected cumulative discounted reward obtained performing action a in state s and following the policy π thereafter.
Revised: = a ] , corresponding to the expected cumulative discounted reward obtained performing action a in state s and following the policy π thereafter.

Original: RL aims at solving an MDP by ﬁnding an optimal policy π ∗ which induces an optimal action-value function
Revised: RL aims at solving an MDP by ﬁnding an optimal policy π ∗ which induces an optimal action-value function

Original: Q ∗ ( s, a ) =
Revised: Q ∗ ( s, a ) =

Original: sup π Q π ( s, a ) .
Revised: sup π Q π ( s, a ) .

Original: For immediate rewards uniformly bounded by R max , the action-value function is bounded by Q max
Revised: For immediate rewards uniformly bounded by R max , the action-value function is bounded by Q max

Original: = R max / 1 − γ .
Revised: = R max / 1 − γ .

Original: For a probability measure µ ∈ M ( S × A ) , we deﬁne the
Revised: For a probability measure µ ∈ M ( S × A ) , we deﬁne the

Original: L p ( µ ) -norm of a measurable function Q ∈ B ( S × A ) as (cid:107) Q (cid:107) p,µ (cid:44) [ (cid:82) S×A | Q ( s, a ) | p dµ ( s, a )] 1 / p .
Revised: L p ( µ ) -norm of a measurable function Q ∈ B ( S × A ) as (cid:107) Q (cid:107) p,µ (cid:44) [ (cid:82) S×A | Q ( s, a ) | p dµ ( s, a )] 1 / p .

Original: Given a sequence of values x 1: n =( x 1 , . . .
Revised: Given a sequence of values x 1: n =( x 1 , . . .

Original: , x n ) for some space X , the empirical L p -norm of a function f : X → R is (cid:107) f (cid:107) p,x 1: n (cid:44) 1 n (cid:80) ni
Revised: , x n ) for some space X , the empirical L p -norm of a function f : X → R is (cid:107) f (cid:107) p,x 1: n (cid:44) 1 n (cid:80) ni

Original: =1 | f ( x i ) | p .
Revised: =1 | f ( x i ) | p .

Original: Conveniently, when X i ∼ µ , we have E [ (cid:107) f (cid:107) p,x 1: n ] = (cid:107) f (cid:107) p,µ .
Revised: Conveniently, when X i ∼ µ , we have E [ (cid:107) f (cid:107) p,x 1: n ] = (cid:107) f (cid:107) p,µ .

Original: From now on, we consider the L 2 -norm when the subscript p is omitted.
Revised: From now on, we consider the L 2 -norm when the subscript p is omitted.

Original: Operators.
Revised: Operators.

Original: We deﬁne the operator P π as ( P π Q )( s, a ) (cid:44) (cid:82) S P ( ds (cid:48) | s, a ) (cid:80) a (cid:48)
Revised: We deﬁne the operator P π as ( P π Q )( s, a ) (cid:44) (cid:82) S P ( ds (cid:48) | s, a ) (cid:80) a (cid:48)

Original: ∈A π ( a (cid:48) | s (cid:48) )
Revised: ∈A π ( a (cid:48) | s (cid:48) )

Original: Q ( s (cid:48) , a (cid:48) ) , for any given policy π .
Revised: Q ( s (cid:48) , a (cid:48) ) , for any given policy π .

Original: Then, the Bellman operator T π : B ( S × A )
Revised: Then, the Bellman operator T π : B ( S × A )

Original: → B ( S × A ) is ( T π Q )( s, a ) (cid:44) r
Revised: → B ( S × A ) is ( T π Q )( s, a ) (cid:44) r

Original: ( s, a )
Revised: ( s, a )

Original: + γ ( P π Q )( s, a ) .
Revised: + γ ( P π Q )( s, a ) .

Original: The optimal Bellman operator T ∗ : B ( S × A )
Revised: The optimal Bellman operator T ∗ : B ( S × A )

Original: → B ( S × A ) is ( T ∗ Q )( s, a ) (cid:44) r
Revised: → B ( S × A ) is ( T ∗ Q )( s, a ) (cid:44) r

Original: ( s, a ) + γ (cid:82) S max a (cid:48) P ( ds (cid:48) | s, a ) Q ( s (cid:48) , a (cid:48) )) .
Revised: ( s, a ) + γ (cid:82) S max a (cid:48) P ( ds (cid:48) | s, a ) Q ( s (cid:48) , a (cid:48) )) .

Original: Finally, the ﬁxed point of the Bellman operator is ( T π Q π ) =
Revised: Finally, the ﬁxed point of the Bellman operator is ( T π Q π ) =

Original: Q π , and the ﬁxed point of the optimal Bellman operator is ( T ∗ Q ∗ ) =
Revised: Q π , and the ﬁxed point of the optimal Bellman operator is ( T ∗ Q ∗ ) =

Original: Q ∗ .
Revised: Q ∗ .

Original: As in Gy ¨ orﬁ et al.
Revised: As in Gy ¨ orﬁ et al.

Original: (2006), we deﬁne the truncation operator β B : B ( S × A )
Revised: (2006), we deﬁne the truncation operator β B : B ( S × A )

Original: → B ( S × A , B ) such that, given a function f ∈ B ( S × A ) , we have β B f ( s, a ) ∈ [ − B, B ] , ∀ ( s, a ) ∈ S × A .
Revised: → B ( S × A , B ) such that, given a function f ∈ B ( S × A ) , we have β B f ( s, a ) ∈ [ − B, B ] , ∀ ( s, a ) ∈ S × A .

Original: 3. A PPROXIMATE VALUE ITERATION
Revised: 3. A PPROXIMATE VALUE ITERATION

Original: The family of approximate value iteration (AVI) algorithms (Sutton & Barto, 2018; Farahmand, 2011; Munos, 2005) aims at approximating the solution of RL problems with large state space where ﬁnding an optimal solution is impractical or impossible.
Revised: The family of approximate value iteration (AVI) algorithms (Sutton & Barto, 2018; Farahmand, 2011; Munos, 2005) aims at approximating the solution of RL problems with large state space where ﬁnding an optimal solution is impractical or impossible.

Original: Starting from an arbitrary estimate
Revised: Starting from an arbitrary estimate

Original: Q 0 , AVI algorithms iteratively approximate the application of the optimal Bellman operator in a suitable functional space, i.e., Q k +1 ≈ T ∗ Q k .
Revised: Q 0 , AVI algorithms iteratively approximate the application of the optimal Bellman operator in a suitable functional space, i.e., Q k +1 ≈ T ∗ Q k .

Original: Intuitively, the approximation error at each iteration plays a central role in terms of convergence to the ﬁxed point Q ∗ .
Revised: Intuitively, the approximation error at each iteration plays a central role in terms of convergence to the ﬁxed point Q ∗ .

Original: The approximation error results from two key aspects: (i) the unknown optimal Bellman operator, and (ii) the representation of the action-value function in a suitable functional space F via the application of a nonlinear operator
Revised: The approximation error results from two key aspects: (i) the unknown optimal Bellman operator, and (ii) the representation of the action-value function in a suitable functional space F via the application of a nonlinear operator

Original: S : B ( S × A ) → F deﬁned as Sy = arg inf f ∈F (cid:107) f − y (cid:107) 2 µ for any y ∈ B ( S × A ) .
Revised: S : B ( S × A ) → F deﬁned as Sy = arg inf f ∈F (cid:107) f − y (cid:107) 2 µ for any y ∈ B ( S × A ) .

Original: We denote the resulting approximation error by
Revised: We denote the resulting approximation error by

Original: [Equation]
Revised: [Equation]

Original: The role of the approximation error in AVI is evinced by the following theorem that bounds the performance loss, w.r.t.
Revised: The role of the approximation error in AVI is evinced by the following theorem that bounds the performance loss, w.r.t.

Original: the optimal performance, at iteration K 1 .
Revised: the optimal performance, at iteration K 1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: ((Farahmand, 2011)).
Revised: ((Farahmand, 2011)).

Original: Let K be a positive integer, ρ an initial state distribution, and
Revised: Let K be a positive integer, ρ an initial state distribution, and

Original: Q max ≤ R max 1 − γ .
Revised: Q max ≤ R max 1 − γ .

Original: Then, for any sequence ( Q k ) Kk =

Revised: Then, for any sequence ( Q k ) Kk =

Original: ⊂ B ( X × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 deﬁned in (1), we have
Revised: ⊂ B ( X × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 deﬁned in (1), we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: − r ) k (cid:34) (cid:88) m ≥γ m (cid:16) c VI 1 ,ρ,µ ( m, K − k ; π (cid:48) K )+ c VI 2 ,ρ,ν ( m + 1; π (cid:48)
Revised: − r ) k (cid:34) (cid:88) m ≥γ m (cid:16) c VI 1 ,ρ,µ ( m, K − k ; π (cid:48) K )+ c VI 2 ,ρ,ν ( m + 1; π (cid:48)

Original: k +1 , . . .
Revised: k +1 , . . .

Original: , π (cid:48) K ) (cid:17)(cid:35)and E ( ε 0 , . . .
Revised: , π (cid:48) K ) (cid:17)(cid:35)and E ( ε 0 , . . .

Original: , ε K − 1 ; r ) =
Revised: , ε K − 1 ; r ) =

Original: (cid:80) K − 1 k =

Revised: (cid:80) K − 1 k =

Original: α 2 rk (cid:107) ε k (cid:107) 2 µ .
Revised: α 2 rk (cid:107) ε k (cid:107) 2 µ .

Original: We refer to Farahmand (2011) for the deﬁnition of c VI 1 , c VI 2 , and α i .
Revised: We refer to Farahmand (2011) for the deﬁnition of c VI 1 , c VI 2 , and α i .

Original:  C URRICULUM REINFORCEMENT LEARNING WITH B ELLMAN RESIDUALS
Revised:  C URRICULUM REINFORCEMENT LEARNING WITH B ELLMAN RESIDUALS

Original: We consider the classic curriculum RL scenario where an agent aims at learning a complex target task by solving a predeﬁned sequence of easier, but increasingly challenging, versions of it (Narvekar et al., 2020).
Revised: We consider the classic curriculum RL scenario where an agent aims at learning a complex target task by solving a predeﬁned sequence of easier, but increasingly challenging, versions of it (Narvekar et al., 2020).

Original: To avoid confusion on the debatable notion of difﬁculty of a task, in this work we adopt a practical point of view where the difﬁculty of a task is related to well-known aspects of RL, e.g., the complexity of the policies for solving them and challenging exploration (Florensa et al., 2017; Ivanovic et al., 2019; Klink et al., 2020).
Revised: To avoid confusion on the debatable notion of difﬁculty of a task, in this work we adopt a practical point of view where the difﬁculty of a task is related to well-known aspects of RL, e.g., the complexity of the policies for solving them and challenging exploration (Florensa et al., 2017; Ivanovic et al., 2019; Klink et al., 2020).

Original: 4. A PPROXIMATE VALUE ITERATION IN CURRICULUM REINFORCEMENT LEARNING
Revised: 4. A PPROXIMATE VALUE ITERATION IN CURRICULUM REINFORCEMENT LEARNING

Original: Let T 1: n = {T 1 , . .
Revised: Let T 1: n = {T 1 , . .

Original: . , T n } be a sequence of increasingly complex MDPs sharing the same state and action spaces, i.e., T i
Revised: . , T n } be a sequence of increasingly complex MDPs sharing the same state and action spaces, i.e., T i

Original: =
Revised: =

Original: (cid:104)S , A , P i , R i , γ (cid:105) , and T n be the target task.
Revised: (cid:104)S , A , P i , R i , γ (cid:105) , and T n be the target task.

Original: The use of AVI algorithms in this curriculum RL setting consists in starting from an arbitrary estimate Q 01 (where the subscript indicates the position of the task in the curriculum) and approximating the application of the taskspeciﬁc optimal Bellman operator for the current task Q k +1 t ≈ T ∗ t Q kt , or when the task at hand changes Q k +1 t +1 ≈ T ∗ t +1 Q kt .
Revised: The use of AVI algorithms in this curriculum RL setting consists in starting from an arbitrary estimate Q 01 (where the subscript indicates the position of the task in the curriculum) and approximating the application of the taskspeciﬁc optimal Bellman operator for the current task Q k +1 t ≈ T ∗ t Q kt , or when the task at hand changes Q k +1 t +1 ≈ T ∗ t +1 Q kt .

Original: We assume that the task changes when the estimate of the action-value of the current task converges.
Revised: We assume that the task changes when the estimate of the action-value of the current task converges.

Original: For simplicity, and without loss of generality, we denote K t the number of iterations needed to converge for each task T t , and denote Q 0 t =
Revised: For simplicity, and without loss of generality, we denote K t the number of iterations needed to converge for each task T t , and denote Q 0 t =

Original: Q K t − 1 t − 1 =
Revised: Q K t − 1 t − 1 =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let K be the current iteration.
Revised: Let K be the current iteration.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: This result can be veriﬁed applying the triangle inequality under the assumption that the algorithm switches task when the stopping condition of the algorithm is veriﬁed, e.g., when reaching a chosen number of iterations K t that allows the action-value function estimate to converge, or using heuristic methods checking the magnitude of the update (Lagoudakis & Parr, 2003; Ernst et al., 2005).
Revised: This result can be veriﬁed applying the triangle inequality under the assumption that the algorithm switches task when the stopping condition of the algorithm is veriﬁed, e.g., when reaching a chosen number of iterations K t that allows the action-value function estimate to converge, or using heuristic methods checking the magnitude of the update (Lagoudakis & Parr, 2003; Ernst et al., 2005).

Original: Note that each component of the sum in (2) can be bounded by the result in Theorem 1.
Revised: Note that each component of the sum in (2) can be bounded by the result in Theorem 1.

Original: We denote the k -step Bellman residual for task t by
Revised: We denote the k -step Bellman residual for task t by

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Given an initial Q 01 ( s, a )=0 for any ( s, a ) ∈ S × A , and considering that
Revised: Given an initial Q 01 ( s, a )=0 for any ( s, a ) ∈ S × A , and considering that

Original: Q ∗ t =Q K t t ≈
Revised: Q ∗ t =

Original: [Equation]
Revised: [Equation]

Original: T ∗ t ) K t Q ∗ t − 1 , we obtain
Revised: ) K t Q ∗ t − 1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: This result shows that, approximately, the action-value function of the target task is equivalent to the sum of k -step Bellman residuals.
Revised: This result shows that, approximately, the action-value function of the target task is equivalent to the sum of k -step Bellman residuals.

Original: Notably, especially if the curriculum is sufﬁciently ﬁne-grained, residuals becomes progressively smoother and smaller.
Revised: Notably, especially if the curriculum is sufﬁciently ﬁne-grained, residuals becomes progressively smoother and smaller.

Original: In the following, we describe how to compute the estimate shown in (4).
Revised: In the following, we describe how to compute the estimate shown in (4).

Original: 4. B OOSTED CURRICULUM REINFORCEMENT LEARNING
Revised: 4. B OOSTED CURRICULUM REINFORCEMENT LEARNING

Original: It is well-known from supervised learning that: (i) small function approximators are easier to train, and thus are desirable to solve simple tasks; (ii) large function approximators can overﬁt or suffer slow training speed in simple tasks, but they are more suitable to handle complex problems.
Revised: It is well-known from supervised learning that: (i) small function approximators are easier to train, and thus are desirable to solve simple tasks; (ii) large function approximators can overﬁt or suffer slow training speed in simple tasks, but they are more suitable to handle complex problems.

Original: Ideally, we want to increase the representativeness of the function approximator with the increase of task complexity.
Revised: Ideally, we want to increase the representativeness of the function approximator with the increase of task complexity.

Original: Recalling (4), we propose to use a separate function approximator for each residual (cid:37) t , i.e. a separate approximator for each task.
Revised: Recalling (4), we propose to use a separate function approximator for each residual (cid:37) t , i.e. a separate approximator for each task.

Original: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni =1 for each task, we denote the empirical Bellman operator by ( ˆ T ∗ t Q )( s i t , a it ) (cid:44) r
Revised: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni =1 for each task, we denote the empirical Bellman operator by ( ˆ T ∗ t Q )( s i t , a it ) (cid:44) r

Original: i t +
Revised: i t +

Original: γ max a (cid:48) Q ( s (cid:48)
Revised: γ max a (cid:48) Q ( s (cid:48)

Original: i t , a (cid:48) ) , and deﬁne the k -step empirical residual as ˜ (cid:37) kt (cid:44) ( ˆ T ∗ t ) k Q 0 t − Q 0 t .
Revised: i t , a (cid:48) ) , and deﬁne the k -step empirical residual as ˜ (cid:37) kt (cid:44) ( ˆ T ∗ t ) k Q 0 t − Q 0 t .

Original: At iteration k , for a current task t , and deﬁning the nonlinear operator ˆ S as the empirical version of the operator S , we compute an approximation
Revised: At iteration k , for a current task t , and deﬁning the nonlinear operator ˆ S as the empirical version of the operator S , we compute an approximation

Original: [Equation]
Revised: [Equation]

Original: Input: Tasks T 1: n , t = 1 , dataset D , Q 01 = 0 ; while t < n + 1 do for k = 0 , . .
Revised: Input: Tasks T 1: n , t = 1 , dataset D , Q 01 = 0 ; while t < n + 1 do for k = 0 , . .

Original: , K t do ˜ (cid:37) kt ← ˆ T ∗ t Q kt − Q 0 t ;
Revised: , K t do ˜ (cid:37) kt ← ˆ T ∗ t Q kt − Q 0 t ;

Original: Q k +1 t ← Q 0 t + β B k arg inf f ∈F (cid:107)
Revised: Q k +1 t ← Q 0 t + β B k arg inf f ∈F (cid:107)

Original: f − ˜ (cid:37) kt (cid:107) µ ; end
Revised: f − ˜ (cid:37) kt (cid:107) µ ; end

Original: Q 0 t +1 ← Q K t t ; t ← t + 1 ; end
Revised: Q 0 t +1 ← Q K t t ; t ← t + 1 ; end

Original: Output: Q K n n = (cid:80) nt =
Revised: Output: Q K n n = (cid:80) nt =

Original: 1 ˆ (cid:37) K t t .
Revised: 1 ˆ (cid:37) K t t .

Original: where we deﬁned ˆ (cid:37) kt = β B k ˆ S ˜ (cid:37) kt for compactness.
Revised: where we deﬁned ˆ (cid:37) kt = β B k ˆ S ˜ (cid:37) kt for compactness.

Original: We call this approach boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting procedure in supervised learning (Schapire, 1990; Freund, 1995), and we show its pseudocode in Algorithm 1.
Revised: We call this approach boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting procedure in supervised learning (Schapire, 1990; Freund, 1995), and we show its pseudocode in Algorithm 1.

Original: Remarks.
Revised: Remarks.

Original: Our approach is based on the key insight that the strength of the function approximator should increase according to the complexity of the task at hand.
Revised: Our approach is based on the key insight that the strength of the function approximator should increase according to the complexity of the task at hand.

Original: In BCRL , we accomplish this estimating an action-value function through a sum of action-value functions of the previous tasks in the curriculum.
Revised: In BCRL , we accomplish this estimating an action-value function through a sum of action-value functions of the previous tasks in the curriculum.

Original: Thus, contrary to curriculum RL which uses the same functional space to learn all the tasks in a curriculum, our approach has the advantage of acting in an augmented functional space resulting from the sum of the functional space of each approximator.
Revised: Thus, contrary to methods using the same functional space for all tasks, our approach has the advantage of acting in an augmented functional space resulting from the sum of the functional space of each approximator.

Original: In supervised learning literature, the function approximators in the sum are commonly known as weak regressors, meaning that boosting has the advantage of estimating complex functions as a sum of simple regressors (Geurts et al., 2006).
Revised: In supervised learning literature, the function approximators in the sum are commonly known as weak regressors, meaning that boosting has the advantage of estimating complex functions as a sum of simple regressors (Geurts et al., 2006).

Original: Similarly, our BCRL procedure can use simple function approximators for each task in the curriculum, with the advantages that (i) easy tasks can be learned with easy function approximators curbing the risk of overﬁtting, (ii) the complexity of the function approximator can be chosen according to the ﬁrst tasks in the curriculum, contrarily to regular curriculum RL where the function approximator needs to be suitable for the target task, and (iii) the training time to ﬁt the model on each task is reduced due to the smoothness of the residuals.
Revised: Similarly, our BCRL procedure can use simple function approximators for each task in the curriculum, with the advantages that (i) easy tasks can be learned with easy function approximators curbing the risk of overﬁtting, (ii) the complexity of the function approximator can be chosen according to the ﬁrst tasks in the curriculum, contrarily to regular curriculum RL where the function approximator needs to be suitable for the target task, and (iii) the training time to ﬁt the model on each task is reduced due to the smoothness of the residuals.

Original: Conversely, at inference time, our method requires a forward pass for each approximator in the ensemble.
Revised: Conversely, at inference time, our method requires a forward pass for each approximator in the ensemble.

Original: Nevertheless, the small size of the regressors results in negligible increase of computation time.
Revised: Nevertheless, the small size of the regressors results in negligible increase of computation time.

Original: Finally, we clarify that the soundness of this procedure is guaranteed by the fact that if the functional space of each weak regressor is Glivenko-Cantelli, i.e., ”the error due to the empirical process goes to zero at least asymptotically”, then, due to the preservation theorem, also the space resulting from the sum of weak regressors is Glivenko-Cantelli (VanDer Vaart & Wellner, 2000;
Revised: Finally, we clarify that the soundness of this procedure is guaranteed by the fact that if the functional space of each weak regressor is Glivenko-Cantelli, i.e., ”the error due to the empirical process goes to zero at least asymptotically”, then, due to the preservation theorem, also the space resulting from the sum of weak regressors is Glivenko-Cantelli (Van

Original:  Tosatto et al., 2017).
Revised: Der Vaart & Wellner, 2000; Tosatto et al., 2017).

Original:  T HEORETICAL ANALYSIS
Revised:  T HEORETICAL ANALYSIS

Original: In this section, we provide a theoretical analysis of BCRL in terms of convergence guarantees and ﬁnite-sample approximation error analysis.
Revised: In this section, we provide a theoretical analysis of BCRL in terms of convergence guarantees and ﬁnite-sample approximation error analysis.

Original: For simplicity and ease of notation, we assume the tasks in the curriculum have the same transition kernels, and different reward functions.
Revised: For simplicity and ease of notation, we assume the tasks in the curriculum have the same transition kernels, and different reward functions.

Original: Nevertheless, our results can be straightforwardly extended to the general case.
Revised: Nevertheless, our results can be straightforwardly extended to the general case.

Original: We refer to the well-known results in
Revised: We refer to the well-known results in

Original: AVI (Farahmand, 2011; Farahmand et al., 2010), and the analysis provided in Tosatto et al. (2017), with which our analysis share some similarities.
Revised: AVI (Farahmand, 2011; Farahmand et al., 2010), and the analysis provided in Tosatto et al. (2017), with which our analysis share some similarities.

Original: 5. C ONVERGENCE GUARANTEES
Revised: 5. C ONVERGENCE GUARANTEES

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q i t ) k
Revised: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q i t ) k

Original: i =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =
Revised: i =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.
Revised: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.

Original: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )
Revised: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )

Original: y (cid:107) µ ≤ λ (cid:107) y
Revised: y (cid:107) µ ≤ λ (cid:107) y

Original: (cid:107) µ for all y ∈ B ( S × A ) .Then, (cid:107) Q [Equation] − Q ∗ t (cid:107) µ ≤ ((1 +
Revised: (cid:107) µ for all y ∈ B ( S × A ) .

Original: [Equation]

Original:  k ) λ + [Equation] k ) (cid:107) Q 0 t − Q ∗ t (cid:107) µ .
Revised: Then, (cid:107) Q k t − Q ∗ t (cid:107) µ ≤ ((1 + L k ) λ + L k ) (cid:107) Q 0 t − Q ∗ t (cid:107) µ .

Original: We observe that the boosting procedure adds the (1 + L k ) λ term in the regular AVI bound, that is obtained for λ = 0 .
Revised: We observe that the boosting procedure adds the (1 + L k ) λ term in the regular AVI bound, that is obtained for λ = 0 .

Original: This result shows that for each task, for appropriate values of λ and L , the boosting procedure converges to the optimal action-value.
Revised: This result shows that for each task, for appropriate values of λ and L , the boosting procedure converges to the optimal action-value.

Original: Corollary 1.
Revised: Corollary 1.

Original: Given the settings of Theorem 2, the sequence of ( Q it ) ki =0 converges to Q ∗ t when λ < 1 −
Revised: Given the settings of Theorem 2, the sequence of ( Q it ) ki =0 converges to Q ∗ t when λ < 1 −

Original: L k 1 + L k and L k < 1 .
Revised: L k 1 + L k and L k < 1 .

Original: 5. F INITE - SAMPLE APPROXIMATION ERROR ANALYSIS
Revised: 5. F INITE - SAMPLE APPROXIMATION ERROR ANALYSIS

Original: In Lemma 1, we show how to derive the AVI performance loss analysis (Farahmand, 2011) for curriculum RL.
Revised: In Lemma 1, we show how to derive the AVI performance loss analysis (Farahmand, 2011) for curriculum RL.

Original: Now, we want to characterize it for our BCRL method.
Revised: Now, we want to characterize it for our BCRL method.

Original: Instead of using the regular
Revised: Instead of using the regular

Original: AVI approximation error deﬁned in (1), we consider the approximation error for each task t in the curriculum deﬁned as
Revised: AVI approximation error deﬁned in (1), we consider the approximation error for each task t in the curriculum deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.

Original: Denoting the regular AVI residual as ¯ (cid:37) kt
Revised: Denoting the regular AVI residual as ¯ (cid:37) kt

Original: = ( T ∗ t ) k . . .
Revised: = ( T ∗ t ) k . . .

Original: ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t − 1 ) K t − 1 . . .
Revised: ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t − 1 ) K t − 1 . . .

Original: ( T ∗ 1 ) K 1 Q 01 , we have
Revised: ( T ∗ 1 ) K 1 Q 01 , we have

Original: [Equation]
Revised: [Equation]

Original: This bound shows the relation between the approximation error of the residual for task t at step k , the AVI Bellman Residual ¯ (cid:37) kt , and the approximation error for the residuals of the previous tasks.
Revised: This bound shows the relation between the approximation error of the residual for task t at step k , the AVI Bellman Residual ¯ (cid:37) kt , and the approximation error for the residuals of the previous tasks.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: We use (9) and (10) to ﬁnally bound the approximation error of the residual as follows.
Revised: We use (9) and (10) to ﬁnally bound the approximation error of the residual as follows.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let B tk = max( (cid:107) ˜ (cid:37)
Revised: Let B tk = max( (cid:107) ˜ (cid:37)

Original: kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).
Revised: kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).

Original: For given dataset D t = { ( s i t , a i t , r i t , s (cid:48)
Revised: For given dataset D t = { ( s i t , a i t , r i t , s (cid:48)

Original: i t ) } N i
Revised: i t ) } N i

Original: =
Revised: =

Original: for each task t , and a given sequence of action-value functions ( Q it )
Revised: for each task t , and a given sequence of action-value functions ( Q it )

Original: ki =0 generated by our BCRL , we have
Revised: ki =0 generated by our BCRL , we have

Original: [Equation]
Revised: [Equation]

Original: This result bounds the error of BCRL by a sum of three components: (i) the approximation error of the residual ¯ (cid:37) k t , (ii) the propagation error due to approximation errors at previous iterations, (iii) the estimation error due to the regression problem with ﬁnite amount of samples.
Revised: This result bounds the error of BCRL by a sum of three components: (i) the approximation error of the residual ¯ (cid:37) k t , (ii) the propagation error due to approximation errors at previous iterations, (iii) the estimation error due to the regression problem with ﬁnite amount of samples.

Original: The advantages of
Revised: The advantages of

Original: BCRL are motivated by the use of residuals ¯ (cid:37) kt and ˜ (cid:37) kt , whose range of values become smaller as the curriculum proceeds, resulting in tighter approximation and estimation error bounds compared to regular curriculum RL.
Revised: BCRL are motivated by the use of residuals ¯ (cid:37) kt and ˜ (cid:37) kt , whose range of values become smaller as the curriculum proceeds, resulting in tighter approximation and estimation error bounds compared to regular curriculum RL.

Original: Moreover, BCRL enjoys advantages in terms of sample-efﬁciency, as fewer samples than regular curriculum RL can be used to achieve comparable estimation errors, due to the smaller range and smoother shape of the target (cid:107) ˜ (cid:37) kt (cid:107) µ compared to Q max t .
Revised: Moreover, BCRL enjoys advantages in terms of sample-efﬁciency, as fewer samples than regular curriculum RL can be used to achieve comparable estimation errors, due to the smaller range and smoother shape of the target (cid:107) ˜ (cid:37) kt (cid:107) µ compared to Q max t .

Original: Iteration 0 .0 .0 .

Revised: Iteration 0 .0 .0 .

Original: .k
Revised: .k

Original: Q k t − Q ∗ t k 1 , µ
Revised: Q k t − Q ∗ t k 1 , µ

Original: T T
Revised: T T

Original: T
Revised: T

Original: BC-FQI B-FQI
Revised: BC-FQI B-FQI

Original: C-FQI FQI
Revised: C-FQI FQI

Original: Iteration − 0 .0 .0 .0 .
Revised: Iteration − 0 .0 .0 .0 .

Original: C u m .
Revised: C u m .

Original: D i s c .
Revised: D i s c .

Original: R e t u r n
Revised: R e t u r n

Original: T T
Revised: T T

Original: T
Revised: T

Original: BC-FQI B-FQI
Revised: BC-FQI B-FQI

Original: C-FQI FQI
Revised: C-FQI FQI

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section serves to provide empirical evidence to the insights obtained in the previous section.
Revised: This section serves to provide empirical evidence to the insights obtained in the previous section.

Original: More precisely, we want to provide evidence that boosted curricula can indeed improve the residual in an AVI setting and with that improve the achieved agent performance.
Revised: More precisely, we want to provide evidence that boosted curricula can indeed improve the residual in an AVI setting and with that improve the achieved agent performance.

Original: Further, we show that the additional ﬂexibility of boosting w.r.t.
Revised: Further, we show that the additional ﬂexibility of boosting w.r.t.

Original: the individual function approximators can be leveraged to choose the best-suited learning approach for each learning task.
Revised: the individual function approximators can be leveraged to choose the best-suited learning approach for each learning task.

Original: We conduct experiments in multiple environments and with two different RL algorithms using the
Revised: We conduct experiments in multiple environments and with two different RL algorithms using the

Original: MushroomRL library (D’Eramo et al., 2021).
Revised: MushroomRL library (D’Eramo et al., 2021).

Original: 2 To evaluate the beneﬁt of curricula in boosted AVI settings, we turn towards the ﬁtted Q -iteration ( FQI ) algorithm in the car-on-hill environment (Ernst et al., 2005), as the low-dimensionality allows for uniform sampling of the state-action space assumed in the previous section and the ground truth optimal Q -function can be computed via bruteforce methods (Ernst et al., 2005).
Revised: 2 To evaluate the beneﬁt of curricula in boosted AVI settings, we turn towards the ﬁtted Q -iteration ( FQI ) algorithm in the car-on-hill environment (Ernst et al., 2005), as the low-dimensionality allows for uniform sampling of the state-action space assumed in the previous section and the ground truth optimal Q -function can be computed via bruteforce methods (Ernst et al., 2005).

Original: Next, we evaluate the FQI algorithm in a maze environment in which exploration is guided by the currently estimated optimal policy.
Revised: Next, we evaluate the FQI algorithm in a maze environment in which exploration is guided by the currently estimated optimal policy.

Original: Finally, we investigate boosting in a control task with DQN (Mnih et al., 2015), in which a point-mass needs to be steered over a narrow pathway subject to external perturbations.
Revised: Finally, we investigate boosting in a control task with DQN (Mnih et al., 2015), in which a point-mass needs to be steered over a narrow pathway subject to external perturbations.

Original: To disentangle the interplay between boosting and the use of a curriculum, we will compare different methods in the experiments: the algorithm under investigation ( FQI , DQN ), a version using both a curriculum with boosting ( BC - FQI , BC - DQN ) as well as two ablations of the second method - one without boosting ( C - FQI , C - DQN ) and one without a curriculum ( B - FQI , B - DQN ).
Revised: To disentangle the interplay between boosting and the use of a curriculum, we will compare different methods in the experiments: the algorithm under investigation ( FQI , DQN ), a version using both a curriculum with boosting ( BC - FQI , BC - DQN ) as well as two ablations of the second method - one without boosting ( C - FQI , C - DQN ) and one without a curriculum ( B - FQI , B - DQN ).

Original: The boosted methods that do not use a curriculum will introduce the additional approximators at the same iteration at which the curricula switches between tasks.
Revised: The boosted methods that do not use a curriculum will introduce the additional approximators at the same iteration at which the curricula switches between tasks.

Original: 6. C AR - ON - HILL
Revised: 6. C AR - ON - HILL

Original: The three car-on-hill tasks T 1 , T 2 and T 3 considered for curriculum learning in this experiment differ in the mass of the car, taking values of 0 .
Revised: The three car-on-hill tasks T 1 , T 2 and T 3 considered for curriculum learning in this experiment differ in the mass of the car, taking values of 0 .

Original: 8 , 1 . 0 and 1 . 2 respectively.
Revised: 8 , 1 . 0 and 1 . 2 respectively.

Original: The default mass of the environment is 1 . 

Revised: The default mass of the environment is 1 .

Original: which means we are considering a version of this environment with more challenging exploration, as the car needs more energy to escape the valley and receive a reward signal.
Revised: which means we are considering a version of this environment with more challenging exploration, as the car needs more energy to escape the valley and receive a reward signal.

Original: The dataset for learning the Q -function in each task is generated by executing 1000 trajectories with a random policy.
Revised: The dataset for learning the Q -function in each task is generated by executing 1000 trajectories with a random policy.

Original: To not make the results in this section dependent on the parametric form of the function approximator, we resort to the tree-based regression investigated in Ernst et al.
Revised: To not make the results in this section dependent on the parametric form of the function approximator, we resort to the tree-based regression investigated in Ernst et al.

Original: The results are visualized in Figure 1.
Revised: The results are visualized in Figure 1.

Original: We can see that the combination of boosting and a curriculum ( BC - FQI ) outperforms all other methods both in terms of ﬁnal approximation error achieved on Tbut also in terms of the performance of the greedy policy derived from the estimated Q -function.
Revised: We can see that the combination of boosting and a curriculum ( BC - FQI ) outperforms all other methods both in terms of ﬁnal approximation error achieved on Tbut also in terms of the performance of the greedy policy derived from the estimated Q -function.

Original: Comparing the residual of BC - FQI and C - FQI , we can see that C - FQI seems to be challenged with adapting the Q -function from T 1 to T 2 , as the jump in residual shows.
Revised: Comparing the residual of BC - FQI and C - FQI , we can see that C - FQI seems to be challenged with adapting the Q -function from T 1 to T 2 , as the jump in residual shows.

Original: BC - FQI does not exhibit such a jump in residual when moving from T 1 to T 2 .
Revised: BC - FQI does not exhibit such a jump in residual when moving from T 1 to T 2 .

Original: Furthermore, BC - FQI reduces the increased residual after switching from T 2 to T 3 while the residual of C - FQI steadily increases once arriving at the target task.
Revised: Furthermore, BC - FQI reduces the increased residual after switching from T 2 to T 3 while the residual of C - FQI steadily increases once arriving at the target task.

Original: We can particularly see that the pre-training in T 1 and T 2 leads to a signiﬁcantly smaller
Revised: We can particularly see that the pre-training in T 1 and T 2 leads to a signiﬁcantly smaller

Original: [Figure]
Revised: [Figure]

Original: k Q k t − Q ∗ t k 1 , µ ... T 5 T
Revised: k Q k t − Q ∗ t k 1 , µ ... T 5 T

Original: T
Revised: T

Original: T T
Revised: T T

Original: T 10 BC-FQI
Revised: T 10 BC-FQI

Original: B-FQI
Revised: B-FQI

Original: C-FQI FQI     

Revised: C-FQI FQI

Original: Iteration −−−−−
Revised: Iteration −−−−−

Original: C u m .
Revised: C u m .

Original: [Figure]
Revised: [Figure]

Original: i s c .
Revised: i s c .

Original: R e t u r n ... T 5 T 6 T
Revised: R e t u r n ... T 5 T 6 T

Original: T
Revised: T

Original: T T
Revised: T T

Original: [Figure]
Revised: [Figure]

Original: Q -function error of BC - FQI compared B - FQI at iteration 40 , which is in line with the discussion in the previous sections.
Revised: Q -function error of BC - FQI compared B - FQI at iteration 40 , which is in line with the discussion in the previous sections.

Original: 6. M AZE
Revised: 6. M AZE

Original: We design a maze environment, in which an agent needs to reach a target position by navigating around two large obstacles.
Revised: We design a maze environment, in which an agent needs to reach a target position by navigating around two large obstacles.

Original: To reach the goal, the agent can choose between 4 discrete actions that move the agent up, down, left, or right.
Revised: To reach the goal, the agent can choose between 4 discrete actions that move the agent up, down, left, or right.

Original: It obtains a dense reward based on the distance to the target and a reward of +5 upon reaching the goal.
Revised: It obtains a dense reward based on the distance to the target and a reward of +5 upon reaching the goal.

Original: We design a curriculum composed oftasks, in which the size of the walls gradually increases (see Figure 3a for a visualization of tasks contained in the curriculum).
Revised: We design a curriculum composed oftasks, in which the size of the walls gradually increases (see Figure 3a for a visualization of tasks contained in the curriculum).

Original: The reward function and the target task are designed to induce highly challenging exploration, requiring the use of curricula to solve the problem.
Revised: The reward function and the target task are designed to induce highly challenging exploration, requiring the use of curricula to solve the problem.

Original: As done for the caron-hill experiment, we use FQI with tree-based regression (Ernst et al., 2005).
Revised: As done for the caron-hill experiment, we use FQI with tree-based regression (Ernst et al., 2005).

Original: For all methods, we collect a dataset for each task running 500 episodes using ε -greedy exploration, decreasing the strength of exploration as the curriculum proceeds.
Revised: For all methods, we collect a dataset for each task running 500 episodes using ε -greedy exploration, decreasing the strength of exploration as the curriculum proceeds.

Original: As can be seen in Figures 2 and 3, regular learning on the target task does not allow ﬁnd a good solution, regardless of whether boosting is used or not ( B - FQI and FQI ).
Revised: As can be seen in Figures 2 and 3, regular learning on the target task does not allow ﬁnd a good solution, regardless of whether boosting is used or not ( B - FQI and FQI ).

Original: This is due to the highly challenging exploration problem that needs to be solved in this environment.
Revised: This is due to the highly challenging exploration problem that needs to be solved in this environment.

Original: The initial random exploration is insufﬁcient to maneuver around the second wall in the target task.
Revised: The initial random exploration is insufﬁcient to maneuver around the second wall in the target task.

Original: Consequently, the agent is not aware of the beneﬁt of this detour, as obvious in the learned Q -function (Figure 3b).
Revised: Consequently, the agent is not aware of the beneﬁt of this detour, as obvious in the learned Q -function (Figure 3b).

Original: Subsequently reducing the exploration does not lead to any new evidence and hence the agent quickly converges to the sub-optimal solution (Figure 2).
Revised: Subsequently reducing the exploration does not lead to any new evidence and hence the agent quickly converges to the sub-optimal solution (Figure 2).

Original: Looking at the results of C - FQI , we see that the curriculum alleviates the aforementioned exploration problem.
Revised: Looking at the results of C - FQI , we see that the curriculum alleviates the aforementioned exploration problem.

Original: However, the estimated Q -function deteriorates before approaching the target task, leading to unreliable policies.
Revised: However, the estimated Q -function deteriorates before approaching the target task, leading to unreliable policies.

Original: By introducing additional capacity as learning proceeds, BC - FQI allows to learn an optimal policy.
Revised: By introducing additional capacity as learning proceeds, BC - FQI allows to learn an optimal policy.

Original: Indeed, we show in the appendix that the residuals of BC - FQI decrease in later iterations, easing the function approximation problem when progressing to the target task.
Revised: Indeed, we show in the appendix that the residuals of BC - FQI decrease in later iterations, easing the function approximation problem when progressing to the target task.

Original: [Figure]
Revised: [Figure]

Original: 4 Steps × 10− 1 .− 1 .− 1 .− 1 .
Revised: 4 Steps × 10− 1 .− 1 .− 1 .− 1 .

Original: C u m .
Revised: C u m .

Original: D i s c .
Revised: D i s c .

Original: R e t u r n × 

Revised: R e t u r n ×

Original: BC-DQN
Revised: BC-DQN

Original: C-DQN
Revised: C-DQN

Original: DQN −−−−−0 × 10(a)
Revised: DQN −−−−−0 × 10(a)

Original: Agent Performance −  −−
Revised: Agent Performance −  −−

Original: LSPI −
Revised: LSPI −

Original: DQN
Revised: DQN

Original: [Figure]
Revised: [Figure]

Original: L INEAR SYSTEM CONTROL
Revised: L INEAR SYSTEM CONTROL

Original: The last task aims at highlighting a more subtle beneﬁt of BCRL :
Revised: The last task aims at highlighting a more subtle beneﬁt of BCRL :

Original: Choosing the architecture of the function approximators appropriately for each task in the curriculum.
Revised: Choosing the architecture of the function approximators appropriately for each task in the curriculum.

Original: We show that this ﬂexibility can improve sample efﬁciency in a control task, in which a point mass needs to be steered towards a goal position over a narrow pathway (Figure 4b).
Revised: We show that this ﬂexibility can improve sample efﬁciency in a control task, in which a point mass needs to be steered towards a goal position over a narrow pathway (Figure 4b).

Original: Along this pathway, the point mass is subject to a drift moving it away from the target position.
Revised: Along this pathway, the point mass is subject to a drift moving it away from the target position.

Original: If the agent falls off the pathway, it is reset to the initial position.
Revised: If the agent falls off the pathway, it is reset to the initial position.

Original: The agent receives a reward of − 1 until reaching the goal.
Revised: The agent receives a reward of − 1 until reaching the goal.

Original: Upon reaching the goal, a reward of 0 is achieved and the episode terminates.
Revised: Upon reaching the goal, a reward of 0 is achieved and the episode terminates.

Original: We learn a policy for solving this target task using DQN .
Revised: We learn a policy for solving this target task using DQN .

Original: We introduce a curriculum for learning this task by ﬁrst training in an LQR system to induce a bias of moving towards the target.
Revised: We introduce a curriculum for learning this task by ﬁrst training in an LQR system to induce a bias of moving towards the target.

Original: In this LQR system, the point mass behaves according to the same dynamics as in the target environment and needs to reach the same target position, but does not face any external perturbations, obtains a reward equal to the negative squared distance to the target position and does not need to stay on a particular pathway.
Revised: In this LQR system, the point mass behaves according to the same dynamics as in the target environment and needs to reach the same target position, but does not face any external perturbations, obtains a reward equal to the negative squared distance to the target position and does not need to stay on a particular pathway.

Original: A beneﬁt of environments with the aforementioned characteristics is that the Q -function can be shown to be a quadratic function of action and state (Bradtke, 1993).
Revised: A beneﬁt of environments with the aforementioned characteristics is that the Q -function can be shown to be a quadratic function of action and state (Bradtke, 1993).

Original: Since the boosting framework does not prescribe the architecture of the individual employed function approximators, we can exploit the aforementioned structure of the ﬁrst (LQR) learning task by employing least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003) with quadratic features.
Revised: Since the boosting framework does not prescribe the architecture of the individual employed function approximators, we can exploit the aforementioned structure of the ﬁrst (LQR) learning task by employing least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003) with quadratic features.

Original: Figure 4a shows that this incorporation of additional structure into the curriculum leads to improved performance over both directly learning on the target task ( DQN ) and not adjusting the structure of the agent to the learning task in the curriculum ( C - DQN ).
Revised: Figure 4a shows that this incorporation of additional structure into the curriculum leads to improved performance over both directly learning on the target task ( DQN ) and not adjusting the structure of the agent to the learning task in the curriculum ( C - DQN ).

Original: It is particularly surprising that C - DQN is performing even worse than DQN .
Revised: It is particularly surprising that C - DQN is performing even worse than DQN .

Original: There are multiple potential causes for this poor performance.
Revised: There are multiple potential causes for this poor performance.

Original: One is that the DQN algorithm takes signiﬁcantly longer to learn in the LQR task compared to LSPI ( 10 . 000 vs. 2 . 000 steps).
Revised: One is that the DQN algorithm takes signiﬁcantly longer to learn in the LQR task compared to LSPI ( 10 . 000 vs. 2 . 000 steps).

Original: Further, the learned Q -function and policy is not as accurate as the one learned with LSPI , as can be seen in Figure 4b.
Revised: Further, the learned Q -function and policy is not as accurate as the one learned with LSPI , as can be seen in Figure 4b.

Original: Consequently, the policy is not as stable.
Revised: Consequently, the policy is not as stable.

Original: Finally, due to the different reward structures of the two learning tasks, the Q -function needs to adjust during learning on the target task, potentially unlearning the stabilizing behavior induced by the initial task.
Revised: Finally, due to the different reward structures of the two learning tasks, the Q -function needs to adjust during learning on the target task, potentially unlearning the stabilizing behavior induced by the initial task.

Original: BC - DQN avoids this risk by not needing to adjust the Q -function learned in the previous task.
Revised: BC - DQN avoids this risk by not needing to adjust the Q -function learned in the previous task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced boosted curriculum reinforcement learning ( BCRL ), a new method for action-value function estimation in curriculum RL.
Revised: We introduced boosted curriculum reinforcement learning ( BCRL ), a new method for action-value function estimation in curriculum RL.

Original: For each task, BCRL approximates the respective action-value function as a sum of residuals of the previous tasks in the curriculum.
Revised: For each task, BCRL approximates the respective action-value function as a sum of residuals of the previous tasks in the curriculum.

Original: This procedure increases the representativeness of the functional space as the complexity of the tasks in the curriculum increases, instead of using a single approximator through all the learning.
Revised: This procedure increases the representativeness of the functional space as the complexity of the tasks in the curriculum increases, instead of using a single approximator through all the learning.

Original: We proved that our BCRL method enjoys advantageous theoretical properties in terms of convergence to the optimal actionvalue function of the target task.
Revised: We proved that our BCRL method enjoys advantageous theoretical properties in terms of convergence to the optimal actionvalue function of the target task.

Original: Moreover, we empirically validated BCRL across different problems and algorithms, showing its overall advantages over regular curriculum RL.
Revised: Moreover, we empirically validated BCRL across different problems and algorithms, showing its overall advantages over regular curriculum RL.

Original: A P ROOFS
Revised: A P ROOFS

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q it )
Revised: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =
Revised: =0 be a sequence of measurable action-value functions for a target task T t obtained following our BCRL procedure (5), where Q 0 t =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.
Revised: Denote L = γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.

Original: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )
Revised: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: λ (cid:107) y (cid:107) µ for all y ∈ B ( S × A ) .
Revised: λ (cid:107) y (cid:107) µ for all y ∈ B ( S × A ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where (13) considers the assumption on the operator ( I − S ) and (14) follows noting that
Revised: where (13) considers the assumption on the operator ( I − S ) and (14) follows noting that

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.

Original: Denoting the regular AVI residual as ¯ (cid:37) kt
Revised: Denoting the regular AVI residual as ¯ (cid:37) kt

Original: = ( T ∗ t ) k . . .
Revised: = ( T ∗ t ) k . . .

Original: ( T ∗ 1 ) K 1 Q 01 −
Revised: ( T ∗ 1 ) K 1 Q 01 −

Original: [Equation]
Revised: [Equation]

Original: T ∗ t − 1 ) K t − 1 . . .
Revised: T ∗ t − 1 ) K t − 1 . . .

Original: ( T ∗ 1 ) K 1 Q 01 , we have
Revised: ( T ∗ 1 ) K 1 Q 01 , we have

Original: [Equation]
Revised: [Equation]

Original: ¯ (cid:37)
Revised: ¯ (cid:37)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ε
Revised: ε

Original: [Equation]
Revised: [Equation]

Original: (cid:107) µ (cid:17) .
Revised: (cid:107) µ (cid:17) .

Original: Proof.
Revised: Proof.

Original: For an arbitrary function f ∈ F and task t , we have (cid:107) f − (cid:37) kt
Revised: For an arbitrary function f ∈ F and task t , we have (cid:107) f − (cid:37) kt

Original: (cid:107) µ ≤ (cid:107) f
Revised: (cid:107) µ ≤ (cid:107) f

Original: − ¯ (cid:37) kt (cid:107) µ + (cid:107) ¯ (cid:37) kt − (cid:37) kt (cid:107) µ .
Revised: − ¯ (cid:37) kt (cid:107) µ + (cid:107) ¯ (cid:37) kt − (cid:37) kt (cid:107) µ .

Original: Then, (cid:13)(cid:13)(cid:13) ¯ (cid:37) kt − (cid:37) kt (cid:13)(cid:13)(cid:13) µ ≤ (cid:13)(cid:13)(cid:13) ( T ∗ t ) k . . .
Revised: Then, (cid:13)(cid:13)(cid:13) ¯ (cid:37) kt − (cid:37) kt (cid:13)(cid:13)(cid:13) µ ≤ (cid:13)(cid:13)(cid:13) ( T ∗ t ) k . . .

Original: ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t ) k Q 0 t (cid:13)(cid:13)(cid:13) µ + (cid:13)(cid:13)(cid:13) ( T ∗ t − 1 ) K t − 1 . . .
Revised: ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t ) k Q 0 t (cid:13)(cid:13)(cid:13) µ + (cid:13)(cid:13)(cid:13) ( T ∗ t − 1 ) K t − 1 . . .

Original: ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ ≤ L k (cid:13) (cid:13)(cid:13)
Revised: ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ ≤ L k (cid:13) (cid:13)(cid:13)

Original: ( T ∗ t − 1 ) K t − 1 . . .
Revised: ( T ∗ t − 1 ) K t − 1 . . .

Original: ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ + (cid:13) (cid:13)(cid:13)
Revised: ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ + (cid:13) (cid:13)(cid:13)

Original: ( T ∗ t − 1 ) K t − 1 . . .
Revised: ( T ∗ t − 1 ) K t − 1 . . .

Original: ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ = (
Revised: ( T ∗ 1 ) K 1 Q 01 − Q 0 t (cid:13)(cid:13)(cid:13) µ = (

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)(cid:13) ( T
Revised: ) (cid:13)(cid:13)(cid:13) ( T

Original: [Equation]
Revised: [Equation]

Original: − 1 ) k t − 1 . . .
Revised: − 1 ) k t − 1 . . .

Original: ( T ∗ 1 ) K 1 Q 01 − Q 0 t
Revised: ( T ∗ 1 ) K 1 Q 01 − Q 0 t

Original: [Table]
Revised: [Table]

Original: Theorem 4.
Revised: Theorem 4.

Original: Let B tk = max( (cid:107) ˜ (cid:37)
Revised: Let B tk = max( (cid:107) ˜ (cid:37)

Original: kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).
Revised: kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).

Original: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni = for each task t , and a sequence of action-value functions ( Q it ) ki =0 generated by our BCRL, we have
Revised: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni = for each task t , and a sequence of action-value functions ( Q it ) ki =0 generated by our BCRL, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Theorem 5.11 from Gy¨orﬁ et al.
Revised: Using Theorem 5.11 from Gy¨orﬁ et al.

Original: (2006), we obtain
Revised: (2006), we obtain

Original: [Equation]
Revised: [Equation]

Original: We complete the proof using (15) and the Cauchy-Schwartz inequality to bound the ﬁrst term.
Revised: We complete the proof using (15) and the Cauchy-Schwartz inequality to bound the ﬁrst term.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we bound the k -step Bellman residual.
Revised: First, we bound the k -step Bellman residual.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value function for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) (cid:37) 1 t (cid:107) ∞
Revised: Since (cid:107) (cid:37) 1 t (cid:107) ∞

Original: ≤ R max t , solving the recursion proves the lemma.
Revised: ≤ R max t , solving the recursion proves the lemma.

Original: We are now able to derive the bound for the k -step empirical Bellman residual.
Revised: We are now able to derive the bound for the k -step empirical Bellman residual.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Iteration −−−
Revised: Iteration −−−

Original: P a % t ( s 0 , a ) # A ... T 5 T
Revised: P a % t ( s 0 , a ) # A ... T 5 T

Original: T
Revised: T

Original: T T
Revised: T T

Original: T
Revised: T

Original: BC-FQI
Revised: BC-FQI

Original: B E XPERIMENTAL DETAILS
Revised: B E XPERIMENTAL DETAILS

Original: This section serves to present additional details on the experiments that did not ﬁt in the main paper, such as algorithm hyperparameters and additional details of the learning environments.
Revised: This section serves to present additional details on the experiments that did not ﬁt in the main paper, such as algorithm hyperparameters and additional details of the learning environments.

Original: These additional details will be provided for each experiment individually in the following subsections.
Revised: These additional details will be provided for each experiment individually in the following subsections.

Original: The computations were executed on a desktop computer with a Geforce RTX 2080, 64 GB memory and an AMD Ryzen 9 3900X processor.
Revised: The computations were executed on a desktop computer with a Geforce RTX 2080, 64 GB memory and an AMD Ryzen 9 3900X processor.

Original: All experiments were run in roughly a week of computation time on this single machine.
Revised: All experiments were run in roughly a week of computation time on this single machine.

Original: The employed MushroomRL library is available under the MIT license.
Revised: The employed MushroomRL library is available under the MIT license.

Original: B. C AR - ON - HILL
Revised: B. C AR - ON - HILL

Original: As mentioned in the main text, we used the FQI algorithm with extra-trees (Ernst et al., 2005) implemented in MushroomRL (D’Eramo et al., 2021) to conduct this experiment.
Revised: As mentioned in the main text, we used the FQI algorithm with extra-trees (Ernst et al., 2005) implemented in MushroomRL (D’Eramo et al., 2021) to conduct this experiment.

Original: The number of trees in the extra-trees implementation was set to 50 , the minimum number of samples required to split an internal node to 5 and the minimum number of samples required to be a leaf node to 2 .
Revised: The number of trees in the extra-trees implementation was set to 50 , the minimum number of samples required to split an internal node to 5 and the minimum number of samples required to be a leaf node to 2 .

Original: All other parameters were left to the implementation defaults of MushroomRL.
Revised: All other parameters were left to the implementation defaults of MushroomRL.

Original: Based on this implementation we created a boosted version that runs the original FQI algorithm on the residuals of the previously learned models.
Revised: Based on this implementation we created a boosted version that runs the original FQI algorithm on the residuals of the previously learned models.

Original: The hyperparameters for those extra-trees models were the same as for the regular FQI .
Revised: The hyperparameters for those extra-trees models were the same as for the regular FQI .

Original: The implementation can be found in the accompanying code.
Revised: The implementation can be found in the accompanying code.

Original: The car-on-hill environment used for this experiment is also implemented in MushroomRL.
Revised: The car-on-hill environment used for this experiment is also implemented in MushroomRL.

Original: B. M AZE
Revised: B. M AZE

Original: The state space of the maze environments consists of the two-dimensional coordinates s =[ s x , s y ] ∈ [0 , 1] 2 .
Revised: The state space of the maze environments consists of the two-dimensional coordinates s =[ s x , s y ] ∈ [0 , 1] 2 .

Original: The initial state is s 0 =[0 , 0] , and the goal state is g 0 =[0 .
Revised: The initial state is s 0 =[0 , 0] , and the goal state is g 0 =[0 .

Original: The four actions displace the agent in x - or y -position by a value of 0 .
Revised: The four actions displace the agent in x - or y -position by a value of 0 .

Original: The discount factor is set to γ =0 .
Revised: The discount factor is set to γ =0 .

Original: 99 and the horizon length is 200 .
Revised: 99 and the horizon length is 200 .

Original: We use a dense reward function R ( · , · , s (cid:48) )
Revised: We use a dense reward function R ( · , · , s (cid:48) )

Original: =
Revised: =

Original: −| g − s (cid:48) | when | g − s (cid:48)
Revised: −| g − s (cid:48) | when | g − s (cid:48)

Original: | > = 0 .
Revised: | > = 0 .

Original: 1 , otherwise R ( · , · , s (cid:48) )
Revised: 1 , otherwise R ( · , · , s (cid:48) )

Original: =
Revised: =

Original: +5 and the episode terminates.
Revised: +5 and the episode terminates.

Original: There are two walls centered at x = 0 .
Revised: There are two walls centered at x = 0 .

Original: 2 and x = 0 .
Revised: 2 and x = 0 .

Original: 6 , both with width w =0 .
Revised: 6 , both with width w =0 .

Original: 05 , and height h t
Revised: 05 , and height h t

Original: depending on the task t .
Revised: depending on the task t .

Original: We design a curriculum composed of 10 tasks, with h 1 ,..., 9 = { 0 . 05 , 0 . 15 , . . .
Revised: We design a curriculum composed of 10 tasks, with h 1 ,..., 9 = { 0 . 05 , 0 . 15 , . . .

Original: , 0 . 85 } and h 10 =0 .
Revised: , 0 . 85 } and h 10 =0 .

Original: Hence in the ﬁnal task T 10 , there is only a very narrow passage through which the walls can be passed.
Revised: Hence in the ﬁnal task T 10 , there is only a very narrow passage through which the walls can be passed.

Original: We again make use of the FQI algorithm with extra-trees implemented in MushroomRL to conduct this experiment, again using 50 trees, a minimum number of 5 samples to split an internal node, a minimum number of 2 samples to be a leaf node and everything else left to the implementation defaults of MushroomRL.
Revised: We again make use of the FQI algorithm with extra-trees implemented in MushroomRL to conduct this experiment, again using 50 trees, a minimum number of 5 samples to split an internal node, a minimum number of 2 samples to be a leaf node and everything else left to the implementation defaults of MushroomRL.

Original: The main conceptual different to the car-on-hill experiment is that exploration is guided by the currently estimated Q -function, as we are employing an εgreedy exploration strategy to generate the samples for learning the Q -functions of the learning tasks.
Revised: The main conceptual different to the car-on-hill experiment is that exploration is guided by the currently estimated Q -function, as we are employing an εgreedy exploration strategy to generate the samples for learning the Q -functions of the learning tasks.

Original: The exploration percentage ε is annealed in later environments.
Revised: The exploration percentage ε is annealed in later environments.

Original: More precisely, we set ε 1 ,..., 10 = { 1 ., 1 ., 1 ., 1 ., .
Revised: More precisely, we set ε 1 ,..., 10 = { 1 ., 1 ., 1 ., 1 ., .

Original: 25 } for the tasks T 1 , . .
Revised: 25 } for the tasks T 1 , . .

Original: , T 10 .
Revised: , T 10 .

Original: Figure 5 visualizes the residuals of the starting state s 0 that the Q -functions learned in the individual tasks, i.e., the additional values the Q -functions needed to predict in addition to the Q -value predicted from the ensemble of previously learned Q -functions.
Revised: Figure 5 visualizes the residuals of the starting state s 0 that the Q -functions learned in the individual tasks, i.e., the additional values the Q -functions needed to predict in addition to the Q -value predicted from the ensemble of previously learned Q -functions.

Original: We see that the magnitude of residuals continuously decreases from task T 1 to task T 6 .
Revised: We see that the magnitude of residuals continuously decreases from task T 1 to task T 6 .

Original: Switching to task T 7 increases the magnitude of the residuals after which it then continuously declines again.
Revised: Switching to task T 7 increases the magnitude of the residuals after which it then continuously declines again.

Original: The reason for this jump is that, while the optimal policy is largely unaffected by the presence of the small walls in the early learning tasks, a more signiﬁcant detour is required starting from task T 7 .
Revised: The reason for this jump is that, while the optimal policy is largely unaffected by the presence of the small walls in the early learning tasks, a more signiﬁcant detour is required starting from task T 7 .

Original: B.2. L INEAR SYSTEM CONTROL
Revised: B. L INEAR SYSTEM CONTROL

Original: The linear system that needs to be controlled is given by
Revised: The linear system that needs to be controlled is given by

Original: [Equation]
Revised: [Equation]

Original: 

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 0 .
Revised: 0 .

Original:    x ˙ xy ˙
Revised:    x [Equation]

Original: y 
Revised: y 

Original:  +   0100  (cid:20) a x a y (cid:21) =
Revised: [Equation]   0100  (cid:20) a x a y (cid:21) =

Original: As + Ba , representing a simple, fully-actuated point-mass with a slight amount of friction.
Revised: As [Equation] Ba , representing a simple, fully-actuated point-mass with a slight amount of friction.

Original: For the LQR task, to which the curriculum learning agents are exposed ﬁrst, the reward function is given by
Revised: For the LQR task, to which the curriculum learning agents are exposed ﬁrst, the reward function is given by

Original: [Equation]
Revised: [Equation]

Revised: ( s , a , s

Revised: [Equation]

Revised: [Equation]

Revised:  3 0 00 1 00 0 30 0 0

Original: [Equation]
Revised: [Equation]

Original: 0
Revised: a

Original: [Equation]
Revised: [Equation]

Revised: (cid:20) 0 .

Original:  0 .(cid:21)
Revised:  0 .(cid:21)

Original: a and the initial state is given by s 0 =
Revised: a and the initial state is given by s 0 =

Original: [ − 4 v x, 0 0 v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .
Revised: [ − 4 v x, 0 0 v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .

Original: In the target task, the agent receives the following reward signal
Revised: In the target task, the agent receives the following reward signal

Original: [Equation]
Revised: [Equation]

Original: 0 , else and its initial position is given by s 0 =
Revised: 0 , else and its initial position is given by s 0 =

Original: [ − 4 v x, 0 y v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .
Revised: [ − 4 v x, 0 y v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .

Original: 1 , 0 . 1] and y within [ − 0 . 4 , 0 . 4] .
Revised: 1 , 0 . 1] and y within [ − 0 . 4 , 0 . 4] .

Original: Furthermore, the agent needs to keep its y -position within [ − 0 . 5 , 0 . 5] in order to stay on the pathway indicated in Figure 4b and it faces an additional drift moving it away from the target, particularly
Revised: Furthermore, the agent needs to keep its y -position within [ − 0 . 5 , 0 . 5] in order to stay on the pathway indicated in Figure 4b and it faces an additional drift moving it away from the target, particularly

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Consequently, the policies pre-trained in the LQR do not quite reach the target but need to explore that they need to counteract this external force while staying on the pathway.
Revised: Consequently, the policies pre-trained in the LQR do not quite reach the target but need to explore that they need to counteract this external force while staying on the pathway.

Original: When learning with LSPI in the LQR system, we represent the policy by a PD-controller with explorative noise
Revised: When learning with LSPI in the LQR system, we represent the policy by a PD-controller with explorative noise

Original: [Equation]
Revised: [Equation]

Original: I ) as this ensures the quadratic form of the optimal Q -function mentioned in the main text (please see
Revised: as this ensures the quadratic form of the optimal Q -function mentioned in the main text (please see

Original: Bradtke (1993) for additional details).
Revised: Bradtke (1993) for additional details).

Original: When learning with DQN in both the LQR and target task, the agent can choose between applying 8 discrete actions
Revised: When learning with DQN in both the LQR and target task, the agent can choose between applying 8 discrete actions

Original: [Equation]
Revised: [Equation]

Original: Consequently, we predict these discrete Q -values with the function approximator learned in the LQR task with LSPI when training on the target task.
Revised: Consequently, we predict these discrete Q -values with the function approximator learned in the LQR task with LSPI when training on the target task.

Original: The Q -networks employed in DQN consist of two hidden layers of 128 neurons and ReLU activations.
Revised: The Q -networks employed in DQN consist of two hidden layers of 128 neurons and ReLU activations.

Original: During training, we realized that the update frequency of the target network in the DQN algorithm had a strong impact on the ﬁnal agent performance.
Revised: During training, we realized that the update frequency of the target network in the DQN algorithm had a strong impact on the ﬁnal agent performance.

Original: Hence, we conducted a grid search testing values in [25 , 50 , 100 , 200 , 500 , 1000 , 2000] .
Revised: Hence, we conducted a grid search testing values in [25 , 50 , 100 , 200 , 500 , 1000 , 2000] .

Original: For regular DQN on the target task, an update frequency of 200 performed best, while BC - DQN yielded best results with an update frequency of 2000 .
Revised: For regular DQN on the target task, an update frequency of 200 performed best, while BC - DQN yielded best results with an update frequency of 2000 .

Original: For C - DQN , best results where achieved when using an update frequency of 1000 in the LQR task and an update frequency of 25 in the target task.
Revised: For C - DQN , best results where achieved when using an update frequency of 1000 in the LQR task and an update frequency of 25 in the target task.

Original: We used Adam for optimization of the network with a learning rate of 10 − 4 and ( β 1 , β 2 ) =
Revised: We used Adam for optimization of the network with a learning rate of 10 − 4 and ( β 1 , β 2 ) =

Original: To exploit the previously learned policy in the target task for the curriculum methods, we used a ﬁxed value of ε = 0 .
Revised: To exploit the previously learned policy in the target task for the curriculum methods, we used a ﬁxed value of ε = 0 .

Original: 1 in the target task.
Revised: 1 in the target task.

Original: For default DQN , we annealed ε from 1 to 0 .
Revised: For default DQN , we annealed ε from 1 to 0 .

Original: 1 over the ﬁrstenvironment steps.
Revised: 1 over the ﬁrstenvironment steps.