Original: B OOSTED C URRICULUM R EINFORCEMENT L EARNING
Revised: B OOSTED C URRICULUM R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difﬁculty.
Revised: Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difﬁculty.

Original: However, ﬁnding an exact way of reusing action-values in this setting is still a poorly understood problem.
Revised: However, ﬁnding an exact way of reusing action-values in this setting is still a poorly understood problem.

Original: In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task.
Revised: In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task.

Original: This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the beneﬁt of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented.
Revised: This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the beneﬁt of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented.

Original: This procedure allows reusing previous action-values while promoting expressiveness of the action-value function.
Revised: This procedure allows reusing previous action-values while promoting expressiveness of the action-value function.

Original: We theoretically study BCRL as an approximate value iteration algorithm, proving its advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function.
Revised: We theoretically study BCRL as an approximate value iteration algorithm, discussing advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function.

Original: Finally, we provide detailed empirical evidence of the beneﬁts of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.
Revised: Finally, we provide detailed empirical evidence of the beneﬁts of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The combination of reinforcement learning (RL) (Sutton & Barto, 2018) algorithms with powerful function approximators, i.e., deep RL (Franc¸ois-Lavet et al., 2018; Mnih et al., 2015), is amajor breakthrough towards solving complex decision making and control problems that were impractical for previous shallow RL methods.
Revised: The combination of reinforcement learning (RL) (Sutton & Barto, 2018) algorithms with powerful function approximators, i.e., deep RL (Franc¸ois-Lavet et al., 2018; Mnih et al., 2015), is a breakthrough towards solving complex decision making and control problems that were impractical for previous shallow RL methods.

Original: However, the outstanding performance of deep RL techniques is obtained at the cost of a massive amount of data needed from the interaction with the environment, hindering the practicality of deep RL methods.
Revised: However, the outstanding performance of deep RL techniques is obtained at the cost of a massive amount of data needed from the interaction with the environment, hindering the practicality of deep RL methods.

Original: Curriculum RL is a biologically inspired approach that frames the problem of learning a complex target task into learning a sequence of simpliﬁed, increasingly difﬁcult, versions of it (Florensa et al., 2017;
Revised: Curriculum RL is a biologically inspired approach that frames the problem of learning a complex target task into learning a sequence of simpliﬁed, increasingly difﬁcult, versions of it (Florensa et al., 2017;

Original: Shao et al., 2018; Ivanovic et al., 2019;
Revised: Shao et al., 2018; Ivanovic et al., 2019;

Original: Narvekar et al., 2020).
Revised: Narvekar et al., 2020).

Original: Commonly, establishing the complexity of a task in RL is not trivial, as an indisputable notion of complexity is still missing.
Revised: Commonly, establishing the complexity of a task in RL is not trivial, as an indisputable notion of complexity is still missing.

Original: However, from a practical point of view, we can consider a task complex if it requires targeted exploration in the environment or complex policies to be solved.
Revised: However, from a practical point of view, we can consider a task complex if it requires targeted exploration in the environment or complex policies to be solved.

Original: In curriculum RL, the notion of task complexity is key, and it is assumed that tailored tasks in a sequence of increasing complexity are presented to the learning agent.
Revised: In curriculum RL, the notion of task complexity is key, and it is assumed that tailored tasks in a sequence of increasing complexity are presented to the learning agent.

Original: Such an appropriate design is, however, a difﬁcult problem, requiring either human expertise (Narvekar et al., 2020) or solutions to automatize the selection of tasks following the learning of the agent (Jiang et al., 2015; Svetlik et al., 2017; Klink et al., 2020).
Revised: Such an appropriate design is, however, a difﬁcult problem, requiring either human expertise (Narvekar et al., 2020) or solutions to automatize the selection of tasks following the learning of the agent (Jiang et al., 2015; Svetlik et al., 2017; Klink et al., 2020).

Original: Regardless of using a handcrafted or an automatically generated curriculum, a common approach adopted by most value-based curriculum RL methods is to use the same function approximator throughout all the tasks (Narvekar et al., 2020): given a task, an approximation of the action-value function is learned, and used as an initialization for learning the action-value function of the next task in the curriculum.
Revised: Regardless of using a handcrafted or an automatically generated curriculum, a common approach adopted by most value-based curriculum RL methods is to use the same function approximator throughout all the tasks (Narvekar et al., 2020): given a task, an approximation of the action-value function is learned, and used as an initialization for learning the action-value function of the next task in the curriculum.

Original: For the success of this procedure, the function approximator needs to be powerful enough to handle the complexity of the target task.
Revised: For the success of this procedure, the function approximator needs to be powerful enough to handle the complexity of the target task.

Original: However, this requirement raises the key issue of adequately designing the function approximator.
Revised: However, this requirement raises the key issue of adequately designing the function approximator.

Original: A complex function approximator is prone to overﬁtting, and sensitive to hyper-parameters, being detrimental for learning simple tasks in the curriculum; conversely, using an overly simple function approximator can hinder the learning of the target task.
Revised: A complex function approximator is prone to overﬁtting, and sensitive to hyper-parameters, being detrimental for learning simple tasks in the curriculum; conversely, using an overly simple function approximator can hinder the learning of the target task.

Original: Moreover, choosing an adequate function approximator a-priori is not trivial, as the target task is only visited at the end, and it is commonly unsolvable by non-curriculum RL methods.
Revised: Moreover, choosing an adequate function approximator a priori is not trivial, as the target task is only visited at the end, and it is commonly unsolvable by non-curriculum RL methods.

Original: In this paper, we introduce a novel approach that models the action-value function of a task as a sum of residuals learned on the previous tasks in the curriculum.
Revised: In this paper, we introduce a novel approach that models the action-value function of a task as a sum of residuals learned on the previous tasks in the curriculum.

Original: Our method increases the representativeness of the function approximator as new tasks are presented to the agent, leading to a procedure that increases the size of the functional space as the curriculum proceeds.
Revised: Our method increases the representativeness of the function approximator as new tasks are presented to the agent, leading to a procedure that increases the size of the functional space as the curriculum proceeds.

Original: This results in a learning procedure that increases the power of the function approximator according to the complexity of the task at hand, using small models in simple problems to avoid overﬁtting, and using large models (resulting from the sum of small ones) to handle the dimensionality of complex problems.
Revised: This results in a learning procedure that increases the power of the function approximator according to the complexity of the task at hand, using small models in simple problems to avoid overﬁtting, and using large models (resulting from the sum of small ones) to handle the dimensionality of complex problems.

Original: Notably, if the curriculum is sufﬁciently ﬁne-grained, the residuals become smoother functions, close to zero, that are easy to learn, even if the respective tasks are harder than the previous ones.
Revised: Notably, if the curriculum is sufﬁciently ﬁne-grained, the residuals become smoother functions, close to zero, that are easy to learn, even if the respective tasks are harder than the previous ones.

Original: We call our method boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting technique in supervised learning (Freund, 1995).
Revised: We call our method boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting technique in supervised learning (Freund, 1995).

Original: We provide a theoretical analysis of our approach under the lens of the approximate value iteration (AVI) framework (Farahmand, 2011).
Revised: We provide a theoretical analysis of our approach under the lens of the approximate value iteration (AVI) framework (Farahmand, 2011).

Original: In AVI, an estimate of the optimal action-value function is obtained as an iterative process that starts from an arbitrary estimate and applies the optimal Bellman operator until convergence.
Revised: In AVI, an estimate of the optimal action-value function is obtained as an iterative process that starts from an arbitrary estimate and applies the optimal Bellman operator until convergence.

Original: It is shown that two sources of error exist in AVI: (i) computation of the optimal Bellman operator, (ii) representation of the action-value function (Farahmand, 2011).
Revised: It is shown that two sources of error exist in AVI: (i) computation of the optimal Bellman operator, (ii) representation of the action-value function (Farahmand, 2011).

Original: While the former is due to the need of using samples to approximate the unknown optimal Bellman operator, the latter depends on the representativeness of the functional space chosen to approximate the action-value function.
Revised: While the former is due to the need of using samples to approximate the unknown optimal Bellman operator, the latter depends on the representativeness of the functional space chosen to approximate the action-value function.

Original: In this work, we formalize the curriculum RL problem in the AVI setting to show that the representation of the action-value function obtained by BCRL results in a tighter bound on the approximation error compared to regular curriculum RL, consequently reducing the performance loss.
Revised: In this work, we formalize the curriculum RL problem in the AVI setting to investigate the representation of the action-value function obtained by BCRL and cases in which this representation may result in a tighter bound on the approximation error compared to regular curriculum RL.

Original: We complement this analysis with an empirical evaluation of BCRL in AVI and RL settings, resorting to the ﬁtted Q -iteration algorithm
Revised: We complement this analysis with an empirical evaluation of BCRL in AVI and RL settings, resorting to the ﬁtted Q -iteration algorithm

Original: (Ernst et al., 2005), least-squares policy iteration (Lagoudakis & Parr, 2003) and deep Q -networks (Mnih et al., 2015), conﬁrming the advantages of BCRL over regular curriculum RL.
Revised: (Ernst et al., 2005), least-squares policy iteration (Lagoudakis & Parr, 2003), and deep Q -networks (Mnih et al., 2015), demonstrating advantages of BCRL over regular curriculum RL.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Learning under a dynamic set of training experience has been investigated in different ﬁelds throughout decades (Saul, 1941; Elman, 1993; Asada et al., 1996; Krueger & Dayan, 2009).
Revised: Learning under a dynamic set of training experience has been investigated in different ﬁelds throughout decades (Saul, 1941; Elman, 1993; Asada et al., 1996; Krueger & Dayan, 2009).

Original: A recent paper by Bengio et al.
Revised: A recent paper by Bengio et al.

Original: (2009) has established the term curriculum learning for the concept of organizing training data of a neural network in a beneﬁcial way for performance and generalization.
Revised: (2009) has established the term curriculum learning for the concept of organizing training data of a neural network in a beneﬁcial way for performance and generalization.

Original: This term carried over to the domain of RL, where curricula over training tasks have been shown to stabilize the highly challenging optimization problem of RL (Narvekar et al., 2020).
Revised: This term carried over to the domain of RL, where curricula over training tasks have been shown to stabilize the highly challenging optimization problem of RL (Narvekar et al., 2020).

Original: While a large body of research in the domain of curriculum reinforcement learning investigates the interesting problem of optimally selecting and scheduling the training tasks for an RL agent to optimize performance on a set of target tasks (Florensa et al., 2017; Ivanovic et al., 2019;
Revised: While a large body of research in the domain of curriculum reinforcement learning investigates the interesting problem of optimally selecting and scheduling the training tasks for an RL agent to optimize performance on a set of target tasks (Florensa et al., 2017; Ivanovic et al., 2019;

Original: Shao et al., 2018; Klink et al., 2020), the architecture of the learning agent is often assumed ﬁxed and the transfer of agent behavior between subsequent tasks in the curriculum is realized by this shared ﬁxed architecture.
Revised: Shao et al., 2018; Klink et al., 2020), the architecture of the learning agent is often assumed ﬁxed and the transfer of agent behavior between subsequent tasks in the curriculum is done by this shared ﬁxed architecture.

Original: However, aside from investigations into the beneﬁt of an adequately chosen evolution of training experience, investigations of an optimal learning architecture (Elsken et al., 2019) and the evolutionof this architecture during learning (Aoki & Siekevitz, 1988; Rumelhart et al., 1988; Ash, 1989; Fahlman, 1990; Lahnaj ¨ arvi et al., 2002; Nitanda & Suzuki, 2018) have been conducted in supervised learning scenarios.
Revised: However, aside from investigations into the beneﬁt of an adequately chosen evolution of training experience, investigations of an optimal learning architecture (Elsken et al., 2019) and its evolution during learning (Aoki & Siekevitz, 1988; Rumelhart et al., 1988; Ash, 1989; Fahlman, 1990; Lahnaj ¨ arvi et al., 2002; Nitanda & Suzuki, 2018) have been conducted in supervised learning scenarios.

Original: The adaptation of these methods to the setting of RL has been limited to on-policy temporal-difference learning (Rivest & Precup, 2003; Vamplew & Ollington, 2005b;a) and it has not been further pursued in recent years.
Revised: The adaptation of these methods to the setting of RL has been limited to on-policy temporal-difference learning (Rivest & Precup, 2003; Vamplew & Ollington, 2005b;a) and it has not been further pursued in recent years.

Revised: We shortly review further approaches to knowledge transfer in Appendix A.

Original: Our work aims to improve the information transfer between subsequent tasks in curriculum value-based reinforcement learning via a boosting procedure (Freund, 1995).
Revised: Our work aims to improve the information transfer between subsequent tasks in curriculum valuebased reinforcement learning via a boosting procedure (Freund, 1995).

Original: The beneﬁt of the boosting framework is that it is well understood from a theoretical perspective (Tosatto et al., 2017), allowing for the derivation of error bounds, as also presented in this paper.
Revised: The beneﬁt of the boosting framework is that it is well understood from a theoretical perspective (Tosatto et al., 2017), allowing for the derivation of error bounds, as also presented in this paper.

Original: Further, it is agnostic to any value-based RL algorithm, avoiding the need to adjust the RL algorithm of choice to be suited for the evolution of the function approximator (Rivest & Precup, 2003; Vamplew & Ollington, 2005a;b).
Revised: Further, it is agnostic to any valuebased RL algorithm, avoiding the need to adjust the RL algorithm of choice to be suited for the evolution of the function approximator (Rivest & Precup, 2003; Vamplew & Ollington, 2005a;b).

Original: Aside from improving knowledge transfer via the learning architecture of the agent, the ﬁeld of transfer learning investigates various approaches for transferring knowledge, which we shortly review in Appendix A.

Original:  P RELIMINARIES 3. D EFINITIONS AND NOTATION
Revised:  P RELIMINARIES 3. D EFINITIONS AND NOTATION

Original: We recall the notions about Markov Decision Processes (MDPs) (Puterman, 1990), and the notation used for approximate value iteration (AVI) (Farahmand, 2011), that are relevant to our work.
Revised: We recall the notions about Markov Decision Processes (MDPs) (Puterman, 1990), and the notation used for approximate value iteration (AVI) (Farahmand, 2011), that are relevant to our work.

Original: For a space Σ , we deﬁne M (Σ)  the set of probability measures over the σ -algebra σ Σ .
Revised: For a space Σ , we deﬁne M (Σ) as the set of probability measures over the σ -algebra σ Σ .

Original: We denote B (Σ , B ) as the space of bounded measurable functions w.r.t.
Revised: We denote B (Σ , B ) as the space of bounded measurable functions w.r.t.

Original: σ Σ with bound 0 <B< ∞ .
Revised: σ Σ with bound 0 <B< ∞ .

Original: A ﬁniteaction discounted MDP is deﬁned as a tuple T
Revised: A ﬁnite-action discounted MDP is deﬁned as a tuple T

Original: = (cid:104)S , A , P , R , γ (cid:105) , where S is a measurable state space,
Revised: = (cid:104)S , A , P , R , γ (cid:105) , where S is a measurable state space,

Original: A is a ﬁnite set of actions, P : S ×
Revised: A is a ﬁnite set of actions, P : S ×

Original: A → M ( S ) is a transition kernel, R : S ×
Revised: A → M ( S ) is a transition kernel, R : S ×

Original: A × S → R is a reward function, and γ ∈ [0 , 1) is a discount factor.
Revised: A × S → R is a reward function, and γ ∈ [0 , 1) is a discount factor.

Original: A policy π is a mapping from the state space S to a probability distribution over the action space A .
Revised: A policy π is a mapping from the state space S to a probability distribution over the action space A .

Original: A policy induces an action-value function Q π ( s, a ) (cid:44) E [Σ ∞ t =

Revised: A policy induces an action-value function Q π ( s, a ) (cid:44) E [Σ ∞ t =

Original: γ t R t | S 

Revised: γ t R t | S

Original: = s,
Revised: = s,

Original: A 

Revised: A

Original: = a ] , corresponding to the expected cumulative discounted reward obtained performing action a in state s and following the policy π thereafter.
Revised: = a ] , corresponding to the expected cumulative discounted reward obtained performing action a in state s and following the policy π thereafter.

Original: RL aims at solving an MDP by ﬁnding an optimal policy π ∗ which induces an optimal action-value function Q ∗ ( s, a ) =
Revised: RL aims at solving an MDP by ﬁnding an optimal policy π ∗ which induces an optimal action-value function Q ∗ ( s, a ) =

Original: sup π Q π ( s, a ) .
Revised: sup π Q π ( s, a ) .

Original: For immediate rewards uniformly bounded by R max , the action-value function is bounded by Q max
Revised: For immediate rewards uniformly bounded by R max , the action-value function is bounded by Q max

Original: = R max / 1 − γ .
Revised: = R max / 1 − γ .

Original: For a probability measure µ ∈M ( S × A ) , we deﬁne the
Revised: For a probability measure µ ∈

Original:  L p ( µ ) -norm of a measurable function Q ∈ B ( S × A ) as (cid:107) Q (cid:107) p,µ (cid:44) [ (cid:82) S×A | Q ( s, a ) | p dµ ( s, a )] 1 / p .
Revised: M ( S × A ) , we deﬁne the L p ( µ ) -norm of a measurable function Q ∈ B ( S × A ) as (cid:107) Q (cid:107) p,µ (cid:44) [ (cid:82) S×A | Q ( s, a ) | p dµ ( s, a )] 1 / p .

Original: Given a sequence of values x 1: n =( x 1 , . . .
Revised: Given a sequence of values x 1: n =( x 1 , . . .

Original: , x n ) for some space X , the empirical L p -norm of a function f : X → R is (cid:107) f (cid:107) p,x 1: n (cid:44)  1 n (cid:80) ni
Revised: , x n ) for some space X , the empirical L p -norm of a function f : X → R is (cid:107) f (cid:107) p,x 1: n (cid:44) (cid:2) 1 n (cid:80) ni

Original: =1 | f ( x i ) | p  .
Revised: =1 | f ( x i ) | p (cid:3) 1 / p .

Original: Conveniently, when X i ∼ µ , we have E [ (cid:107) f (cid:107) p,x 1: n ] = (cid:107) f (cid:107) p,µ .
Revised: Conveniently, when X i ∼ µ , we have E [ (cid:107) f (cid:107) p,x 1: n ] = (cid:107) f (cid:107) p,µ .

Original: From now on, we consider the L 2 -norm when the subscript p is omitted.
Revised: From now on, we consider the L 2 -norm when the subscript p is omitted.

Original: Operators.
Revised: Operators.

Original: We deﬁne the operator P π as ( P π Q )( s, a ) (cid:44) (cid:82) S P ( ds (cid:48) | s, a ) (cid:80) a (cid:48) ∈A π ( a (cid:48) | s (cid:48) ) Q ( s (cid:48) , a (cid:48) ) , for any given policy π . Then, the Bellman operator T π : B ( S × A )
Revised: We deﬁne the optimal Bellman operator T ∗ : B ( S × A )

Original: → B ( S × A ) is ( T π Q )( s, a ) (cid:44) r ( s, a )
Revised: → B ( S × A ) as

Original: + γ ( P π Q )( s, a ) . The optimal Bellman operator T ∗ : B ( S × A ) → B ( S × A ) is ( T ∗ Q )( s, a ) (cid:44) r ( s, a ) + γ (cid:82) S max a (cid:48) P ( ds (cid:48) | s, a ) Q ( s (cid:48) , a (cid:48) )) .
Revised: [Equation]

Original: Finally, the ﬁxed point of the  Bellman operator is ( T π Q π ) =
Revised: The ﬁxed point of the optimal Bellman operator is ( T ∗ Q ∗ ) =

Original: Q π , and the ﬁxed point of the optimal Bellman operator is ( T ∗ Q ∗ ) =

Original: Q ∗ .
Revised: Q ∗ .

Original: As in Gy ¨ orﬁ et al.
Revised: As in Gy ¨ orﬁ et al.

Original: (2006), we deﬁne the truncation operator β B : B ( S × A )
Revised: (2006), we deﬁne the truncation operator β B : B ( S × A )

Original: → B ( S × A , B ) such that, given a function f ∈ B ( S × A ) , we have β B f ( s, a ) ∈ [ − B, B ] , ∀ ( s, a ) ∈ S × A .
Revised: → B ( S × A , B ) such that, given a function f ∈ B ( S × A ) , we have β B f ( s, a ) ∈ [ − B, B ] , ∀ ( s, a ) ∈ S × A .

Original: 3. A PPROXIMATE VALUE ITERATION
Revised: 3. A PPROXIMATE VALUE ITERATION

Original: The family of approximate value iteration (AVI) algorithms (Sutton & Barto, 2018; Farahmand, 2011; Munos, 2005) aims at approximating the solution of RL problems with large state space where ﬁnding an optimal solution is impractical or impossible.
Revised: The family of approximate value iteration (AVI) algorithms (Sutton & Barto, 2018; Farahmand, 2011; Munos, 2005) aims at approximating the solution of RL problems with large state space, where ﬁnding an optimal solution is impractical or impossible.

Original: Starting from an arbitrary estimate
Revised: Starting from an arbitrary estimate

Original: Q 0 , AVI algorithms iteratively approximate the application of the optimal Bellman operator in a suitable functional space, i.e., Q k +1 ≈ T ∗ Q k .
Revised: Q 0 , AVI algorithms iteratively approximate the application of the optimal Bellman operator in a suitable function space, i.e., Q k +1 ≈ T ∗ Q k .

Original: Intuitively, the approximation error at each iteration plays a central role in terms of convergence to the ﬁxed point Q ∗ .
Revised: Intuitively, the approximation error at each iteration plays a central role in terms of convergence to the ﬁxed point Q ∗ .

Original: The approximation error results from two key aspects: (i) the unknown optimal Bellman operator, and (ii) the representation of the action-value function in a suitable functional space F via the application of a nonlinear operator
Revised: The approximation error results from two key aspects: (i) the unknown optimal Bellman operator, and (ii) the representation of the action-value function in a suitable function space F via the application of a nonlinear operator

Original: S : B ( S × A ) → F deﬁned as Sy = arg inf f ∈F (cid:107) f − y (cid:107) 2 µ for any y ∈ B ( S × A ) .
Revised: S : B ( S × A ) → F deﬁned as Sy = arg inf f ∈F (cid:107) f − y (cid:107) 2 µ for any y ∈ B ( S × A ) .

Original: Then, we deﬁne
Revised: Then, we deﬁne

Original: Q k +1 =
Revised: Q k +1 =

Original: S ( T ∗ Q k ) , and the resulting approximation error by
Revised: S ( T ∗ Q k ) , and the resulting approximation error by

Original: [Equation]
Revised: [Equation]

Original: The role of the approximation error in AVI is evinced by the following theorem that bounds the performance loss, w.r.t.
Revised: The role of the approximation error in AVI is evinced by the following theorem that bounds the performance loss, w.r.t.

Original: the optimal performance, at iteration K 1 .
Revised: the optimal performance, at iteration K 1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: (Theorem 3.4 of Farahmand (2011)).
Revised: (Theorem 3.4 of Farahmand (2011)).

Original: Let K be a positive integer, ρ an initial state distribution, and Q max ≤ R max 1 − γ .
Revised: Let K be a positive integer, ρ a distribution over states s , and Q max ≤ R max 1 − γ .

Original: Then, for any sequence ( Q k ) Kk =

Revised: Then, for any sequence ( Q k ) Kk =

Original: ⊂ B ( S × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 , we have
Revised: ⊂ B ( S × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 , we have

Original: [Equation]
Revised: [Equation]

Original: where E ( ε 0 , . .
Revised: where E ( ε 0 , . .

Original: . , ε K − 1 ) =
Revised: . , ε K − 1 ) =

Original: (cid:80) K − 1 k =

Revised: (cid:80) K − 1 k =

Original: C ( k ) (cid:107) ε k (cid:107) 22 ,µ .
Revised: C ( k ) (cid:107) ε k (cid:107) 2 µ .

Original: We point out that the upper bound in Theorem 1 is a high-level version of the one reported in Farahmand (2011), where we omitted some technicalities that go beyond the scope of this paper replacing them with functions G , A , B , and C .
Revised: We point out that the upper bound in Theorem 1 is a high-level version of the one reported in Farahmand (2011), where we omitted some technicalities that go beyond the scope of this paper replacing them with functions G , A , B , and C .

Original: This simpliﬁed version highlights that the magnitude of the difference between the optimal action-value function Q ∗ and the estimate at the K -th iteration Q K , is proportional to the magnitude of the approximation errors ε k as deﬁned in Equation 1.
Revised: This simpliﬁed version highlights that the magnitude of the difference between the optimal action-value function Q ∗ and the estimate at the K -th iteration Q K , is proportional to the magnitude of the approximation errors ε k as deﬁned in Equation 1.

Original: We refer the reader to the appendix for an extended version of the equation, and to Farahmand (2011) for further details about it.
Revised: We refer the reader to the appendix for an extended version of the equation, and to Farahmand (2011) for further details about it.

Original:  C URRICULUM REINFORCEMENT LEARNING WITH B ELLMAN RESIDUALS
Revised:  C URRICULUM REINFORCEMENT LEARNING WITH B ELLMAN RESIDUALS

Original: We consider the classic curriculum RL scenario where an agent aims at learning a complex target task by solving a predeﬁned sequence of easier, but increasingly challenging, versions of it (Narvekar et al., 2020).
Revised: We consider the classic curriculum RL scenario where an agent aims at learning a complex target task by solving a predeﬁned sequence of easier, but increasingly challenging, versions of it (Narvekar et al., 2020).

Original: To avoid confusion on the debatable notion of difﬁculty of a task, in this work we adopt a practical point of view where the difﬁculty of a task is related to well-known aspects of RL, e.g., the complexity of the policies for solving them and challenging exploration (Florensa et al., 2017; Ivanovic et al., 2019; Klink et al., 2020).
Revised: To avoid confusion on the debatable notion of difﬁculty of a task, in this work we adopt a practical point of view where the difﬁculty of a task is related to well-known aspects of RL, e.g., the complexity of the policies for solving them and challenging exploration (Florensa et al., 2017; Ivanovic et al., 2019; Klink et al., 2020).

Original: 4. A PPROXIMATE VALUE ITERATION IN CURRICULUM REINFORCEMENT LEARNING
Revised: 4. A PPROXIMATE VALUE ITERATION IN CURRICULUM REINFORCEMENT LEARNING

Original: Let T 1: n = {T 1 , . .
Revised: Let T 1: n = {T 1 , . .

Original: . , T n } be a sequence of increasingly complex MDPs sharing the same state and action spaces, i.e., T i
Revised: . , T n } be a sequence of increasingly complex MDPs sharing the same state and action spaces, i.e., T i

Original: =
Revised: =

Original: (cid:104)S , A , P i , R i , γ (cid:105) , and T n be the target task.
Revised: (cid:104)S , A , P i , R i , γ (cid:105) , and T n be the target task.

Original: The use of AVI algorithms in this curriculum RL setting consists in starting from an arbitrary estimate Q 01 (where the subscript indicates the position of the task in the curriculum) and approximating the application of the taskspeciﬁc optimal Bellman operator for the current task Q k +1 t ≈ T ∗ t Q kt .
Revised: The use of AVI algorithms in this curriculum RL setting consists in starting from an arbitrary estimate Q 01 (where the subscript indicates the position of the task in the curriculum) and approximating the application of the taskspeciﬁc optimal Bellman operator for the current task Q k +1 t ≈ T ∗ t Q kt .

Original: We assume that the task changes when the estimate of the action-value of the current task converges.
Revised: We assume that the task changes when the estimate of the action-value of the current task converges.

Original: For simplicity, and without loss of generality, we denote K t the number of iterations needed to converge for each task
Revised: For simplicity, and without loss of generality, we denote K t the number of iterations needed to converge for each task

Original: T t , and denote Q 0 t =
Revised: T t , and denote Q 0 t =

Original: Q K t − 1 t − 1 =
Revised: Q K t − 1 t − 1 =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let K be the current iteration.
Revised: Let K be the current iteration.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: This result can be veriﬁed applying the triangle inequality under the assumption that the algorithm switches task when the stopping condition of the algorithm is veriﬁed, e.g., when reaching a chosen number of iterations K t that allows the action-value function estimate to converge, or using heuristic methods checking the magnitude of the update (Lagoudakis & Parr, 2003; Ernst et al., 2005).
Revised: This result can be veriﬁed applying the triangle inequality under the assumption that the algorithm switches task when the stopping condition of the algorithm is veriﬁed, e.g., when reaching a chosen number of iterations K t that allows the action-value function estimate to converge, or using heuristic methods checking the magnitude of the update (Lagoudakis & Parr, 2003; Ernst et al., 2005).

Original: Note that each component of the sum in (2) can be bounded by the result in Theorem 1.
Revised: Since zero is not a positive integer, the components in (2) cannot be bounded by the result in Theorem 1, which motivates our theoretical analysis in section 5.

Original: We denote the  k -step Bellman residual for task t by
Revised: We denote the analytic k -step Bellman residual for task t by

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Given an initial Q 01 ( s, a )=0 for any ( s, a ) ∈ S × A , and considering that
Revised: Given an initial Q 01 ( s, a )=0 for any ( s, a ) ∈ S × A , and considering that

Original: Q ∗ t = Q K t t
Revised: Q ∗ t = Q K t t

Original: ≈ ( T ∗ t ) K t Q ∗ t − 1 , we obtain(cid:37) kn + n − 1 (cid:88)
Revised: ≈ ( T ∗ t ) K t Q ∗ t − 1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: This result shows that, approximately, the action-value function of the target task is  equivalent to the sum of k -step Bellman residuals.
Revised: This result shows that, under above assumptions, the action-value function of the target task is approximately equivalent to the sum of k -step Bellman residuals.

Original: Notably, especially if the curriculum is sufﬁciently ﬁne-grained, residuals becomes progressively smoother and smaller.
Revised: Notably, especially if the curriculum is sufﬁciently ﬁne-grained, residuals are expected to become progressively smoother and smaller.

Original: In the following, we describe how to compute the estimate  shown in (4).
Revised: In the following, we describe how to compute an estimate of the k -step residuals shown in (4).

Original: 4. B OOSTED CURRICULUM REINFORCEMENT LEARNING
Revised: 4. B OOSTED CURRICULUM REINFORCEMENT LEARNING

Original: It is well-known from supervised learning that: (i) small function approximators are easier to train, and thus are desirable to solve simple tasks; (ii) large function approximators can overﬁt or suffer slow training speed in simple tasks, but they are more suitable to handle complex problems.
Revised: It is well-known from supervised learning that: (i) small function approximators are easier to train, and thus are desirable to solve simple tasks; (ii) large function approximators can overﬁt or suffer slow training speed in simple tasks, but they are more suitable to handle complex problems.

Original: Ideally, we want to increase the representativeness of the function approximator with the increase of task complexity.
Revised: Ideally, we want to increase the representativeness of the function approximator with the increase of task complexity.

Original: Recalling (4), we propose to use a separate function approximator for each residual  (cid:37) t , i.e. a separate approximator for each task.
Revised: Recalling (4), we propose to use a separate function approximator for each residual ¯ (cid:37) kt , i.e. a separate approximator for each task.

Original: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni =1 for each task, we denote the empirical Bellman operator by ( ˆ T ∗ t Q )( s i t , a i t ) (cid:44) r
Revised: Given a dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni =1 for each task, we denote the empirical Bellman operator by ( ˆ T ∗ t Q )( s i t , a i t ) (cid:44) r

Original: i t +
Revised: i t +

Original: γ max a (cid:48) Q ( s (cid:48)
Revised: γ max a (cid:48) Q ( s (cid:48)

Original: i t , a (cid:48) ) , and deﬁne the k -step empirical residual as ˜ (cid:37) kt (cid:44)( ˆ T ∗ t) k Q 0 t − Q 0 t .
Revised: i t , a (cid:48) ) , and deﬁne the k -step empirical residual as ˜ (cid:37) kt (cid:44) ˆ T ∗ t Q k − 1 t − Q 0 t .

Original: At iteration k , for a current task t , and deﬁning the nonlinear operator ˆ S ˜ (cid:37) kt = arg inf f ∈F (cid:107) f − ˜ (cid:37) kt (cid:107) 2 D t as the empirical version of the operator S , we compute an approximation
Revised: At iteration k , for a current task t , and deﬁning the nonlinear operator ˆ S ˜ (cid:37) kt = arg inf f ∈F (cid:107) f − ˜ (cid:37) kt (cid:107) 2 D t as the empirical version of the operator S , we compute an approximation

Original: [Equation]
Revised: [Equation]

Original: where we deﬁned ˆ (cid:37) kt = β B k ˆ S ˜ (cid:37) kt for compactness.
Revised: where we deﬁned ˆ (cid:37) kt = β B k ˆ S ˜ (cid:37) kt for compactness.

Original: For ease of notation, we use Q kt also to denote the Q -function obtained by applying the empirical Bellman operator.
Revised: For ease of notation, we use Q kt also to denote the Q -function obtained by applying the empirical Bellman operator.

Original: We call this approach boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting procedure in supervised learning (Schapire, 1990; Freund, 1995), and we show its pseudocode in Algorithm 1.
Revised: We call this approach boosted curriculum reinforcement learning ( BCRL ), for its resemblance with the boosting procedure in supervised learning (Schapire, 1990; Freund, 1995), and we show its pseudocode in Algorithm 1.

Original: Remarks.
Revised: Remarks.

Original: Our approach is based on the key insight that the strength of the function approximator should increase according to the complexity of the task at hand.
Revised: Our approach is based on the key insight that the strength of the function approximator should increase according to the complexity of the task at hand.

Original: In BCRL , we accomplish this  estimating an action-value function through a sum of action-value functions of the previous tasks in the curriculum.
Revised: In BCRL , we realize this increase in capacity by estimating an action-value function through a sum of action-value residuals of the previous tasks in the curriculum.

Original: Thus, contrary to methods using the same functional space for all tasks, our approach has the advantage of acting in an augmented functional space resulting from the sum of the functional space of each approximator.
Revised: Thus, contrary to methods using the same function space for all tasks, our approach has the advantage of acting in an augmented function space resulting from the sum of the function space of each approximator.

Original: In supervised learning literature, the function approximators in the sum are commonly known as weak regressors, meaning that boosting has the advantage of estimating complex functions as a sum of simple regressors (Geurts et al., 2006).
Revised: In supervised learning literature, the function approximators in the sum are commonly known as weak regressors, meaning that boosting has the advantage of estimating complex functions as a sum of simple regressors (Geurts et al., 2006).

Original: Similarly, our
Revised: Similarly, our

Original: BCRL procedure can use simple function approximators for each task in the curriculum, with the advantages that (i) easy tasks can be learned with easy function approximators curbing the risk of overﬁtting, (ii) the complexity of the function approximator can be chosen according to the ﬁrst tasks in the curriculum, contrarily to regular curriculum RL where the function approximator needs to be suitable for the target task, and (iii) the training time to ﬁt the model on each task is reduced due to the smoothness of the residuals.
Revised: BCRL procedure can use simple function approximators for each task in the curriculum, with the advantages that (i) easy tasks can be learned with easy function approximators curbing the risk of overﬁtting, (ii) the complexity of the function approximator can be chosen according to the individual tasks in the curriculum, contrarily to regular curriculum RL where the function approximator needs to be suitable for the target task.

Original: Conversely, at inference time, our method requires a forward pass for each approximator in the ensemble.
Revised: Conversely, at inference time, our method requires a forward pass for each approximator in the ensemble.

Original: Nevertheless, the small size of the regressors results in negligible increase of computation time.

Original: Finally, we clarify that the soundness of this procedure is guaranteed by the fact that if the functional space of each weak regressor is Glivenko-Cantelli, i.e., ”the error due to the empirical process goes to zero at least asymptotically”, then, due to the preservation theorem, also the space resulting from the sum of weak regressors is Glivenko-Cantelli (Van
Revised: Finally, we clarify that the soundness of this procedure is guaranteed by the fact that if the functional space of each weak regressor is Glivenko-Cantelli, i.e., ”the error due to the empirical process goes to zero at least asymptotically”, then, due to the preservation theorem, also the space resulting from the sum of weak regressors is Glivenko-Cantelli (Van

Original: Der Vaart & Wellner, 2000; Tosatto et al., 2017).
Revised: Der Vaart & Wellner, 2000; Tosatto et al., 2017).

Original:  T HEORETICAL ANALYSIS
Revised:  T HEORETICAL ANALYSIS

Original: We provide a theoretical analysis of BCRL in terms of convergence guarantees and ﬁnite-sample approximation error analysis.
Revised: We provide a theoretical analysis of BCRL in terms of convergence guarantees and ﬁnite-sample approximation error analysis.

Original: We assume the tasks in the curriculum have the same transition kernels, and different reward functions.

Original: Nevertheless, our results can be straightforwardly extended to the general case.

Original: We refer to the well-known results in AVI (Farahmand, 2011; Farahmand et al., 2010), and the analysis provided in Tosatto et al.
Revised: We refer to the well-known results in AVI (Farahmand, 2011; Farahmand et al., 2010), and the analysis provided in Tosatto et al.

Original: (2017), with which our analysis share some similarities.
Revised: (2017), with which our analysis share some similarities.

Original: 5. C ONVERGENCE GUARANTEES
Revised: 5. C ONVERGENCE GUARANTEES

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q i t ) k
Revised: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q it ) ki

Original: i =0 be a sequence of measurable action-value functions for atarget task T t obtained following our BCRL procedure (5), where Q 0 t =
Revised: =0 be a sequence of measurable action-value functions for a task T t obtained following our BCRL procedure (5), where Q 0 t =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Denote L= γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.
Revised: Denote L the Lipschitz coefﬁcient of the optimal Bellman operator w.r.t.

Original: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )
Revised: a norm (cid:107)·(cid:107) , and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )

Original: y (cid:107)µ ≤ λ (cid:107) y
Revised: y (cid:107) ≤ λ (cid:107) y

Original: (cid:107)µ for all y ∈ B ( S × A ) .
Revised: (cid:107) for all y ∈ B ( S × A ) .

Original: Then, 
Revised: Then, (cid:107) Q kt − Q ∗ t (cid:107) ≤ (cid:32) λ k − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: The assumption on the operator S stated by Theorem 2 dictates that the approximation power of the used functional space should be good enough to learn the target function y ; speciﬁcally, the upper bound of the approximation error is proportional to the magnitude of the function to approximate.
Revised: The assumption on the operator S stated by Theorem 2 dictates that the approximation power of the used functional space should be good enough to learn the target function y ; speciﬁcally, the upper bound of the approximation error is proportional to the magnitude of the function to approximate.

Original: This assumption holds for reasonable choices of function approximators and it is present in related works on the derivation of ﬁnite-time bounds for policy iteration (Munos, 2003), and value iteration (Munos, 2005; Tosatto et al., 2017).
Revised: This assumption holds for reasonable choices of function approximators and it is present in related works on the derivation of ﬁnite-time bounds for policy iteration (Munos, 2003), and value iteration (Munos, 2005; Tosatto et al., 2017).

Original: We observe that the boosting procedure adds the (1+ L k ) λ term in the regular AVI bound, that is obtained for λ = 0 .

Original: This result shows that for each task, for appropriate values of λ and L ,  the boosting procedure converges to the optimal action-value.
Revised: The following result shows that for each task, the interplay between L , λ and Q 0 t decides upon the approximation error w.r.t. Q ∗ t .

Original: Corollary 1.
Revised: Corollary 1.

Original: Given the settings of Theorem 2, the sequence of ( Q it ) ki =0 converges to Q ∗ t when
Revised: Given the settings of Theorem 2, if L < 1 and λ < (cid:15) (1 − L ) 1+ L(cid:15) with (cid:15) ∈ [0 , 1] then

Original: [Equation]
Revised: [Equation]

Revised: If now Q 0 t ≈ Q ∗ t − 1 is close to Q ∗ t in L p ( µ ) -norm, we can hope to achieve a small error in task T t .

Original: 5. F INITE - SAMPLE APPROXIMATION ERROR ANALYSIS
Revised: 5. F INITE - SAMPLE APPROXIMATION ERROR ANALYSIS

Original: In Lemma 1, we show how to derive the AVI performance loss analysis(Farahmand, 2011) for curriculum RL. Now, we want to characterize it for our BCRL method. Instead of using the regular AVI approximation error deﬁned in (1), we consider the approximation error foreach task t in the curriculum deﬁned as
Revised: Given the inapplicability of Theorem 1 to the decomposition in Lemma 1, we now provide an alternative analysis for our BCRL method, in which the approximation errors for task t translate to

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Revised: (Theorem 5.3 of Farahmand (2011))

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value functions for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.

Original: Denoting the regular AVI residual as ¯ (cid:37) kt = ( T ∗ t ) k . . . ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t − 1 ) K t − 1 . . . ( T ∗ 1 ) K 1 Q 01 , we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: This bound shows the relation between the approximation error of the residual for task t at step k , the AVI Bellman Residual ¯ (cid:37) kt , and the approximation error for the residuals of the  previous  tasks.
Revised: This bound shows the relation between the approximation error of the residual for task t at step k , the AVI Bellman Residual ¯ (cid:37) kt , and the approximation error for previous residuals in task t . While the error is independent of previous tasks, the next Lemma shows a connection between tasks.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value functions for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: The above bound is based on the approximation errors ε it in task T t as well as on the one-step bellman residual (cid:37) 1 t =

Revised: T ∗ t Q 0 t − Q 0 t .

Revised: Intuitively, this residual will be small, if Q 0 t =

Revised: Q ∗ t − 1 is a good approximation to Q ∗ t .

Original: We use (9) and (10) to ﬁnally bound the approximation error of the residual as follows.
Revised: We use (9) and (10) to ﬁnally bound the approximation error of the residual.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let B tk = max( (cid:107) ˜ (cid:37) kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , i.e., the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).
Revised: Let B kt = max( (cid:107) ˜ (cid:37) kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , i.e., the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).

Original: Forgiven dataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni =1 for each task t , and a given sequence of action-value functions ( Q it ) ki =0 generated by BCRL , we have
Revised: For a given sequence of action-value functions ( Q it ) ki =0 generated by BCRL , we have

Original: [Equation]
Revised: [Equation]

Original: This result bounds the  error of BCRL by a sum of three components: (i) the approximation error of the  residual ¯ (cid:37) k t , (ii) the propagation error due to approximation errors at previous iterations, (iii) the estimation error due to the regression problem with  ﬁnite amount of samples.
Revised: This result bounds the expected error of BCRL by a sum of three components: (i) the approximation error of the analytic residual ¯ (cid:37) kt , (ii) the propagation error due to approximation errors at previous iterations, (iii) the estimation error due to the regression problem with a ﬁnite amount of samples.

Original: The advantages of BCRL are motivated by the use of  residuals ¯ (cid:37) kt and ˜ (cid:37) kt, whose range of values become smaller as the curriculum proceeds, resulting in tighter approximation and estimation error bounds compared to regular curriculum RL.
Revised: BCRL can be advantageous if the range of values of the residuals ¯ (cid:37) kt and ˜ (cid:37) kt become smaller as the curriculum proceeds, which can be expected for a sufﬁciently ﬁne-grained curriculum.

Original: Moreover, BCRL enjoys advantages in terms of sample-efﬁciency, as fewer samples than  regular curriculum RL can be used to achieve comparable estimation errors, due to the smaller rangeand smoother shape of the target (cid:107) ˜ (cid:37) kt (cid:107) µ compared to Q max t .
Revised: In this case, fewer samples than for regular curriculum RL can be used to achieve comparable estimation errors, due to the smaller range of the target (cid:107) ˜ (cid:37) kt (cid:107) µ compared to Q max t .

Original: Iteration 0 .0 .0 .

Revised: Iteration 0 .0 .0 .

Original: .k
Revised: .k

Original: Q k t − Q ∗ t k 1 , µ
Revised: Q k t − Q ∗ t k 1 , µ

Original: T
Revised: T

Original: T
Revised: T

Original: T 3 BC-FQI
Revised: T 3 BC-FQI

Original: B-FQI
Revised: B-FQI

Original: C-FQI FQI
Revised: C-FQI FQI

Original: Iteration − 0 .0 .0 .0 .
Revised: Iteration − 0 .0 .0 .0 .

Original: C u m .
Revised: C u m .

Original: D i s c .
Revised: D i s c .

Original: R e t u r n
Revised: R e t u r n

Original: T T
Revised: T T

Original: T
Revised: T

Original: BC-FQI B-FQI
Revised: BC-FQI B-FQI

Original: C-FQI FQI
Revised: C-FQI FQI

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section serves to provide empirical evidence to the insights obtained in the previous section.
Revised: This section serves to provide empirical evidence to the insights obtained in the previous section.

Original: More precisely, we want to provide evidence that boosted curricula can indeed improve the residual in an AVI setting and with that improve the achieved agent performance.
Revised: More precisely, we want to provide evidence that boosted curricula can indeed improve the residual in an AVI setting and with that improve the achieved agent performance.

Original: Further, we show that the additional ﬂexibility of boosting w.r.t.
Revised: Further, we show that the additional ﬂexibility of boosting w.r.t.

Original: the individual function approximators can be leveraged to choose the best-suited learning approach for each learning task.
Revised: the individual function approximators can be leveraged to choose the best-suited learning approach for each learning task.

Original: We conduct experiments in multiple environments and with two different RL algorithms using the
Revised: We conduct experiments in multiple environments and with two different RL algorithms using the

Original: MushroomRL library (D’Eramo et al., 2021).
Revised: MushroomRL library (D’Eramo et al., 2021).

Original: 2 To evaluate the beneﬁt of curricula in boosted AVI settings, we turn towards the ﬁtted Q -iteration ( FQI ) algorithm in the car-on-hill environment (Ernst et al., 2005), as the low-dimensionality allows for uniform sampling of the state-action space assumed in the previous section and the ground truth optimal Q -function can be computed via bruteforce methods (Ernst et al., 2005).
Revised: 2 To evaluate the beneﬁt of curricula in boosted AVI settings, we turn towards the ﬁtted Q -iteration ( FQI ) algorithm in the car-on-hill environment (Ernst et al., 2005), as the low-dimensionality allows for uniform sampling of the state-action space assumed in the previous section and the ground truth optimal Q -function can be computed via bruteforce methods (Ernst et al., 2005).

Original: Next, we evaluate the FQI algorithm in a maze environment in which exploration is guided by the currently estimated optimal policy.
Revised: Next, we evaluate the FQI algorithm in a maze environment in which exploration is guided by the currently estimated optimal policy.

Original: Finally, we investigate boosting in a control task with DQN (Mnih et al., 2015), in which a point-mass needs to be steered over a narrow pathway subject to external perturbations.
Revised: Finally, we investigate boosting in a control task with DQN (Mnih et al., 2015), in which a point-mass needs to be steered over a narrow pathway subject to external perturbations.

Original: To disentangle the interplay between boosting and the use of a curriculum, we will compare different methods in the experiments: the algorithm under investigation ( FQI , DQN ), a version using both a curriculum with boosting ( BC - FQI , BC - DQN ) as well as two ablations of the second method - one without boosting ( C - FQI , C - DQN ) and one without a curriculum ( B - FQI ).
Revised: To disentangle the interplay between boosting and the use of a curriculum, we will compare different methods in the experiments: the algorithm under investigation ( FQI , DQN ), a version using both a curriculum with boosting ( BC - FQI , BC - DQN ) as well as two ablations of the second method - one without boosting ( C - FQI , C - DQN ) and one without a curriculum ( B - FQI ).

Original: The boosted methods that do not use a curriculum will introduce the additional approximators at the same iteration at which the curricula switches between tasks.
Revised: The boosted methods that do not use a curriculum will introduce the additional approximators at the same iteration at which the curricula switches between tasks.

Original: 6. C AR - ON - HILL
Revised: 6. C AR - ON - HILL

Original: The three car-on-hill tasks T 1 , T 2  and T 3 considered for curriculum learning in this experiment differ in the mass of the car, taking values of 0 .
Revised: The three car-on-hill tasks T 1 , T 2 , and T 3 considered for curriculum learning in this experiment differ in the mass of the car, taking values of 0 .

Original: 8 , 1 . 0  and 1 .2 respectively.
Revised: 8 , 1 . 0 , and 1 . respectively.

Original: The default mass of the environment is 1 . 

Revised: The default mass of the environment is 1 .

Original: which means we are considering a version of this environment with more challenging exploration, as the car needs more energy to escape the valley and receive a reward signal.
Revised: which means we are considering a version of this environment with more challenging exploration, as the car needs more energy to escape the valley and receive a reward signal.

Original: The dataset for learning the Q -function in each task is generated by executing 1000 trajectories with a random policy.
Revised: The dataset for learning the Q -function in each task is generated by executing 1000 trajectories with a random policy.

Original: To not make the results in this section dependent on the parametric form of the function approximator, we resort to the tree-based regression investigated in Ernst et al.
Revised: To not make the results in this section dependent on the parametric form of the function approximator, we resort to the tree-based regression investigated in Ernst et al.

Original: The results are visualized in Figure 1.
Revised: The results are visualized in Figure 1.

Original: We can see that the combination of boosting and a curriculum ( BC - FQI ) outperforms all other methods both in terms of ﬁnal approximation error achieved on Tbut also in terms of the performance of the greedy policy derived from the estimated Q -function.
Revised: We can see that the combination of boosting and a curriculum ( BC - FQI ) outperforms all other methods both in terms of ﬁnal approximation error achieved on Tbut also in terms of the performance of the greedy policy derived from the estimated Q -function.

Original: Comparing the residual of BC - FQI and C - FQI , we can see that C - FQI seems to be challenged with adapting the Q -function from T 1 to T 2 , as the jump in residual shows.
Revised: Comparing the residual of BC - FQI and C - FQI , we can see that C - FQI seems to be challenged with adapting the Q -function from T 1 to T 2 , as the jump in residual shows.

Original: BC - FQI does not exhibit such a jump in residual when moving from T 1 to T 2 .
Revised: BC - FQI does not exhibit such a jump in residual when moving from T 1 to T 2 .

Original: Furthermore, BC - FQI reduces the increased residual after switching from T 2 to T 3 while the residual of C - FQI steadily increases once arriving at the target task.
Revised: Furthermore, BC - FQI reduces the increased residual after switching from T 2 to T 3 while the residual of C - FQI steadily increases once arriving at the target task.

Original: We can particularly see that the pre-training in T 1 and T 2 leads to a signiﬁcantly smaller
Revised: We can particularly see that the pre-training in T 1 and T 2 leads to a signiﬁcantly smaller

Original: Q -function error of BC - FQI compared  B - FQI at iteration 40 , which is in line with the discussion in the previous sections.
Revised: Q -function error of BC - FQI compared to B - FQI at iteration 40 , which is in line with the discussion in the previous sections.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Q k t − Q ∗ t k 1 , µ ... T 5 T
Revised: Q k t − Q ∗ t k 1 , µ ... T 5 T

Original: T
Revised: T

Original: T T
Revised: T T

Original: T 10 BC-FQI
Revised: T 10 BC-FQI

Original: B-FQI
Revised: B-FQI

Original: C-FQI FQI     

Revised: C-FQI FQI

Original: Iteration −−−−−
Revised: Iteration −−−−−

Original: C u m .

Original: [Figure]
Revised: [Figure]

Original: i s c .
Revised: i s c .

Original: R e t u r n ... T 5 T 6 T
Revised: R e t u r n ... T 5 T 6 T

Original: T
Revised: T

Original: T T
Revised: T T

Original: [Figure]
Revised: [Figure]

Original: 6. M AZE
Revised: 6. M AZE

Original: We design a maze environment, in which an agent needs to reach a target position by navigating around two large obstacles.
Revised: We design a maze environment, in which an agent needs to reach a target position by navigating around two large obstacles.

Original: To reach the goal, the agent can choose between 4 discrete actions that move the agent up, down, left, or right.
Revised: To reach the goal, the agent can choose between 4 discrete actions that move the agent up, down, left, or right.

Original: It obtains a dense reward based on the distance to the target and a reward of +5 upon reaching the goal.
Revised: It obtains a dense reward based on the distance to the target and a reward of +5 upon reaching the goal.

Original: We design a curriculum composed oftasks, in which the size of the walls gradually increases (see Figure 3a for a visualization of tasks contained in the curriculum).
Revised: We design a curriculum composed of 10 tasks, in which the size of the walls gradually increases (see Figure 3a for a visualization of tasks contained in the curriculum).

Original: The reward function and the target task are designed to induce highly challenging exploration, requiring the use of curricula to solve the problem.
Revised: The reward function and the target task are designed to induce highly challenging exploration, requiring the use of curricula to solve the problem.

Original: As done for the caron-hill experiment, we use FQI with tree-based regression (Ernst et al., 2005).
Revised: As done for the car-on-hill experiment, we use FQI with tree-based regression (Ernst et al., 2005).

Original: For all methods, we collect a dataset for each task running 500 episodes using ε -greedy exploration, decreasing the strength of exploration as the curriculum proceeds.
Revised: For all methods, we collect a dataset for each task running 500 episodes using ε -greedy exploration, decreasing the strength of exploration as the curriculum proceeds.

Original: As can be seen in Figures 2 and 3, regular learning on the target task does not allow ﬁnd a good solution, regardless of whether boosting is used or not ( B - FQI and FQI ).
Revised: As can be seen in Figures 2 and 3, regular learning on the target task does not allow ﬁnding a good solution, regardless of whether boosting is used or not ( B - FQI and FQI ).

Original: This is due to thehighly challenging exploration problem that needs to be solved in this environment.
Revised: The poor performance highlights the challenging exploration problem that needs to be solved in this environment.

Original: The initial random exploration is insufﬁcient to maneuver around the second wall in the target task.
Revised: The initial random exploration is insufﬁcient to maneuver around the second wall in the target task.

Original: Consequently, the agent is not aware of the beneﬁt of this detour, as obvious in the learned Q -function (Figure 3b).
Revised: Consequently, the agent is not aware of the beneﬁt of this detour, as obvious in the learned Q -function (Figure 3b).

Original: Subsequently reducing the exploration does not lead to any new evidence and hence the agent quickly converges to the sub-optimal solution (Figure 2).
Revised: Subsequently reducing the exploration does not lead to any new evidence and hence the agent quickly converges to the sub-optimal solution (Figure 2).

Original: Looking at the results of C - FQI , we see that the curriculum alleviates the aforementioned exploration problem.
Revised: Looking at the results of C - FQI , we see that the curriculum alleviates the aforementioned exploration problem.

Original: However, the estimated Q -function deteriorates before approaching the target task, leading to unreliable policies.
Revised: However, the estimated Q -function deteriorates before approaching the target task, leading to unreliable policies.

Original: By introducing additional capacity as learning proceeds, BC - FQI allows to learn an optimal policy.
Revised: By introducing additional capacity as learning proceeds, BC - FQI allows learning an optimal policy.

Original: Indeed, we show in the appendix that the residuals of BC - FQI decrease in later iterations, easing the function approximation problem when progressing to the target task.
Revised: Indeed, we show in the appendix that the residuals of BC - FQI decrease in later iterations, easing the function approximation problem when progressing to the target task.

Original: 6. L INEAR SYSTEM CONTROL
Revised: 6. L INEAR SYSTEM CONTROL

Original: The last task aims at highlighting a more subtle beneﬁt of BCRL :
Revised: The last task aims at highlighting a more subtle beneﬁt of BCRL :

Original: Choosing the architecture of the function approximators appropriately for each task in the curriculum.
Revised: Choosing the architecture of the function approximators appropriately for each task in the curriculum.

Original: We show that this ﬂexibility can improve sample efﬁciency in a control task, in which a point mass needs to be steered towards
Revised: We show that this ﬂexibility can improve sample efﬁciency in a control task, in which a point mass needs to be steered towards

Original: [Figure]
Revised: [Figure]

Original: 4 Steps × 10− 1 .− 1 .− 1.− 1 .
Revised: 4 Steps × 10− 1 .− 1 .− 1 .

Original: C u m .
Revised: C u m .

Original: D i s c .
Revised: D i s c .

Original: R e t u r n× 10 2 BC-DQN C-DQN DQN RPL PPR  −−−−−0 × 10(a){+10 3 × 
+}
Revised: R e t u r n BC-DQN C-DQN DQN RPL PPR SC-DQN SBC-DQN −−−−−0 × {+10 3 × 
+}

Original: Agent performance − 5 −−
Revised: [Figure] − 5 −−

Original: LSPI −
Revised: LSPI −

Original: DQN  a goal position over a narrow pathway along which it is subject to a drift moving it away from the target position (Figure 4b).
Revised: DQN [Figure] a goal position over a narrow pathway along which it is subject to a drift moving it away from the target position (Figure 4b).

Original: The agent receives a reward of − 1 until reaching the goal, which issues a reward of 0 and ends the episode.
Revised: The agent receives a reward of − 1 until reaching the goal, which issues a reward of 0 and ends the episode.

Original: If the agent falls off the pathway, it is reset to the initial position.
Revised: If the agent falls off the pathway, it is reset to the initial position.

Original: We learn a policy for solving this target task using DQN and introduce a curriculum that ﬁrst trains in an easier system to induce a bias of moving towards the target.
Revised: We learn a policy for solving this target task using DQN and introduce a curriculum that ﬁrst trains in an easier system to induce a bias of moving towards the target.

Original: In this easier system, the point mass does not face external perturbations, obtains a reward equal to the negative squared distance to the target position and does not need to stay on a particular pathway.
Revised: In this easier system, the point mass does not face external perturbations, obtains a reward equal to the negative squared distance to the target position, and does not need to stay on a particular pathway.

Original: Since the easier environment is a linear quadratic (LQ) problem (see appendix C.3), the Q -function is known to be a quadratic function of state and action (Bradtke, 1993).
Revised: Since the easier environment is a linear-quadratic (LQ) problem (see appendix C.3), the Q -function is known to be a quadratic function of state and action (Bradtke, 1993).

Original: The boosting framework allows to exploit this knowledge by using quadratic features in the ﬁrst learning task and only rely on a neural network to realize the Q -function in the target task.
Revised: The boosting framework allows to exploit this knowledge by using quadratic features in the ﬁrst learning task and only relying on a neural network to realize the Q -function in the target task.

Original: With quadratic features, the Q -function can be efﬁciently learned from experience with least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003).
Revised: With quadratic features, the Q -function can be efﬁciently learned from experience with least-squares policy iteration ( LSPI ) (Lagoudakis & Parr, 2003).

Original: In the boosted curriculum evaluated in this task ( BC - DQN ), we hence ﬁrst learn a Q -function using LSPI in the LQ task and then adjust this Q -function in the target task with a residual using DQN .
Revised: In the boosted curriculum evaluated in this task ( BC - DQN ), we hence ﬁrst learn a Q -function using LSPI in the LQ task and then adjust this Q -function in the target task with a residual using DQN .

Original: Figure 4a shows that this incorporation of additional structure into the curriculum leads to improved performance over both directly learning on the target task ( DQN ) and  not adjusting the structure of the agent to the learning task in the curriculum ( C - DQN ).It is particularly surprising that C - DQN is performing even worse than DQN .
Revised: Figure 4a shows that this incorporation of additional structure into the curriculum ( BC - DQN ) improves performance over both directly learning on the target task ( DQN ) and a regular curriculum that does not adjust the structure of the agent to the learning task in the curriculum ( C - DQN ).

Original: There are multiple potential causes for this poor performance. One is that the DQN algorithm takes signiﬁcantly longer to learn in the LQ problem compared to LSPI ( 10 . 000 vs. 2 . 000 steps), diminishing the beneﬁt of bootstrapping the policy in the easier learning task.
Revised: We further see that two other transfer knowledge baselines - Residual Policy Learning ( RPL , Silver et al. and Probabilistic Policy Reuse ( PPR , Fern ´ andez et al.

Original: We further see no accelerated learning of C - DQN on the target task compared to DQN . This may be caused by the less accurate Q -function learned in the LQ problemwith DQN compared to LSPI (Figure 4b), which does not perform as targeted movements to the goal. We also evaluate residual policy learning ( RPL , Silver et al.
Revised: (2010)) - do not improve learning performance over regular learning on the target task (please see Appendix A for a discussion of the methods and why we chose them as baselines). Reusing the quadratic Q -function from the LQ problem as a shaping reward ( SC - DQN , Ng et al.

Original: and probabilistic policy reuse ( PPR , Fern ´ andez et al.
Revised: (1999)) drastically improves performance.

Original: leveraging the policy learned in the LQ problem with LSPI (details in C.3).
Revised: Combined with boosting ( SBC - DQN ), this improvement is even stronger.

Original: We see that
Revised: It is particularly surprising that

Original: RPL increases learning speed compared to C - DQN ,but both RPL  and PPR do not improve upon DQN . 
Revised: C - DQN , RPL , and PPR do not improve over DQN . For C - DQN , this poor performance may result from the DQN algorithm learning slower in the LQ problem compared to LSPI ( 10 . 000 vs. 2 . 000 steps), while further learning a less accurate Q -function (Figure 4b). For RPL and PPR , it is hard to provide explanations, as both methods rely on the LSPI Q -function. Nonetheless, the success of a shaping reward based on the same LSPI Q -function shows the impact of how existing knowledge is reused.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced boosted curriculum reinforcement learning ( BCRL ), a new method for action-value function estimation in curriculum RL.
Revised: We introduced boosted curriculum reinforcement learning ( BCRL ), a new method for action-value function estimation in curriculum RL.

Original: For each task, BCRL approximates the respective action-value function as a sum of residuals of the previous tasks in the curriculum, increasing the representativeness of the functional space alongside the complexity of the tasks.
Revised: For each task, BCRL approximates the respective action-value function as a sum of residuals of the previous tasks in the curriculum, increasing the representativeness of the functional space alongside the complexity of the tasks.

Original: We proved that BCRL enjoys advantageous theoretical properties in terms of convergence to the optimal action-value function of the target task.
Revised: We proved that BCRL enjoys advantageous theoretical properties in terms of convergence to the optimal action-value function of the target task.

Original: Moreover, we empirically validated BCRL across different problems and algorithms, showing its advantages over regular curriculum RL.
Revised: Moreover, we empirically validated BCRL across different problems and algorithms, showing its advantages over regular curriculum RL and knowledge transfer baselines.

Original: Our work is a step towards understanding the complex interplay between curricula and the structure of the learning agent.

Original: We assume that the curricula are so-called sequence curricula, while recent progress in automated curriculum generationdeﬁnes curricula as a distribution over tasks (Narvekar et al., 2020).Adapting our theoretical framework to these recent advances would allow to couple them with BCRL . Further, the complexity of the network, and hence the required computation, increases with each task. Finding ways to decay the inﬂuence of older Q -functions may allow for pruning, avoiding linear growth of the network.
Revised: Future research on the presented ﬁndings should investigate ways to counteract the (linearly) growing complexity of the (boosted) agent for long task sequences and to adapt the BCRL framework to recent advances in automated curriculum generation (Narvekar et al., 2020).

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: We aim to guarantee the reproducibility of our work by providing the proofs of the main paper theorems as well as additional details of the conducted experiments in the appendix.
Revised: We aim to guarantee the reproducibility of our work by providing the proofs of the main paper theorems and additional details of the conducted experiments in the appendix.

Original: Further, we provide the code for running the experiments alongside the submission and will make this code publicly available upon acceptance
Revised: Further, we provide the code for running the experiment at https://github.com/psclklnk/boosted_crl .  A CKNOWLEDGMENTS This project has received funding from the DFG project PA3179/1-1 (ROBOLEAP)

Original: A D ISCUSSION OF RELATED T RANSFER L EARNING M ETHODS
Revised: A D ISCUSSION OF RELATED TRANSFER LEARNING METHODS

Original: Given our focus on knowledge transfer via Q -functions in curriculum RL, our investigations are closely related to the ﬁeld of transfer learning and we hence want to situate our method in this ﬁeld by this short review.
Revised: Given our focus on knowledge transfer via Q -functions in curriculum RL, our investigations are closely related to the ﬁeld of transfer learning and we hence want to situate our method in this ﬁeld by this short review.

Original: Transfer learning is a well-established concept in RL and multiple surveys have been conducted in the single- and multi-agent RL setting (Taylor & Stone, 2009; Lazaric, 2012; Da Silva et al., 2018; Da Silva & Costa, 2019).
Revised: Transfer learning is a well-established concept in RL and multiple surveys have been conducted in the single- and multi-agent RL setting (Taylor & Stone, 2009; Lazaric, 2012; Da Silva et al., 2018; Da Silva & Costa, 2019).

Original: Apart from detecting a change in the current task (Hernandez-Leal et al., 2017), determining from which task to transfer knowledge (Fern´andez & Veloso, 2006; Barrett & Stone, 2015; Hernandez-Leal & Kaisers, 2017) to the current task is an important problem in transfer learning.
Revised: Apart from detecting a change in the current task (Hernandez-Leal et al., 2017), determining from which task to transfer knowledge (Fern´andez & Veloso, 2006; Barrett & Stone, 2015; Hernandez-Leal & Kaisers, 2017) to the current task is an important problem in transfer learning.

Original: For an adequately chosen curriculum, the aforementioned problems are less severe as a) the agent is assumed to be aware of the task change and b) the knowledge from the most recent task in the curriculum can be assumed to be the most relevant for solving the current one.
Revised: For an adequately chosen curriculum, the aforementioned problems are less severe as a) the agent is assumed to be aware of the task change and b) the knowledge from the most recent task in the curriculum can be assumed to be the most relevant for solving the current one.

Original: The transfer of knowledge between tasks can then happen in various forms, e.g., via samples (Lazaric et al., 2008; Lazaric & Restelli, 2011), policies (Fern´andez & Veloso, 2006; Fern´andez et al., 2010; Silver et al., 2018), reward functions (Brys et al., 2015) or demonstrations/teachers (Chernova & Veloso, 2009; Billard et al., 2016;
Revised: The transfer of knowledge between tasks can then happen in various forms, e.g., via samples (Lazaric et al., 2008; Lazaric & Restelli, 2011), policies (Fern´andez & Veloso, 2006; Fern´andez et al., 2010; Silver et al., 2018), reward functions (Brys et al., 2015) or demonstrations/teachers (Chernova & Veloso, 2009; Billard et al., 2016;

Original: Koert et al., 2020).
Revised: Koert et al., 2020).

Original: When sharing value- or Q -functions, which resembles the boosting scenario investigated in this paper, incorporating abstract concepts like objects, classes or relations (Koga et al., 2013; 2014;
Revised: When sharing value- or Q -functions, which resembles the boosting scenario investigated in this paper, incorporating abstract concepts like objects, classes, or relations (Koga et al., 2013; 2014;

Original: Da Silva & Costa, 2017; Silva & Costa, 2018) can drastically improve the generalization of learned
Revised: Da Silva & Costa, 2017; Silva & Costa, 2018) can drastically improve the generalization of learned

Original: Q - or value-functions between tasks at the expense of additional user-speciﬁed information.
Revised: Q - or value-functions between tasks at the expense of additional user-speciﬁed information.

Original: When not assuming any additional information of the aforementioned forms in a value- or Q -function transfer setting, methods that extend upon the naive idea of initializing the Q -function of the next task with  the one learned in the previous one, as e.g. done in Narvekar et al. (2016), are rather sparse.
Revised: When not assuming any additional information of the aforementioned forms in a value- or Q -function transfer setting, methods that extend upon the naive idea of initializing the Q -function of the Boutsioukis et al. (2011) investigate the idea of sparsely initializing the Q -function in the new task with a bias value for the optimal actions of the previous task in discrete state-action spaces.

Original: Rusu et al.
Revised: Rusu et al.

Original: (2016) present a particular neural network architecture – progressive neural networks – that aims to improve transfer between tasks by adding so called columns for each encountered task.
Revised: (2016) present a particular neural network architecture – progressive neural networks – that aims to improve transfer between tasks by adding so-called columns for each encountered task.

Original: Those additional columns are connected to the columns of the previous task by learnable weight matrices.
Revised: Those additional columns are connected to the columns of the previous task by learnable weight matrices.

Original: Reward shaping (Ng et al., 1999) allows to reuse a value function from a source task as a shaping function in a target task.
Revised: Reward shaping (Ng et al., 1999) allows reusing a value function from a source task as a shaping function in a target task.

Original: Both progressive neural networks and reward shaping are orthogonal to our investigation of boosting and its inﬂuence on the residual of Q -function approximators in curriculum RL.
Revised: Both progressive neural networks and reward shaping are orthogonal to our investigation of boosting and its inﬂuence on the residual of Q -function approximators in curriculum RL.

Original: As we show in Figure 5, boosting hence can still provide beneﬁts when combined with such approaches.
Revised: As we show in Figure 4a, boosting hence can still provide beneﬁts when combined with such approaches.

Original: Boutsioukis et al.

Original: (2011) investigate the idea of sparsely initializing the Qfunction in the new task with a bias value for the optimal actions of the previous task in discrete state-action spaces.

Original: We compare to residual policy learning (RPL, Silver et al.
Revised: For the comparison in Section 6.3, we choose residual policy learning (RPL, Silver et al.

Original: in Section 6.3, as it follows a similar idea of learning a residual, however in the form of a policy, andfurthermore avoids the complication of the overall curriculum by e.g. introducing a secondary RL objective (Brys et al., 2015) to realize the knowledge transfer between tasks or the need to choose appropriate bias values for Q -function initialization (Boutsioukis et al., 2011).
Revised: (2018)), as it follows a similar idea of learning a residual, however in the form of a policy, and avoids the complication of the overall curriculum by e.g. introducing a secondary RL objective (Brys et al., 2015) to realize the knowledge transfer between tasks or the need to choose appropriate bias values for Q -function initialization (Boutsioukis et al., 2011).

Original: For the environment in Section 6.3, the idea of a residual policy is well-suited since the 8 discrete actions in the target task (see appendix C.3) allows to deﬁne reasonable ways to combine the actions of the previous and the current policy.
Revised: For the environment in Section 6.3, the idea of a residual policy is well-suited since the 8 discrete actions in the target task (see appendix C.3) allow to deﬁne reasonable ways to combine the actions of the previous and the current policy.

Original: For the tasks in Section 6.1 and 6.2, combining actions is problematic due to the action spaces of the tasks.
Revised: For the tasks in Section 6.1 and 6.2, combining actions is problematic due to the action spaces of the tasks.

Original: For the car-on-hill task, combining the two opposing actions of moving left or right will either[Figure] always ignore the actions of the previous policy or ignore the actions of the current policy.
Revised: For the car-on-hill task, combining the two opposing actions of moving left or right will either always ignore the actions of the previous policy or ignore the actions of the current policy.

Original: The same problem occurs for the maze task with its four actions left , right , up  and down .
Revised: The same problem occurs for the maze task with its four actions left , right , up , and down .

Original: Further, we  evaluate probabilistic policy reuse ( PPR , Fern´andez et al.
Revised: Apart from RPL , we also evaluate probabilistic policy reuse ( PPR , Fern´andez et al.

Original: on the linear control task as it – like RPL – introduces no crucial additional design choices in a value function based curriculum.
Revised: as it – like RPL – introduces no crucial additional design choices in the curriculum.

Original: B A DDITIONAL THEOREMS AND PROOFS
Revised: B A DDITIONAL THEOREMS AND PROOFS

Original: We report an extended version of our Theorem 1 based on Theorem 3.4 in Farahmand (2011).
Revised: We report an extended version of our Theorem 1 based on Theorem 3.4 in Farahmand (2011).

Original: Theorem 1.
Revised: Theorem 1.

Original: (Theorem 3.4 of Farahmand (2011)).
Revised: (Theorem 3.4 of Farahmand (2011)).

Original: Let K be a positive integer, ρ an initial state distribution, and Q max ≤ R max 1 − γ .
Revised: Let K be a positive integer, ρ an initial state distribution, and Q max ≤ R max 1 − γ .

Original: Then, for any sequence ( Q k ) [-Kk =
-] ⊂ B ( S × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 , we have
Revised: Then, for any sequence ( Q k ) [-Kk =
-]K k =0 ⊂ B ( S × A , Q max ) , and the corresponding sequence ( ε k ) K − 1 k =0 , we have

Original: [Equation]
Revised: [Equation]

Original: where E ( ε 0 , . .
Revised: where E ( ε 0 , . .

Original: . , ε K − 1 ; r )
Revised: . , ε K − 1 ; r )

Original: =
Revised: =

Original: (cid:80) K − 1 k =

Revised: (cid:80) K − 1 k =

Original: α 2 rk (cid:107) ε k (cid:107) 2 µ , and C VI ,ρ,µ and α k as deﬁned, respectively, in Definition 3.1 and Equation 3.3 in Farahmand (2011).
Revised: α 2 rk (cid:107) ε k (cid:107) 2 µ , and C VI ,ρ,µ and α k as deﬁned, respectively, in Definition 3.1 and Equation 3.3 in Farahmand (2011).

Original: We proceed reporting the proofs of the theorems and lemmas described in the main paper.
Revised: We proceed reporting the proofs of the theorems and lemmas described in the main paper.

Revised: As a start, we ﬁrst restate the most important identities:

Revised: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q it )
Revised: Let Q ∗ t − 1 be the learned action-value for task T t − 1 , and ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of measurable action-value functions for atarget task T t obtained following our BCRL procedure (5), where Q 0 t =
Revised: =0 be a sequence of measurable action-value functions for a task T t obtained following our BCRL procedure (5), where Q 0 t =

Original: Q ∗ t − 1 .
Revised: Q ∗ t − 1 .

Original: Denote L= γC µ →∞ the Lipschitz coefﬁcient of the optimal Bellman operator, where C µ →∞ is the one-step concentrability coefﬁcient w.r.t.
Revised: Denote L the Lipschitz coefﬁcient of the optimal Bellman operator w.r.t.

Original: µ (Farahmand, 2011), and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )
Revised: a norm (cid:107)·(cid:107) , and assume the operator S is such that ∃ λ > 0 : (cid:107) ( I − S )

Original: y (cid:107)µ ≤ λ (cid:107) y
Revised: y (cid:107) ≤ λ (cid:107) y

Original: (cid:107)µ for all y ∈ B ( S × A ) .
Revised: (cid:107) for all y ∈ B ( S × A ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: Before carrying out the proof, we want to clarify the role of the chosen norm (cid:107)·(cid:107) .

Revised: While it is well known that the Bellman operator is a contraction for the supremum norm (cid:107)·(cid:107) ∞ (since (cid:107) T ∗ Q 1 − T ∗ Q 2 (cid:107) ∞ ≤ γ (cid:107) Q 1 − Q 2 (cid:107) ∞ for any Q 1 , Q 2 ∈ B ( S × A ) ), the use of the supremum norm puts a tougher requirement on the chosen function approximator S , as for the supremum norm the largest approximation error determines λ .

Revised: The use of the L p ( µ ) -norms eases the aforementioned requirement on S , however, for the L p ( µ ) -norm it is not guaranteed that the Bellman operator is a contraction for any Q 1 , Q 2 ∈ B ( S × A ) .

Revised: While Farahmand (2011) upper bounded the Lipschitz coefﬁcient of the optimal Bellman operator for L p ( µ ) -norms by

Original: [Equation]
Revised: [Equation]

Revised: where µ ( s, a ) = µ S ( s ) π b ( a | s ) , γC µ →∞ is typically larger than one.

Revised: Note, however, that this is only an upper-bound and grid-world simulations indicated that L < 1 for many Q 1 , Q 2 ∈ B ( S × A ) , however, not all.

Revised: While AVI convergence results in L p ( µ ) -norm such as (Munos, 2005) obviously apply to BCRL , the bound in Theorem 2 particularly highlights the dependency on the Q -function learned in task t − 1 .

Revised: After this discussion, we now prove the actual theorem:

Revised: [Equation]

Original: where (13) considers the assumption on the operator ( I − S ) and (14) follows noting that 
Revised: where (14) considers the assumption on the operator ( I − S ) and (15) follows noting that [Equation] Q [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Revised: t (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: 0 t

Revised: [Equation]

Revised: t (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: Unrolling the recursion in (15) then proves the desired result:

Revised: [Equation]

Revised: [Table]

Revised: Corollary 1.

Revised: Given the settings of Theorem

Revised: [Equation]

Revised: if L < 1 and λ < (cid:15)

Revised: [Equation]

Revised: [Equation]

Revised: L

Revised: [Equation]

Revised: 1+ L(cid:15) with (cid:15) ∈ [0 , 1] then

Revised: [Equation]

Revised: Proof.

Revised: If we assume λ = 1 − LL − δ with 0 < δ ≤ 1 − LL

Revised: [Equation]

Revised: then it follows that L (1 + λ ) = 1 − Lδ <and hence

Revised: [Equation]

Revised: As a next step

Revised: [Equation]

Revised: we require that the above term is upper bounded by a factor (cid:15) ∈ [0 , 1] .

Revised: This requirement yields

Revised: [Equation]

Revised: Note that it holds that 0 < δ = 1 − L L + L 2 (cid:15) ≤ 1 − LL for any (cid:15) ∈ [0 , 1] such that our initial assumption on λ are fulﬁlled.

Revised: Inserting this inequality into our deﬁnition of λ then ﬁnally yields

Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Revised: (Theorem 5.3 of Farahmand (2011))

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value functions for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8), F ⊆ B ( S × A ) be a subset of measurable action-value functions.

Original: Denoting the regular AVI residual as ¯ (cid:37) kt
Revised: Then we have

Original: = ( T ∗ t ) k . . .
Revised: [Equation] Proof.

Original: ( T ∗ 1 ) K 1 Q 01 − ( T ∗ t − 1 ) K t − 1 . . .
Revised: Although this theorem is the same as Theorem 5.3 in (Farahmand, 2011), we still provide the proof here for ease of understanding.

Original: ( T ∗ 1 ) K 1 Q 01 , we have
Revised: For an arbitrary function f ∈ F and task t , we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: ¯ (cid:37) kt − (cid:37) kt (cid:107) µ . Then, (cid:13)(cid:13)(cid:13) ¯ (cid:37) kt − (cid:37) kt (cid:13)(cid:13)(cid:13) µ = (cid:13)(cid:13)(cid:13) ( T ∗ t ) k Q 0 t − T ∗ t Q k − 1 t (cid:13)(cid:13)(cid:13) µ ≤ L (cid:13)(cid:13)(cid:13) ( T ∗ t ) k − 1 Q 0 t − Q k − 1 t (cid:13)(cid:13)(cid:13) µ = L (cid:13)(cid:13)(cid:13) ( T ∗ t ) k − 1 Q 0 t − T ∗ t

Original: Then,
Revised: Q k − 2 t + T ∗ t Q k − 2 t − Q k − 1 t (cid:13)(cid:13)(cid:13) µ = L (cid:13)(cid:13)(cid:13) ( T ∗ t ) k − 1 Q 0 t − T ∗ t Q k − 2 t + ε k − 1 t (cid:13)(cid:13)(cid:13) µ ≤ L (cid:18)(cid:13)(cid:13)(cid:13) ε k − 1 t (cid:13)(cid:13)(cid:13) µ + (cid:13)(cid:13)(cid:13) ( T ∗ t ) k − 1 Q 0 t − T ∗ t Q k − 2 t (cid:13)(cid:13)(cid:13) µ (cid:19) ≤ L (cid:18)(cid:13)(cid:13)(cid:13)

Original: [Table]
Revised: ε k − 1 t (cid:13)(cid:13)(cid:13) µ + (cid:13)(cid:13)(cid:13) ¯ (cid:37) k − 1 t − (cid:37) k − 1 t (cid:13)(cid:13)(cid:13) µ (cid:19) ≤ L (cid:18)(cid:13)(cid:13)(cid:13) ε k − 1 t (cid:13)(cid:13)(cid:13) µ + L (cid:18)(cid:13)(cid:13)(cid:13)

Revised: ε k − 2 t (cid:13)(cid:13)(cid:13) µ + (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Revised: For the last equality, we used that (cid:107) ¯ (cid:37) 1 t − (cid:37) 1 t (cid:107) µ = (cid:107) T ∗ t Q 0 t − T ∗ t Q 0 t (cid:107) µ = 0 .

Original: Theorem 4.
Revised: Theorem 4.

Original: Let B tk = max( (cid:107) ˜ (cid:37) kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).
Revised: Let B kt = max( (cid:107) ˜ (cid:37) kt (cid:107) ∞ , 1) , and V F + be the VC-dimension of F + , which is the class of all subgraphs of functions f ∈ F (Gy¨orﬁ et al., 2006).

Original: Given adataset D t = { ( s it , a it , r it , s (cid:48) it ) } Ni = for each task t , and a sequence of action-value functions ( Q it ) ki =0 generated by our BCRL, we have
Revised: Given a sequence of action-value functions ( Q it ) ki =0 generated by our BCRL, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Theorem 11.5 from Gy¨orﬁ et al.
Revised: Using Theorem 11.5 from Gy ¨ orﬁ et al.

Original: (2006), we obtain
Revised: (2006), we obtain

Original: [Equation]
Revised: [Equation]

Revised: Completing the proof requires to relate (cid:107) f − (cid:37)

Revised: kt

Revised: (cid:107) 2 µ to (16).

Original: We complete the proof using(15) and the Cauchy-Schwarz inequality to bound the ﬁrst term.
Revised: This is achieved using the CauchySchwarz inequality: [Equation] Plugging above inequality into the result obtained from Gy¨orﬁ et al. (2006) concludes the proof

Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value functions for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: ∞

Original: [Equation]
Revised: [Equation]

Revised: γ

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we bound the k -step Bellman residual.
Revised: As a ﬁrst step, we bound the k -step Bellman residual using the following Lemma.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let ( Q it )
Revised: Let ( Q it )

Original: ki
Revised: ki

Original: =0 be a sequence of action-value function for each task t , ( ε it ) ki
Revised: =0 be a sequence of action-value functions for each task t , ( ε it ) ki

Original: =0 be the respective sequence of approximation errors as deﬁned in (8).
Revised: =0 be the respective sequence of approximation errors as deﬁned in (8).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table]

Original: Iteration
Revised: [Equation] [Table] [Equation]

Original: [Figure]
Revised: [Equation]

Original: −− P a % t ( s 0 , a ) # A ... T 5 T T T 
Revised: We need to bound the individual terms (cid:107) T ∗ t Q i − 1 t − Q

Original: T T BC-FQI Proof.
Revised: i − 1 t (cid:107) ∞ .

Original: [Equation] [Figure]
Revised: We proceed as Tosatto et al. (2017): (cid:107) T ∗

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Equation] [Figure]
Revised: ∞ =

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: Inserting (21) into (20) then yields

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table]

Original: [Equation]
Revised: k − 1 (cid:88) i = (cid:0) (cid:107) ε it (cid:107) ∞ + (cid:107) T ∗ t Q i − 1 t − Q i − 1 t (cid:107) ∞ (cid:1) + (cid:107) (cid:37) 1 t (cid:107) ∞ ≤ γ k − 1 (cid:88) i =   (cid:107)

Original: [Figure] [Equation] [Figure] [Equation]
Revised: ε it (cid:107) ∞ +(1 + γ ) i − 1 (cid:88)

Original: [Figure]
Revised: j = γ i − 1 − j (cid:107) ε jt (cid:107) ∞ + γ i − 1 (cid:107) (cid:37) 1 t (cid:107) ∞   + (cid:107) (cid:37) 1 t (cid:107) ∞

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: The proof is completed by reformulating (1 + γ )(1 − γ k − i ) − (1 − γ )

Original: [Equation] [Figure] [Equation]
Revised: = 2 γ − (1 + γ ) γ k − i .

Original: [Figure]
Revised: Equipped with the result from Lemma 3, we are now ready to prove Lemma 2 by unrolling (cid:107) ˜ (cid:37) kt (cid:107) ∞ and using (19):

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) (cid:37) 1 t (cid:107) ∞

Original: ≤ R max t , solving the recursion proves the lemma.
Revised: ) ∈D [Equation] ε [Equation]

Revised: ≤ γ (cid:107) ε

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: R max t + γ (cid:107) (cid:37) k −

Revised: [Equation]

Revised: t (cid:107)

Revised: [Equation]

Revised: ≤ γ (cid:107) ε k −

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: ∞ + γ k −

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: γ

Revised: [Equation]

Revised: [Equation]

Revised: γ )

Revised: [Equation]

Original: We are now able to derive the bound for the k -step empirical Bellman residual.
Revised: k [Equation] − [Equation]

Revised: 1 −

Revised: [Equation]

Revised: ε it

Original: [Equation]
Revised: [Equation]

Revised: R max t

Revised: [Equation]

Revised: [Equation]

Revised: γ

Original: The bound is proved using (18).
Revised: [Equation] t [Equation] ∞ .

Original: C E XPERIMENTAL DETAILS
Revised: C E XPERIMENTAL DETAILS

Original: This section serves to present additional details on the experiments that did not ﬁt in the main paper, such as algorithm hyperparameters and additional details of the learning environments.
Revised: This section serves to present additional details on the experiments that did not ﬁt in the main paper, such as algorithm hyperparameters and additional details of the learning environments.

Original: These additional details will be provided for each experiment individually in the following subsections.
Revised: These additional details will be provided for each experiment individually in the following subsections

Original: The computations were executed on a desktop computer with a Geforce RTX 2080, 64 GB memory and an AMD Ryzen 9 3900X processor.
Revised: The computations were executed on a desktop computer with a Geforce RTX 2080, 64 GB memory and an AMD Ryzen 9 3900X processor.

Original: All experiments were run in roughly a week of computation time on this single machine.
Revised: All experiments were run in roughly a week of computation time on this single machine.

Original: The employed MushroomRL library is available under the MIT license.
Revised: The employed MushroomRL library is available under the MIT license.

Revised: Iteration −−−

Revised: P a % t ( s 0 , a ) # A ... T 5 T

Original: C.
Revised: T T T T

Revised: [Figure]

Original:  C AR - ON - HILL
Revised: C. C AR - ON - HILL

Original: As mentioned in the main text, we used the FQI algorithm with extra-trees (Ernst et al., 2005) implemented in MushroomRL (D’Eramo et al., 2021) to conduct this experiment.
Revised: As mentioned in the main text, we used the FQI algorithm with extra-trees (Ernst et al., 2005) implemented in MushroomRL (D’Eramo et al., 2021) to conduct this experiment.

Original: The number of trees in the extra-trees implementation was set to 50 , the minimum number of samples required to split an internal node to 5  and the minimum number of samples required to be a leaf node to 2 .
Revised: The number of trees in the extra-trees implementation was set to 50 , the minimum number of samples required to split an internal node to 5 , and the minimum number of samples required to be a leaf node to 2 .

Original: All other parameters were left to the implementation defaults of MushroomRL.
Revised: All other parameters were left to the implementation defaults of MushroomRL.

Original: Based on this implementation we created a boosted version that runs the original FQI algorithm on the residuals of the previously learned models.
Revised: Based on this implementation we created a boosted version that runs the original FQI algorithm on the residuals of the previously learned models.

Original: The hyperparameters for those extra-trees models were the same as for the regular FQI .
Revised: The hyperparameters for those extra-trees models were the same as for the regular FQI .

Original: The implementation can be found in the accompanying code.
Revised: The implementation can be found in the accompanying code.

Original: The car-on-hill environment used for this experiment is also implemented in MushroomRL.
Revised: The car-on-hill environment used for this experiment is also implemented in MushroomRL.

Original: C. M AZE
Revised: C. M AZE

Original: The state space of the maze environments consists of the two-dimensional coordinates s =[ s x , s y ] ∈ [0 , 1] 2 .
Revised: The state-space of the maze environments consists of the two-dimensional coordinates s =[ s x , s y ] ∈ [0 , 1] 2 .

Original: The initial state is s 0 =[0 , 0] , and the goal state is g 0 =[0 .
Revised: The initial state is s 0 =[0 , 0] , and the goal state is g 0 =[0 .

Original: The four actions displace the agent in x - or y -position by a value of 0 .
Revised: The four actions displace the agent in x - or y -position by a value of 0 .

Original: The discount factor is set to γ =0 .
Revised: The discount factor is set to γ =0 .

Original: 99 and the horizon length is 200 .
Revised: 99 and the horizon length is 200 .

Original: We use a dense reward function R ( · , · , s (cid:48) )
Revised: We use a dense reward function R ( · , · , s (cid:48) )

Original: =
Revised: =

Original: −| g − s (cid:48) | when | g − s (cid:48)
Revised: −| g − s (cid:48) | when | g − s (cid:48)

Original: | > = 0 .
Revised: | > = 0 .

Original: 1 , otherwise R ( · , · , s (cid:48) )
Revised: 1 , otherwise R ( · , · , s (cid:48) )

Original: =
Revised: =

Original: +5 and the episode terminates.
Revised: +5 and the episode terminates.

Original: There are two walls centered at x = 0 .
Revised: There are two walls centered at x = 0 .

Original: 2 and x = 0 .
Revised: 2 and x = 0 .

Original: 6 , both with width w =0 .
Revised: 6 , both with width w =0 .

Original: 05 , and height h t
Revised: 05 , and height h t

Original: depending on the task t .
Revised: depending on the task t .

Original: We design a curriculum composed of 10 tasks, with h 1 ,..., 9 = { 0 . 05 , 0 . 15 , . . .
Revised: We design a curriculum composed of 10 tasks, with h 1 ,..., 9 = { 0 . 05 , 0 . 15 , . . .

Original: , 0 . 85 } and h 10 =0 .
Revised: , 0 . 85 } and h 10 =0 .

Original: Hence in the ﬁnal task T 10 , there is only a very narrow passage through which the walls can be passed.
Revised: Hence in the ﬁnal task T 10 , there is only a very narrow passage through which the walls can be passed.

Original: We again make use of the FQI algorithm with extra-trees implemented in MushroomRL to conduct this experiment, again using 50 trees, a minimum number of 5 samples to split an internal node, a minimum number of 2 samples to be a leaf node and everything else left to the implementation defaults of MushroomRL.
Revised: We again make use of the FQI algorithm with extra-trees implemented in MushroomRL to conduct this experiment, again using 50 trees, a minimum number of 5 samples to split an internal node, a minimum number of 2 samples to be a leaf node, and everything else left to the implementation defaults of MushroomRL.

Original: The main conceptual different to the car-on-hill experiment is that exploration is guided by the currently estimated Q -function, as we are employing an εgreedy exploration strategy to generate the samples for learning the Q -functions of the learning tasks.
Revised: The main conceptual difference to the car-on-hill experiment is that exploration is guided by the currently estimated Q -function, as we are employing an εgreedy exploration strategy to generate the samples for learning the Q -functions of the learning tasks.

Original: The exploration percentage ε is annealed in later environments.
Revised: The exploration percentage ε is annealed in later environments.

Original: More precisely, we set ε 1 ,..., 10 = { 1 ., 1 ., 1 ., 1 ., .
Revised: More precisely, we set ε 1 ,..., 10 = { 1 ., 1 ., 1 ., 1 ., .

Original: 25 } for the tasks T 1 , . .
Revised: 25 } for the tasks T 1 , . .

Original: , T 10 .
Revised: , T 10 .

Original: Figure 6 visualizes the residuals of the starting state s 0 that the Q -functions learned in the individual tasks, i.e., the additional values the Q -functions needed to predict in addition to the Q -value predicted from the ensemble of previously learned Q -functions.
Revised: Figure 5 visualizes the residuals of the starting state s 0 that the Q -functions learned in the individual tasks, i.e., the additional values the Q -functions needed to predict in addition to the Q -value predicted from the ensemble of previously learned Q -functions.

Original: We see that the magnitude of residuals continuously decreases from task T 1 to task T 6 .
Revised: We see that the magnitude of residuals continuously decreases from task T 1 to task T 6 .

Original: Switching to task T 7 increases the magnitude of the residuals after which it then continuously declines again.
Revised: Switching to task T 7 increases the magnitude of the residuals after which it then continuously declines again.

Original: The reason for this jump is that, while the optimal policy is largely unaffected by the presence of the small walls in the early learning tasks, a more signiﬁcant detour is required starting from task T 7 .
Revised: The reason for this jump is that, while the optimal policy is largely unaffected by the presence of the small walls in the early learning tasks, a more signiﬁcant detour is required starting from task T 7 .

Original: C. L INEAR SYSTEM CONTROL
Revised: C. L INEAR SYSTEM CONTROL

Original: The linear system that needs to be controlled is given by˙ s =   ˙ x ¨ x ˙ y ¨ y 
Revised: The linear system that needs to be controlled is given by

Original:  =      0 − 0 .
Revised: [Equation]     0 − 0 .

Original: 1       0 − 0 .
Revised: 1       0 − 0 .

Original:    x ˙ xy ˙
Revised:    x ˙ xy ˙

Original: y 
Revised: y 

Original:  +   0100  (cid:20) a x a y (cid:21) =
Revised:  +   0100  (cid:20) a x a y (cid:21) =

Original: As + Ba , representing a simple, fully-actuated point-mass with a slight amount of friction.
Revised: As + Ba , representing a simple, fully-actuated point-mass with a slight amount of friction.

Original: For the linear quadratic (LQ) task, to which the curriculum learning agents are exposed ﬁrst, the reward function is given by[-r ( s , a , s (cid:48) ) = − s T   3 0 00 1 
 [Equation] [Equation]-]
Revised: For the linear quadratic (LQ) task, to which the curriculum learning agents are exposed ﬁrst, the reward function is given by[-r ( s , a , s (cid:48) ) = − s T   3 0 00 1 
 [Equation] [Equation]-]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: a and the initial state is given by s 0 =
Revised: a and the initial state is given by s 0 =

Original: [ − 4 v x, 0 0 v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .
Revised: [ − 4 v x, 0 0 v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .

Original: In the target task, the agent receives the following reward signal
Revised: In the target task, the agent receives the following reward signal

Original: [Equation]
Revised: [Equation]

Original: 0 , else and its initial position is given by s 0 =
Revised: 0 , else and its initial position is given by s 0 =

Original: [ − 4 v x, 0 y v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .
Revised: [ − 4 v x, 0 y v y, 0 ] T , where v x, 0 and v y, 0 are sampled uniformly within [ − 0 .

Original: 1 , 0 . 1] and y within [ − 0 . 4 , 0 . 4] .
Revised: 1 , 0 . 1] and y within [ − 0 . 4 , 0 . 4] .

Original: Furthermore, the agent needs to keep its y -position within [ − 0 . 5 , 0 . 5] in order to stay on the pathway indicated in Figure 4b and it faces an additional drift moving it away from the target, particularly[Equation]
Revised: Furthermore, the agent needs to keep its y -position within [ − 0 . 5 , 0 . 5] in order to stay on the pathway indicated in Figure 4b and it faces an additional drift moving it away from the target, particularly

Original: [Equation]
Revised: [Equation]

Original: Consequently, the policies pre-trained in the LQ task do not quite reach the target but need to explore that they need to counteract this external force while staying on the pathway.
Revised: Consequently, the policies pre-trained in the LQ task do not quite reach the target but need to explore that they need to counteract this external force while staying on the pathway.

Original: To learn with LSPI in the LQ task, we represent the policy by a PD-controller with explorative noise
Revised: To learn with LSPI in the LQ task, we represent the policy by a PD-controller with explorative noise

Original: [Equation]
Revised: [Equation]

Original: as this ensures the quadratic form of the optimal Q -function mentioned in the main text.
Revised: as this ensures the quadratic form of the optimal Q -function mentioned in the main text.

Original: More precisely, it can be shown that
Revised: More precisely, it can be shown that

Original: [Equation]
Revised: [Equation]

Original: where φ ( s , a ) computes linear and quadratic terms of s and a (please see Bradtke (1993) for additional details).
Revised: where φ ( s , a ) computes linear and quadratic terms of s and a (please see Bradtke (1993) for additional details).

Original: When learning with DQN in both the LQ and target task, the agent can choose between applying 8 discrete actions[Equation] [Equation] .(cid:21)
Revised: When learning with DQN in both the LQ and target task, the agent can choose between applying 8 discrete actions

Original: [Equation]
Revised: [Equation]

Original: (cid:21) .

Original: Consequently, we predict these discrete Q -values with the function approximator learned in the LQ task with LSPI when training on the target task.
Revised: Consequently, we predict these discrete Q -values with the function approximator learned in the LQ task with LSPI when training on the target task.

Original: The Q -networks employed in DQN consist of two hidden layers of 128 neurons and ReLU activations.
Revised: The Q -networks employed in DQN consist of two hidden layers of 128 neurons and ReLU activations.

Original: During training, we realized that the update frequency of the target network in the DQN algorithm had a strong impact on the ﬁnal agent performance.
Revised: During training, we realized that the update frequency of the target network in the DQN algorithm had a strong impact on the ﬁnal agent performance.

Original: Hence, we conducted a grid search testing values in [25 , 50 , 100 , 200 , 500 , 1000 , 2000] .
Revised: Hence, we conducted a grid search testing values in [25 , 50 , 100 , 200 , 500 , 1000 , 2000 , 4000] .

Original: For regular DQN on the target task, an update frequency of 200 performed best, while BC - DQN yielded  best results with an update frequency of 2000 .
Revised: For regular DQN on the target task, an update frequency of 200 performed best, while BC - DQN yielded the best results with an update frequency of 2000 .

Original: For C - DQN , best results where achieved when using an update frequency of 1000 in the LQ task and an update frequency of 25 in the target task.
Revised: For C - DQN , best results were achieved when using an update frequency of 1000 in the LQ task and an update frequency of 25 in the target task.

Original: We used
Revised: When employing a shaping reward, takingenvironment steps between an update of the target network performed best for both boosted and non-boosted DQN .

Original:  Adam for optimization of the network with a learning rate of 10 − 4 and ( β 1 , β 2 ) =
Revised: We used Adam for optimization of the network with a learning rate of 10 − 4 and ( β 1 , β 2 ) =

Original: To exploit the previously learned policy in the target task for the curriculum methods, we used a ﬁxed value of ε = 0 .
Revised: To exploit the previously learned policy in the target task for the curriculum methods, we used a ﬁxed value of ε = 0 .

Original: 1 in the target task.
Revised: 1 in the target task.

Original: For default DQN , we annealed ε from 1 to 0 .
Revised: For default DQN , we annealed ε from 1 to 0 .

Original: 1 over the ﬁrstenvironment steps.
Revised: 1 over the ﬁrst 1000 environment steps.

Original: As mentioned in the main paper, we additionally investigate residual policy learning ( RPL ) and probabilistic policy reuse ( PPR ) in this environment.
Revised: As mentioned in the main paper, we additionally investigate residual policy learning ( RPL ) and probabilistic policy reuse ( PPR ) in this environment.

Original: Both approaches exploit a policy π source learned in a source task (for our experiment this is the LQ task) to accelerate learning in the target task.
Revised: Both approaches exploit a policy π source learned in a source task (for our experiment this is the LQ task) to accelerate learning in the target task.

Original: As the methods are agnostic to the structure of the source task policy, we make use of the policy learned with LSPI , as this policy exhibits a more structured movement towards the target (see Figure 4b) and further requires less environment interactions to be learned (see Figure 4a).
Revised: As the methods are agnostic to the structure of the source task policy, we make use of the policy learned with LSPI , as this policy exhibits a more structured movement towards the target (see Figure 4b) and further requires fewer environment interactions to be learned (see Figure 4a).

Original: To implement RPL (Silver et al., 2018), we need to specify how to combine the actions a i and a j of the source task policy π source and the residual policy to be learned in the target task.
Revised: To implement RPL (Silver et al., 2018), we need to specify how to combine the actions a i and a j of the source task policy π source and the residual policy to be learned in the target task.

Original: In our experiment, we can simply add the actions a i and a j and then select the discrete action that is more similar to the combined action in terms of euclidean distance.
Revised: In our experiment, we can simply add the actions a i and a j and then select the discrete action that is more similar to the combined action in terms of euclidean distance.

Original: Ifthere are multiple actionsthat are within the same minimum distance to the combined action, we choose the action that is more similar to the action of the residual policy, i.e. more similar to a j .
Revised: If multiple actions are within the same minimum distance to the combined action, we choose the action that is more similar to the action of the residual policy, i.e. more similar to a j .

Original: This allows the residual policy to still move opposite to the previous policy if necessary, while preserving the bias induced by the previous policy if the residual policy does not produce an action exactly opposing the one from the previous policy.
Revised: This allows the residual policy to still move opposite to the previous policy if necessary while preserving the bias induced by the previous policy if the residual policy does not produce an action exactly opposing the one from the previous policy.

Original: RPL has an offset parameter that controls for how many steps the residual policy does not interfere with the source task policy.
Revised: RPL has an offset parameter that controls for how many steps the residual policy does not interfere with the source task policy.

Original: For the described way of combining actions, not interfering means simply choosing the same actions as the source task policy.
Revised: For the described way of combining actions, not interfering means simply choosing the same actions as the source task policy.

Original: We tried values of 0 , 2000  and 4000 steps for this offset value, where 2000 performed best.
Revised: We tried values of 0 , 2000 , and 4000 steps for this offset value, whereperformed best.

Original: PPR requires the user to choose a decay factor ν to anneal the likelihood of executing the source task policy via p ( π source ) =
Revised: PPR requires the user to choose a decay factor ν to anneal the likelihood of executing the source task policy via p ( π source ) =

Original: ν t , where t is the number of past interactions (note that we follow the interpretation of PPR by Brys et al.
Revised: ν t , where t is the number of past interactions (note that we follow the interpretation of PPR by Brys et al.

Original: (2015) for transfer from a single source task).
Revised: (2015) for transfer from a single source task).

Revised: We tried three different values ν =0 .

Revised: 998 , ν =0 .

Revised: 9996 and ν =0 . 99988 , with ν =0 .

Revised: 998 performing best.

Revised: With this value, ν t =0 .

Original:  As for the other baselines, we do a grid search over the target network update frequency, choosing the best.
Revised: 1 after t ≈ 1150 environment steps. As for the other baselines, we do a grid search over the target network update frequency, choosing the best.

Original: For RPL , a value of 25 performed best.
Revised: For RPL , a value of 25 performed best.

Original: For PPR , a value ofperformed slightly better than 25 or 50 .
Revised: For PPR , a value of 200 performed slightly better than lower ones.

Revised: For both methods, update frequencies larger than 1000 did not learn at