Original: L IPSCHITZ L IFELONG R EINFORCEMENT L EARNING
Revised: L IPSCHITZ L IFELONG R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the problem of reusing prior experience when an agent is facing a series of Reinforcement Learning (RL) tasks.
Revised: We consider the problem of knowledge transfer when an agent is facing a series of Reinforcement Learning (RL) tasks.

Original: We introduce a novel metric between
Revised: We introduce a novel metric between

Original: Markov Decision Processes and focus on the study and exploitation of the optimal value function’s Lipschitz continuity in the task space with respect to that metric.
Revised: Markov Decision Processes and establish that close MDPs have close optimal value functions, that is that optimal value functions are Lipschitz continuous with respect to tasks.

Original: These theoretical results lead us to a value transfer method for Lifelong RL, which we use to build a PAC-MDP algorithm that exploits continuity to accelerate learning.
Revised: These theoretical results lead us to a value transfer method for Lifelong RL, which we use to build a PAC-MDP algorithm with improved convergence rate.

Original: We illustrate the beneﬁts of the method in Lifelong RL experiments.
Revised: We illustrate the beneﬁts of the method in Lifelong RL experiments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Lifelong Reinforcement Learning (RL) is a problem where an agent faces a series of RL tasks, drawn sequentially.
Revised: Lifelong Reinforcement Learning (RL) is an online problem where an agent faces a series of RL tasks, drawn sequentially.

Original: Transferring the knowledge of prior experience while solving new tasks is a key question in that setting (see Lazaric (2012) or Taylor and Stone (2009) for surveys).
Revised: Transferring the knowledge of prior experience while solving new tasks is a key question in that setting (Lazaric, 2012; Taylor and Stone, 2009).

Original: We elaborate on the intuitive idea that similar tasks should allow a large amount of transfer.
Revised: We elaborate on the intuitive idea that similar tasks should allow a large amount of transfer.

Original: Thus, an agent able to measure this similarity online  should be able to perform transfer from prior source tasks accordingly.
Revised: An agent able to compute online a similarity measure between source tasks and the current target task should be able to perform transfer.

Original: Measuring the amount of inter-task similarity is not new (Carroll and Seppi, 2005;
Revised: By measuring the amount of inter-task similarity, we design a novel method for value transfer, practically deployable in the online Lifelong RL setting.

Original: Fern´andez and Veloso, 2006; Lazaric et al., 2008; Mahmud et al., 2013; Brunskill and Li, 2013;
Revised: Speciﬁcally, we introduce a metric between MDPs and prove that the optimal Q-value function is Lipschitz continuous with respect to MDPs.

Original: Ammar et al., 2014; Song et al., 2016). Following this idea, we present a novel method for value transfer, practically deployable in the Lifelong RL setting.
Revised: This property allows to compute a provable upper-bound on the optimal value function of a target task, given the learned optimal value function of a source task. Knowing this upper bound allows to accelerate the convergence of an RMax algorithm (Brafman and Tennenholtz, 2002).

Original: We prove that the transfer method is guaranteed to be non-negative, meaning that the transfer cannot cause performance degradation.
Revised: This transfer method is non-negative (it cannot cause performance degradation) as the computed upper bound does not underestimate the optimal Q-value function.

Original: Our contributions are as follows.
Revised: Our contributions are as follows.

Original: First, we study theoretically the Lipschitz continuity of the optimal value function in the task space (Section 3).
Revised: First, we study theoretically the Lipschitz continuity of the optimal value function in the task space (Section 3).

Original: Then, we use this continuity property to propose a value-transfer method based on a local distance between MDPs (Section 4).
Revised: Then, we use this continuity property to propose a value-transfer method based on a local distance between MDPs (Section 4).

Original: Full knowledge of both MDPs is not required and the transfer is non-negative, which makes the method  practical and safe.
Revised: Full knowledge of both MDPs is not required and the transfer is non-negative, which makes the method both practical and safe.

Original: In Section 4.2, we build a PAC-MDP algorithm called Lipschitz RMax , applying this transfer method online in the Lifelong RL setting.
Revised: In Section 4.2, we build a PAC-MDP algorithm called Lipschitz RMax , applying this transfer method online in the Lifelong RL setting.

Original: We provide sample and computational complexity boundsaccordingly and showcase the algorithm in Lifelong RL experiments (Section 5).
Revised: We provide sample and computational complexity bounds and showcase the algorithm in Lifelong RL experiments (Section 5).

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Reinforcement Learning (RL) (Sutton and Barto, 1998) is a framework for sequential decision making.
Revised: Reinforcement Learning (RL) (Sutton and Barto, 1998) is a framework for sequential decision making.

Original: The problem is typically modeled as a Markov Decision Process (MDP) (Puterman, 2014) consisting in a 4-tuple (cid:104)S , A , R, T (cid:105) where S is a state space, A an action space, R as is the expected reward of taking action a in state s and T ass (cid:48) is the transition probability of reaching state s (cid:48) when taking action a in state s .
Revised: The problem is typically modeled as a Markov Decision Process (MDP) (Puterman, 2014) consisting in a 4-tuple (cid:104)S , A , R, T (cid:105) where S is a state space, A an action space, R as is the expected reward of taking action a in state s and T ass (cid:48) is the transition probability of reaching state s (cid:48) when taking action a in state s .

Original: Without loss of generality, we assume R as ∈ [0 , 1] .
Revised: Without loss of generality, we assume R as ∈ [0 , 1] .

Original: Given a discount factor γ ∈ [0 , 1) , the expected cumulative return (cid:80) t γ t R a t s t obtained along a trajectory starting with state s and action a is noted Q ( s, a ) and called the Q-function.
Revised: Given a discount factor γ ∈ [0 , 1) , the expected cumulative return (cid:80) t γ t R a t s t obtained along a trajectory starting with state s and action a is noted Q ( s, a ) and called the Q-function.

Original: The optimal Q-function Q ∗ is the highest attainable expected return from s, a and V ∗ ( s ) = max a ∈A Q ∗ ( s, a ) is the optimal value function in s .
Revised: The optimal Q-function Q ∗ is the highest attainable expected return from s, a and V ∗ ( s ) = max a ∈A Q ∗ ( s, a ) is the optimal value function in s .

Original: Lifelong RL (Silver et al., 2013; Brunskill and Li, 2014; Abel et al., 2018) is the problem of experiencing  a series of MDPs drawn from an unknown distribution.
Revised: Lifelong RL (Silver et al., 2013; Brunskill and Li, 2014) is the problem of experiencing online a series of MDPs drawn from an unknown distribution.

Original: Each time an MDP is sampled, a classical RL problem takes place where the agent is able to interact with the environment to maximize its expected return.
Revised: Each time an MDP is sampled, a classical RL problem takes place where the agent is able to interact with the environment to maximize its expected return.

Original: In this setting, it is reasonable to think that knowledge gained on previous MDPs could be re-used to improve the performance in new MDPs.
Revised: In this setting, it is reasonable to think that knowledge gained on previous MDPs could be re-used to improve the performance in new MDPs.

Original: In this paper, we provide a novel method for such transfer by characterizing the way the optimal Q-function can evolve across tasks.
Revised: In this paper, we provide a novel method for such transfer by characterizing the way the optimal Q-function can evolve across tasks.

Original: We restrict the scope of the study to the case where sampled MDPs share the same state-action space S × A .
Revised: We restrict the scope of the study to the case where sampled MDPs share the same state-action space S × A .

Original: For brevity, we will refer indifferently to MDPs, models or tasks, and write them M = (cid:104) R, T (cid:105) .
Revised: For brevity, we will refer indifferently to MDPs, models or tasks, and write them M = (cid:104) R, T (cid:105) .

Original: Using a metric between MDPs has the appealing characteristic of quantifying the amount of similarity between tasks, which intuitively should be linked to the amount of transfer achievable.
Revised: Using a metric between MDPs has the appealing characteristic of quantifying the amount of similarity between tasks, which intuitively should be linked to the amount of transfer achievable.

Original: Song et al.
Revised: Song et al.

Original: (2016) deﬁne a metric based on the bi-simulation metric introduced by Ferns et al.
Revised: (2016) deﬁne a metric based on the bi-simulation metric introduced by Ferns et al.

Original: (2004) and the
Revised: (2004) and the

Original: Wasserstein metric (Villani, 2008).
Revised: Wasserstein metric (Villani, 2008).

Original: Value transfer is performed between states with low bi-simulation distances.
Revised: Value transfer is performed between states with low bi-simulation distances.

Original: However, this metric requires knowing both MDPs completely and is thus unusable in the
Revised: However, this metric requires knowing both MDPs completely and is thus unusable in the

Original: Lifelong RL setting where we expect to perform transfer before having learned the current MDP.
Revised: Lifelong RL setting where we expect to perform transfer before having learned the current MDP.

Original: Further, the transfer technique they propose does allow negative transfer (see Appendix, Section A).
Revised: Further, the transfer technique they propose does allow negative transfer (see Appendix, Section A).

Original: Carroll and Seppi (2005) also deﬁne a value-transfer method based on a measure of similarity between tasks.
Revised: Carroll and Seppi (2005) also deﬁne a value-transfer method based on a measure of similarity between tasks.

Original: However, this measure is not computable online and thus not applicable to the
Revised: However, this measure is not computable online and thus not applicable to the

Original: Lifelong RL setting.
Revised: Lifelong RL setting.

Original: Mahmud et al.
Revised: Mahmud et al.

Original: and Brunskill and Li (2013) propose MDP clustering methods respectively using a metric quantifying the regret of running the optimal policy of one MDP in the other MDP and the L 1 norm between the MDP models.
Revised: and Brunskill and Li (2013) propose MDP clustering methods respectively using a metric quantifying the regret of running the optimal policy of one MDP in the other MDP and the L 1 norm between the MDP models.

Original: An advantage of clustering is to prune the set of possible source tasks.
Revised: An advantage of clustering is to prune the set of possible source tasks.

Original: They use their approach for policy transfer, which differs from the value-transfer method proposed in this paper.
Revised: They use their approach for policy transfer, which differs from the value-transfer method proposed in this paper.

Original: Ammar et al.
Revised: Ammar et al.

Original: (2014)use a Restricted Boltzmann Machine to learn the model of a source MDP and view the prediction error on a target MDP as a dissimilarity measure in the task space.
Revised: (2014) learn the model of a source MDP and view the prediction error on a target MDP as a dissimilarity measure in the task space.

Original: Their method makes use of samples from both tasks and is not readily applicable to the online setting considered in this paper.
Revised: Their method makes use of samples from both tasks and is not readily applicable to the online setting considered in this paper.

Original: Lazaric et al.
Revised: Lazaric et al.

Original: (2008) provide a practical method for sample transfer, computing a similarity metric reﬂecting the probability of the models to be identical.
Revised: (2008) provide a practical method for sample transfer, computing a similarity metric reﬂecting the probability of the models to be identical.

Original: Their approach is applicable in a batch RL setting as opposed to our online setting.
Revised: Their approach is applicable in a batch RL setting as opposed to our online setting.

Original: The approach developed by Sorg and Singh (2009) is very similar to ours in the sense that they prove bounds on the optimal Q-function for new tasks, assuming that both MDPs are known and that a soft homomorphism exists between the state spaces.
Revised: The approach developed by Sorg and Singh (2009) is very similar to ours in the sense that they prove bounds on the optimal Q-function for new tasks, assuming that both MDPs are known and that a soft homomorphism exists between the state spaces.

Original: Brunskill and Li (2013) also provide a method that can be used for Q-function bounding inthe multi-task RL setting.
Revised: Brunskill and Li (2013) also provide a method that can be used for Q-function bounding in multi-task RL.

Original: Abel et al.
Revised: Abel et al.

Original: (2018) present the MaxQInit algorithm, providing transferred bounds on the Q-function with high probability while preserving PAC-MDP guarantees.
Revised: (2018) present the MaxQInit algorithm, providing transferred bounds on the Q-function with high probability while preserving PAC-MDP guarantees.

Original: Given a set ofpreviously solved tasks, they derive the probability that the maximum over the Q-values of previous MDPs isindeed an upper bound on the current task’s optimal Q-function.
Revised: Given a set of solved tasks, they derive the probability that the maximum over the Q-values of previous MDPs is an upper bound on the current task’s optimal Q-function.

Original: This results in a method that performs non-negative transfer with high probability once enough tasks have been sampled.
Revised: This results in a method for non-negative transfer with high probability once enough tasks have been sampled.

Original:  L IPSCHITZ CONTINUITY OF Q- FUNCTIONS
Revised:  L IPSCHITZ CONTINUITY OF Q- FUNCTIONS

Original: The intuition we build on is that similar MDPs should have similar optimal Q-functions.
Revised: The intuition we build on is that similar MDPs should have similar optimal Q-functions.

Original: Formally, this insight can be translated into a continuity property of the optimal Q-functions over the MDP space M .
Revised: Formally, this insight can be translated into a continuity property of the optimal Q-functions over the MDP space M .

Revised: The remainder of this section mathematically formalizes this intuition that will be used in the next Section to derive a practical method for value transfer.

Original: To that end, we introduce a local pseudo-metric characterizing the distance between the models of two MDPs at a particular state-action pair.
Revised: To that end, we introduce a local pseudo-metric characterizing the distance between the models of two MDPs at a particular state-action pair.

Original: A reminder and a detailed discussion on the metrics (and related objects) used herein can be found in the Appendix, Section B.
Revised: A reminder and a detailed discussion on the metrics (and related objects) used herein can be found in the Appendix, Section B.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given two tasks M = (cid:104) R, T (cid:105) and ¯ M = (cid:104) ¯ R, ¯ T (cid:105) , and a function f : S → R + , we deﬁne the pseudo-metric between models at ( s, a ) ∈ S ×
Revised: Given two tasks M = (cid:104) R, T (cid:105) and ¯ M = (cid:104) ¯ R, ¯ T (cid:105) , and a function f : S → R + , we deﬁne the pseudo-metric between models at ( s, a ) ∈ S ×

Original: A w.r.t.
Revised: A w.r.t.

Original: f as:
Revised: f as:

Original: [Equation]
Revised: [Equation]

Original: This pseudo-metric is relative to a positive function f .
Revised: This pseudo-metric is relative to a positive function f .

Original: We implicitly cast this deﬁnition in the context of discrete state spaces.
Revised: We implicitly cast this deﬁnition in the context of discrete state spaces.

Original: The extension to continuous spaces is straightforward but beyond the scope of this paper.
Revised: The extension to continuous spaces is straightforward but beyond the scope of this paper.

Original: Let Q ∗ M denote the optimal Q-function of MDP M ∈ M .
Revised: Let Q ∗ M denote the optimal Q-function of MDP M ∈ M .

Original: Proposition 1 (Local pseudo-Lipschitz continuity) .
Revised: Proposition 1 (Local pseudo-Lipschitz continuity) .

Original: For two MDPs M, ¯ M , for all ( s, a ) ∈ S × A ,
Revised: For two MDPs M, ¯ M , for all ( s, a ) ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Original: with the MDPs local pseudo-metric ∆ M ¯ M ( s, a ) (cid:44) min (cid:110)
Revised: with the MDPs local pseudo-metric ∆ M ¯ M ( s, a ) (cid:44) min (cid:110)

Original: d ¯ MM ( s, a ) , d M ¯ M ( s, a ) (cid:111) , and the local MDP dissimilarity d ¯ M M : S ×
Revised: d ¯ MM ( s, a ) , d M ¯ M ( s, a ) (cid:111) , and the local MDP dissimilarity d ¯ MM : S ×

Original: A → R is the unique solution to the following ﬁxed-point equation for d :
Revised: A → R is the unique solution to the following ﬁxed-point equation for d :

Original: [Equation]
Revised: [Equation]

Original: All the proofs of the paper can be found in the Appendix.
Revised: All the proofs of the paper can be found in the Appendix.

Original: This result establishes that the distance between the optimal Q-functions of two MDPs at ( s, a ) ∈ S × A is controlled by a local dissimilarity between the MDPs.
Revised: This result establishes that the distance between the optimal Q-functions of two MDPs at ( s, a ) ∈ S × A is controlled by a local dissimilarity between the MDPs.

Original: The latter follows a ﬁxed-point equation (Equation 3), which can be solved by
Revised: The latter follows a ﬁxed-point equation (Equation 3), which can be solved by

Original: Dynamic Programming (DP) (Bellman, 1957).
Revised: Dynamic Programming (DP) (Bellman, 1957).

Original: Note that, although the local MDP dissimilarity d ¯ MM is asymmetric, ∆ M ¯ M ( s, a ) is a pseudo-metric, hence the name pseudo-Lipschitz continuity .
Revised: Note that, although the local MDP dissimilarity d ¯ MM is asymmetric, ∆ M ¯ M ( s, a ) is a pseudo-metric, hence the name pseudo-Lipschitz continuity .

Original: Similar results for the value function of a ﬁxed policy and the optimal value function V ∗ M can easily be derived (Appendix, Section D).
Revised: Similar results for the value function of a ﬁxed policy and the optimal value function V ∗ M can easily be derived (Appendix, Section D).

Revised: Thus, overall, the optimal Q-functions of two close MDPs (in the sense of Equation 1) are themselves close to each other.

Original: A consequence of Proposition 1 is a global pseudo-Lipschitz continuity:
Revised: A consequence of Proposition 1 is a global pseudo-Lipschitz continuity:

Original: Proposition 2 (Global pseudo-Lipschitz continuity) .
Revised: Proposition 2 (Global pseudo-Lipschitz continuity) .

Original: For two MDPs M , ¯ M , for all ( s, a ) ∈ S × A ,
Revised: For two MDPs M , ¯ M , for all ( s, a ) ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Revised: , with δ ¯ MM (cid:44)

Original:  Despite being interesting from a theoretical perspective, we do not use this result for transfer because it is impractical to compute.
Revised: [Equation] Despite being interesting from a theoretical perspective, we do not use this result for transfer because it is impractical to compute.

Original: Indeed, estimating the maximum in Equation 4 might be as hard as solving both MDPs (which, when it happens, is too late for transfer to be useful).
Revised: Indeed, estimating the maximum in Equation 4 might be as hard as solving both MDPs (which, when it happens, is too late for transfer to be useful).

Original:  T RANSFER USING THE L IPSCHITZ CONTINUITY
Revised:  T RANSFER USING THE L IPSCHITZ CONTINUITY

Original: We use the theoretical results from Section 3 to introduce a  value transfer method, andbuild an algorithm applying this method in the Lifelong RL setting. Value transfer allows to drive the exploration in a new task and accelerate learning.From Proposition 1, one can naturally deﬁne a local upper bound on the optimal Q-function of an MDP given the optimal Q-function of another MDP. Deﬁnition 2.
Revised: A purpose of value transfer, when interacting online with a new MDP, is to initialize the value function and drive the exploration to accelerate learning.

Original: Given two tasks M and ¯ M , for all ( s, a ) ∈ S × A , the Lipschitz upper bound on Q ∗ M induced by Q ∗ ¯ M is deﬁned as U ¯ M ( s, a ) ≥ Q ∗ M ( s, a ) with:

Original: [Equation]

Original: s,

Original: [Equation]

Original: This upper bound allows shrinking the maximum upper bound 11 − γ on the optimal Q-function of an

Original: MDP.

Original: In Lifelong RL, we aim to exploit this property in a method guaranteeing three conditions: C1.
Revised: We aim to exploit value transfer in a method guaranteeing three conditions: C1.

Original: the resulting algorithm is PAC-MDP (Strehl et al., 2009); C2.
Revised: the resulting algorithm is PAC-MDP (Strehl et al., 2009); C2.

Original: the transfer accelerates learning; C3.
Revised: the transfer accelerates learning; C3.

Original: the transfer is non-negative.
Revised: the transfer is non-negative.

Original: To that end, we build on the RMax algorithm (Brafman and 
Revised: From Proposition 1, one can naturally deﬁne a local upper bound on the optimal Q-function of an MDP given the optimal Q-function of another MDP. Deﬁnition 2. Given two tasks M and ¯ M , for all ( s, a ) ∈ S × A , the Lipschitz upper bound on Q ∗ M induced by Q ∗ ¯ M is deﬁned as U ¯ M ( s, a ) ≥ Q ∗ M ( s, a ) with: [Equation]

Original:  Tennenholtz, 2002), which satisﬁes condition C1.
Revised: The optimism in the face of uncertainty principle leads to consider that the long-term expected return from any state is the 11 − γ maximum return, unless proven otherwise. The RMax algorithm (Brafman and Tennenholtz, 2002) in particular explores an MDP so as to shrink this upper bound.

Original: RMax is a model-based, online RL algorithm with PAC-MDP guarantees (Strehl et al., 2009) which means that convergence to near-optimal policy is guaranteed in a polynomial number of steps.
Revised: RMax is a model-based, online RL algorithm with PAC-MDP guarantees (Strehl et al., 2009) which means that convergence to near-optimal policy is guaranteed in a polynomial number of steps.

Original: It relies on an optimistic model initialization that yields an optimistic  Q-function, then explores greedily w.r.t.
Revised: It relies on an optimistic model initialization that yields an optimistic upper bound U on the optimal Q-function, then acts greedily w.r.t.

Original: this Q-function.
Revised: U .

Original: By default, it assigns the maximum upper bound U ( s, a ) = 11 − γon long-term returns estimates Q butcan take advantage of any tighter admissible heuristic.
Revised: By default, it takes the maximum value U ( s, a ) = 11 − γ but any tighter upper bound is admissible.

Original: Thus, shrinking the optimistic upper bound with Equation 5 is expected to improve the learning speed for new tasks in Lifelong RL.
Revised: Thus, shrinking U with Equation 5 is expected to improve the learning speed for new tasks in Lifelong RL.

Original: In RMax, during the resolution of a task M , S × A is split into a subset of known state-action pairs K and its complement K c of unknown pairs.
Revised: In RMax, during the resolution of a task M , S × A is split into a subset of known state-action pairs K and its complement K c of unknown pairs.

Original: A state-action pair is known if the number of collected reward and transition samples allows estimating an (cid:15) -accurate model in L 1 -norm with probability higher than 1 − δ .
Revised: A state-action pair is known if the number of collected reward and transition samples allows estimating an (cid:15) -accurate model in L 1 -norm with probability higher than 1 − δ .

Original: We refer to (cid:15) and δ as the RMax precision parameters .
Revised: We refer to (cid:15) and δ as the RMax precision parameters .

Original: This translates into a threshold n known on the number of visits n ( s, a ) to a pair s, a that are necessary to reach this precision.
Revised: This translates into a threshold n known on the number of visits n ( s, a ) to a pair s, a that are necessary to reach this precision.

Original: Given the experience of a set of m MDPs ¯ M = { ¯ M 1 , . . .
Revised: Given the experience of a set of m MDPs ¯ M = { ¯ M 1 , . . .

Original: , ¯ M m } , we deﬁne the total bound as the minimum over all the Lipschitz bounds induced by each previous MDP.
Revised: , ¯ M m } , we deﬁne the total bound as the minimum over all the Lipschitz bounds induced by each previous MDP.

Original: Proposition 3.
Revised: Proposition 3.

Original: Given a partially known task M = (cid:104) R, T (cid:105) , the set of known state-action pairs K , and the set of Lipschitz bounds on Q ∗ M induced by previous tasks (cid:8)
Revised: Given a partially known task M = (cid:104) R, T (cid:105) , the set of known state-action pairs K , and the set of Lipschitz bounds on Q ∗ M induced by previous tasks (cid:8)

Original: U ¯ M 1 , . .
Revised: U ¯ M 1 , . .

Original: , U ¯ M m (cid:9) , the function Q deﬁned below is an upper bound on Q ∗ M for all s, a ∈ S × A .
Revised: , U ¯ M m (cid:9) , the function Q deﬁned below is an upper bound on Q ∗ M for all s, a ∈ S × A .

Original: [Equation]
Revised: [Equation]

Original: withU ( s, a ) = min (cid:110) 11 − γ , U ¯ M 1 ( s, a [Equation] , U ¯ M m ( s
Revised: with

Original: [Equation]
Revised: [Equation]

Original: a [Equation] (cid:111) . Traditionally in RMax, Equation 6 is solved to a precision (cid:15) Q via Value Iteration.
Revised: Traditionally in RMax, Equation 6 is solved to a precision (cid:15) Q via Value Iteration.

Original: This yields a function Q that is a valid heuristic (provable upper bound on Q ∗ M ) for the exploration of MDP M .
Revised: This yields a function Q that is a valid heuristic (provable upper bound on Q ∗ M ) for the exploration of MDP M .

Original: 4. A TRACTABLE UPPER BOUND ON Q ∗ M
Revised: 4. A TRACTABLE UPPER BOUND ON Q ∗ M

Revised: The key issue addressed in this Section is how to actually compute U ( s, a ) .

Original: Consider two tasks M and ¯ M , on which vanilla RMax has been applied, yielding the respective sets of known state-action pairs K and ¯ K , along with the learned models ˆ M = (cid:104)
Revised: Consider two tasks M and ¯ M , on which vanilla RMax has been applied, yielding the respective sets of known state-action pairs K and ¯ K , along with the learned models ˆ M = (cid:104)

Original: ˆ T, ˆ R (cid:105) and ˆ¯ M = (cid:104) ˆ¯ T, ˆ¯ R (cid:105) , and the upper bounds Q and ¯ Q respectively on Q ∗ M and Q ∗ ¯ M .
Revised: ˆ T, ˆ R (cid:105) and ˆ¯ M = (cid:104) ˆ¯ T, ˆ¯ R (cid:105) , and the upper bounds Q and ¯ Q respectively on Q ∗ M and Q ∗ ¯ M .

Original: Equation 6 allows the transfer of knowledge from ¯ M to M if U ¯ M ( s, a ) can be computed.
Revised: Equation 6 allows the transfer of knowledge from ¯ M to M if U ¯ M ( s, a ) can be computed.

Original: Unfortunately, the true optimal value functions, transition and reward models are unknown.
Revised: Unfortunately, the true optimal value functions, transition and reward models, necessary to compute U ¯ M , are unknown.

Original: Thus, we propose to compute a looser upper bound based on the learned models and value functions.
Revised: Thus, we propose to compute a looser upper bound based on the learned models and value functions.

Revised: First, we provide an upper bound ˆ D M ¯ M on the pseudo metric between models M and ¯ M .

Original: Proposition 4.
Revised: Proposition 4.

Original: Given two tasks M and ¯ M , K and ¯ K the respective sets of state-action pairs where their models are known with accuracy (cid:15) in L 1 -norm with probability at least 1 − δ ,
Revised: Given two tasks M and ¯ M , K and ¯ K the respective sets of state-action pairs where their models are known with accuracy (cid:15) in L 1 -norm with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: with the following deﬁnition of the upper bound on the pseudo-metric between models ˆ D M ¯ M :
Revised: with the following deﬁnition of the upper bound on the pseudo-metric between models ˆ D M ¯ M :

Original: [Equation]
Revised: [Equation]

Original: where B = (cid:15) (cid:0) 1 + γ max s (cid:48) ¯ V ( s (cid:48) )
Revised: where B = (cid:15) (cid:0) 1 + γ max s (cid:48) ¯ V ( s (cid:48) )

Original: (cid:1) .
Revised: (cid:1) .

Original:  ˆ D M ¯ M  can be calculated analytically (see Appendix, Section H).
Revised: This upper bound ˆ D M ¯ M on the distance between MDPs can be calculated analytically (see Appendix, Section H).

Original: The magnitude of the B term is controlled by (cid:15) .
Revised: The magnitude of the B term is controlled by (cid:15) .

Original: In the case where no information is available on the maximum value of ¯ V , B = (cid:15) 1 − γ .
Revised: In the case where no information is available on the maximum value of ¯ V , B = (cid:15) 1 − γ .

Original: (cid:15) measures the accuracy with which the tasks are known: the smaller (cid:15) , the tighter the B bound.
Revised: (cid:15) measures the accuracy with which the tasks are known: the smaller (cid:15) , the tighter the B bound.

Original: Note that ¯ V is used as an upper bound on the true V ∗ ¯ M .
Revised: Note that ¯ V is used as an upper bound on the true V ∗ ¯ M .

Original: In many cases, max s (cid:48) V ∗ ¯ M ( s (cid:48) )
Revised: In many cases, max s (cid:48) V ∗ ¯ M ( s (cid:48) )

Original: (cid:28) 11 − γ ; e.g. for stochastic shortest path problems, which feature rewards only upon reaching terminal states, max s (cid:48)
Revised: (cid:28) 11 − γ ; e.g. for stochastic shortest path problems, which feature rewards only upon reaching terminal states, max s (cid:48)

Original: V ∗ ¯ M ( s (cid:48) )
Revised: V ∗ ¯ M ( s (cid:48) )

Original: = 1 and thus B = (1 + γ ) (cid:15) is a tighter bound for transfer.
Revised: = 1 and thus B = (1 + γ ) (cid:15) is a tighter bound for transfer.

Original: Using ˆ D M ¯ M and Equation 3, one can derive an upper bound ˆ d ¯ MM of d ¯ MM  detailed in Proposition 5.
Revised: Using ˆ D M ¯ M and Equation 3, one can derive an upper bound ˆ d ¯ MM on d ¯ MM , detailed in Proposition 5.

Original: Proposition 5.
Revised: Proposition 5.

Original: Given two tasks M and ¯ M , K the set of state-action pairs for which (cid:104) R, T (cid:105) is known with accuracy (cid:15) in L 1 -norm with probability at least 1 − δ .
Revised: Given two tasks M and ¯ M , K the set of state-action pairs for which (cid:104) R, T (cid:105) is known with accuracy (cid:15) in L 1 -norm with probability at least 1 − δ .

Original: If γ (1 + (cid:15) ) < 1 , the solution ˆ d ¯ MM of the following ﬁxed-point equation of ˆ d is an upper bound on d ¯ MM with probability at least 1 − δ : ˆ d ( s, a ) = ˆ D M ¯ M ( s, a )
Revised: If γ (1 + (cid:15) ) < 1 , the solution ˆ d ¯ MM of the following ﬁxed-point equation on ˆ d is an upper bound on d ¯ MM with probability at least 1 − δ : ˆ [Equation]

Original: +
Revised: a

Original: [Equation]
Revised: [Equation]

Original: Similarly as in Proposition 4, the condition γ (1 + (cid:15) ) < 1 illustrates the fact that for a large return horizon (large γ ), a high accuracy (small (cid:15) ) is needed for the bound to be computable.
Revised: Similarly as in Proposition 4, the condition γ (1 + (cid:15) ) < 1 illustrates the fact that for a large return horizon (large γ ), a high accuracy (small (cid:15) ) is needed for the bound to be computable.

Original: Finally, a tractable upper bound on Q ∗ M given ¯ M with high probability is given by
Revised: Finally, a tractable upper bound on Q ∗ M given ¯ M with high probability is given by

Original: [Equation]
Revised: [Equation]

Original: And the associated upper bound on U ( s, a ) (Equation 6) given previous tasks ¯ M = { ¯ M i } mi
Revised: And the associated upper bound on U ( s, a ) (Equation 6) given previous tasks ¯ M = { ¯ M i } mi

Original: =1 is
Revised: =1 is

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 4. L IPSCHITZ RM AX
Revised: This upper bound can be used to transfer knowledge from a partially solved task to a target task.

Original: In Lifelong RL, MDPs are encountered sequentially. Applying RMax to task M yields the set of known state-action pairs K , the learned models ˆ T and ˆ R , and the upper bound Q on Q ∗ M .
Revised: If ˆ U ( s, a ) ≤ 11 − γ for some ( s, a ) pairs, then the convergence rate can be improved. As complete knowledge of both tasks is not needed, it can be applied online in a Lifelong RL setting.

Original: Saving this information when the task changes allows to compute the upper bound of Equation 10 for the 
Revised: In the next section, we explicit an algorithm that leverages this value transfer method.

Original: Initialize ˆ M = ∅ .
Revised: Initialize ˆ M = ∅ .

Original: for each newly sampled MDP M do
Revised: for each newly sampled MDP M do

Original: Initialize Q ( s, a ) = 11 − γ , ∀ s, a , and K = ∅
Revised: Initialize Q ( s, a ) = 11 − γ , ∀ s, a , and K = ∅

Original: Initialize ˆ T and ˆ R (RMax initialization)
Revised: Initialize ˆ T and ˆ R (RMax initialization)

Original: Q ← UpdateQ ( ˆ M , ˆ T, ˆ R ) for t ∈ [1 , max number of steps ] do s = current state, a = arg max a (cid:48) Q ( s, a (cid:48) )
Revised: Q ← UpdateQ ( ˆ M , ˆ T, ˆ R ) for t ∈ [1 , max number of steps ] do s = current state, a = arg max a (cid:48) Q ( s, a (cid:48) )

Original: Observe reward r and next state s (cid:48) n ( s, a ) ← n ( s, a ) +if n ( s, a ) < n known then
Revised: Observe reward r and next state s (cid:48) n ( s, a ) ← n ( s, a ) +if n ( s, a ) < n known then

Original: Store ( s, a, r, s (cid:48) ) if n ( s, a ) =
Revised: Store ( s, a, r, s (cid:48) ) if n ( s, a ) =

Original: n known then
Revised: n known then

Original: Update K , ˆ T ass (cid:48) and ˆ R as
Revised: Update K , ˆ T ass (cid:48) and ˆ R as

Original: Q ← UpdateQ ( ˆ M , ˆ T, ˆ R )
Revised: Q ← UpdateQ ( ˆ M , ˆ T, ˆ R )

Original: Save ˆ M = (cid:16) ˆ T, ˆ R, K, Q (cid:17) in ˆ M
Revised: Save ˆ M = (cid:16) ˆ T, ˆ R, K, Q (cid:17) in ˆ M

Original: Function UpdateQ ( ˆ M , ˆ T, ˆ R ) : for ¯ M ∈ ¯ M do
Revised: Function UpdateQ ( ˆ M , ˆ T, ˆ R ) : for ¯ M ∈ ¯ M do

Original: Compute ˆ D M ¯ M and ˆ D ¯ MM (Eq. 7)
Revised: Compute ˆ D M ¯ M and ˆ D ¯ MM (Eq. 7)

Original: Compute ˆ d ¯ MM and ˆ d M ¯ M (DP on Eq.
Revised: Compute ˆ d ¯ MM and ˆ d M ¯ M (DP on Eq.

Original: Compute ˆ U ¯ M (Eq. 9)
Revised: Compute ˆ U ¯ M (Eq. 9)

Original: Compute ˆ U (Eq. 10)
Revised: Compute ˆ U (Eq. 10)

Original: Compute and return Q (DP on Eq.
Revised: Compute and return Q (DP on Eq.

Revised: 6 using ˆ U )

Original: 6 using ˆ U ) new task, and to use it to shrink the optimistic heuristic of RMax.
Revised: In Lifelong RL, MDPs are encountered sequentially. Applying RMax to task M yields the set of known state-action pairs K , the learned models ˆ T and ˆ R , and the upper bound Q on Q ∗ M . Saving this information when the task changes allows to compute the upper bound of Equation 10 for the new task, and to use it to shrink the optimistic heuristic of RMax.

Original: This effectively transfers value functions between tasks based on task similarity.
Revised: This effectively transfers value functions between tasks based on task similarity.

Original: As the new task is explored, the task similarity is  assessed with better conﬁdence, reﬁning the values of ˆ D M ¯ M , ˆ d ¯ MM and eventually ˆ U , allowing for more efﬁcient transfer where the task similarity is appraised.
Revised: As the new task is explored online, the task similarity is progressively assessed with better conﬁdence, reﬁning the values of ˆ D M ¯ M , ˆ d ¯ MM and eventually ˆ U , allowing for more efﬁcient transfer where the task similarity is appraised.

Original: The resulting algorithm,Lipschitz
Revised: The resulting algorithm,

Original:  RMax (LRMax), is presented in Algorithm 1.
Revised: Lipschitz RMax (LRMax), is presented in Algorithm 1.

Original: To avoid ambiguities with ¯ M , we use ˆ M to store learned features ( ˆ T , ˆ R , K and Q ) about previous MDPs.
Revised: To avoid ambiguities with ¯ M , we use ˆ M to store learned features ( ˆ T , ˆ R , K , Q ) about previous MDPs.

Original: In a nutshell, the behaviour of LRMax on a given task M is precisely that of RMax, but with a tighter admissible heuristic ˆ U that becomes better as the new task is explored (while this heuristic remains constant in vanilla RMax).
Revised: In a nutshell, the behavior of LRMax on a given task M is precisely that of RMax, but with a tighter admissible heuristic ˆ U that becomes better as the new task is explored (while this heuristic remains constant in vanilla RMax).

Original: LRMax is PAC-MDP (Condition C1) as stated in Properties 6 and 7 below.
Revised: LRMax is PAC-MDP (Condition C1) as stated in Propositions 6 and 7 below.

Original: With S = |S| and A = |A| , the sample complexity of vanilla RMax is ˜ O
Revised: With S = |S| and A = |A| , the sample complexity of vanilla RMax is ˜ O

Original: ( S 2 A/ ( (cid:15) 3 (1 − γ ) 3 )) , which is improved by LRMax in
Revised: ( S 2 A/ ( (cid:15) 3 (1 − γ ) 3 )) , which is improved by LRMax in

Original: Proposition 6 and meets Condition C2.
Revised: Proposition 6 and meets Condition C2.

Original: Finally ˆ U is a proved upper bound with high probability on
Revised: Finally ˆ U is a proved upper bound with high probability on

Original: Q ∗ M , which avoids negative transfer and meets Condition C3.
Revised: Q ∗ M , which avoids negative transfer and meets Condition C3.

Original: Proposition 6 (Sample complexity (Strehl et al., 2009)) .
Revised: Proposition 6 (Sample complexity (Strehl et al., 2009)) .

Original: With probability 1 − δ , the greedy policy w.r.t.
Revised: With probability 1 − δ , the greedy policy w.r.t.

Original: Q computed by LRMax achieves an (cid:15) -optimal return in MDP M for all but (when logarithmic factors are ignored)
Revised: Q computed by LRMax achieves an (cid:15) -optimal return in MDP M after

Original: [Equation]
Revised: [Equation]

Original: time steps, with ˆ U deﬁned in Equation 10 a non-static, decreasing quantity, upper bounded by 11 − γ .
Revised: samples (when logarithmic factors are ignored), with ˆ U deﬁned in Equation 10 a non-static, decreasing quantity, upper bounded by 11 − γ .

Original: Consequently from Proposition 6, the sample complexity of LRMax is no worse than that of RMax.
Revised: Consequently from Proposition 6, the sample complexity of LRMax is no worse than that of RMax.

Original: Proposition 7 (Computational complexity) .
Revised: Proposition 7 (Computational complexity) .

Original: The total computational complexity of Lipschitz RMax is
Revised: The total computational complexity of Lipschitz RMax is

Original: [Equation]
Revised: [Equation]

Original: with T the number of time steps, (cid:15) Q the precision of value iteration and N the number of tasks.
Revised: with τ the number of interaction steps, (cid:15) Q the precision of value iteration and N the number of tasks.

Original: 4. R EFINING LRM AX BOUNDS WITH MAXIMUM MODEL DISTANCE
Revised: 4. R EFINING LRM AX BOUNDS WITH MAXIMUM MODEL DISTANCE

Original: LRMax relies on upper bounds on the local distances between tasks (Equation 8).
Revised: LRMax relies on upper bounds on the local distances between tasks (Equation 8).

Original: The quality of the Lipschitz bound on Q ∗ M greatly depends on the quality of those estimates and can be improved accordingly.
Revised: The quality of the Lipschitz bound on Q ∗ M greatly depends on the quality of those estimates and can be improved accordingly.

Original: We discuss two methods to provide ﬁner estimates.
Revised: We discuss two methods to provide ﬁner estimates.

Original: First, from the deﬁnition of D M ¯ M γV ∗ ¯ M ( s, a ) , it is easy to show that model pseudo-distances are always upper bounded by 1+ γ− γ .
Revised: First, from the deﬁnition of D M ¯ M γV ∗ ¯ M ( s, a ) , it is easy to show that model pseudo-distances are always upper bounded by 1+ γ− γ .

Original: However, in practice, the tasks experienced in Lifelong RL might not cover the full span of possible MDPs and may be systematically closer to each other than 1+ γ 1 − γ .
Revised: However, in practice, the tasks experienced in Lifelong RL might not cover the full span of possible MDPs and may be systematically closer to each other than 1+ γ 1 − γ .

Original: For instance, the distances between variations of the Breakout video game are much smaller than 1+ γ 1 − γ .

Original: More generally, the distance between two games in the Arcade Learning Environment (ALE) (Bellemare et al., 2013), isalso smaller than the maximum distance between any two MDPs deﬁned on the common state-action space of the ALE.
Revised: For instance, the distance between two games in the Arcade Learning Environment (ALE) (Bellemare et al., 2013), is smaller than the maximum distance between any two MDPs deﬁned on the common state-action space of the ALE (extended discussion in Appendix, Section L).

Original: Let D max ( s, a ) (cid:44) max M, ¯ M ∈M 2 { D M ¯ M γV ∗ ¯ M ( s, a ) } be the maximum model distance at a particular s, a pair.
Revised: Let D max ( s, a ) (cid:44) max M, ¯ M ∈M 2 { D M ¯ M γV ∗ ¯ M ( s, a ) } be the maximum model distance at a particular s, a pair.

Original: Prior knowledge might indicate a smaller upper bound for D max ( s, a ) than 1+ γ 1 − γ .
Revised: Prior knowledge might indicate a smaller upper bound for D max ( s, a ) than 1+ γ 1 − γ .

Original: We will note such an upper bound D max .
Revised: We will note such an upper bound D max .

Original: Solving Equation 8 boils down to accumulating ˆ D M ¯ M ( s, a ) values in ˆ d ( s, a ) .
Revised: Solving Equation 8 boils down to accumulating ˆ D M ¯ M ( s, a ) values in ˆ d ( s, a ) .

Original: Reducing a ˆ D M ¯ M ( s, a ) estimate in a single ( s, a ) pair actually reduces ˆ d ( s, a ) in all ( s, a ) pairs.
Revised: Reducing a ˆ D M ¯ M ( s, a ) estimate in a single ( s, a ) pair actually reduces ˆ d ( s, a ) in all ( s, a ) pairs.

Original: Thus, replacing ˆ D M ¯ M ( s, a ) in Equation 8 by min { D max , ˆ D M ¯ M ( s, a ) } , provides amuch smaller upper bound ˆ d ¯ MM on d ¯ MM , and thus a smaller ˆ U which allows transfer if it is lesser than 11 − γ .
Revised: Thus, replacing ˆ D M ¯ M ( s, a ) in Equation 8 by min { D max , ˆ D M ¯ M ( s, a ) } , provides a smaller upper bound ˆ d ¯ MM on d ¯ MM , and thus a smaller ˆ U which allows transfer if it is lesser than 11 − γ .

Original: Consequently, such an upper bound D max can make a difference between successful and unsuccessful transfer, even if its value is of little importance.
Revised: Consequently, such an upper bound D max can make a difference between successful and unsuccessful transfer, even if its value is of little importance.

Original: Conversely, setting a value for D max quantiﬁes the distance between MDPs where transfer is efﬁcient.
Revised: Conversely, setting a value for D max quantiﬁes the distance between MDPs where transfer is efﬁcient.

Original: Furthermore, one can estimate online the value of D max ( s, a ) , lifting the previous hypothesis of available prior knowledge.
Revised: Furthermore, one can estimate online the value of D max ( s, a ) , lifting the previous hypothesis of available prior knowledge.

Original: One can build an empirical estimate of the maximum model distance at s, a : ˆ D max ( s, a ) (cid:44) max M,
Revised: One can build an empirical estimate of the maximum model distance at s, a : ˆ D max ( s, a ) (cid:44) max M,

Original: ¯ M ∈
Revised: ¯ M ∈

Original: ˆ M 2 { ˆ D M ¯ M ( s, a ) } , ˆ M being the set of explored tasks.
Revised: ˆ M 2 { ˆ D M ¯ M ( s, a ) } , ˆ M being the set of explored tasks.

Original: The pitfall being that, with few explored tasks, ˆ D max ( s, a ) could underestimate D max ( s, a ) .
Revised: The pitfall being that, with few explored tasks, ˆ D max ( s, a ) could underestimate D max ( s, a ) .

Original: Propositionprovides a lower bound on the probability that ˆ D max ( s, a ) does not underestimate D max ( s, a ) , depending on the number of sampled tasks.
Revised: Proposition 8 provides a lower bound on the probability that ˆ D max ( s, a ) does not underestimate D max ( s, a ) , depending on the number of sampled tasks.

Original: In turn this indicates when ˆ D max ( s, a )is an upper bound on D max ( s, a ) with high probability, which can be combined with Algorithm 1 to improve the performance.
Revised: In turn this indicates when ˆ D max ( s, a ) upper bounds D max ( s, a ) with high probability, which can be combined with Algorithm 1 to improve the performance.

Original: Proposition 8.
Revised: Proposition 8.

Original: Consider an algorithm producing (cid:15) -accurate in L 1 -norm model estimates with probability at least 1 − δ for a subset of S
Revised: Consider an algorithm producing (cid:15) -accurate in L 1 -norm model estimates with probability at least 1 − δ for a subset of S

Original: ×
Revised: ×

Original: A after interacting with an MDP.
Revised: A after interacting with an MDP.

Original: For all s, a ∈ S × A , after sampling m tasks with p min = min M ∈M Pr ( M ) , the following lower bound holds:
Revised: For all s, a ∈ S × A , after sampling m tasks with p min = min M ∈M Pr ( M ) , the following lower bound holds:

Original: [Equation]
Revised: [Equation]

Original: The assumption of a lower bound p min on the sampling probability of a task implies that M is ﬁnite and is commonly seen as a non-adversarial task sampling strategy (Abel et al., 2018).
Revised: The assumption of a lower bound p min on the sampling probability of a task implies that M is ﬁnite and is commonly seen as a non-adversarial task sampling strategy (Abel et al., 2018).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The experiments reported here 1 illustrate how 1) LRMax allows for early performance increase in Lifelong RL by efﬁciently transferring knowledge between tasks; 2) the Lipschitz bound of
Revised: The experiments reported here 1 illustrate how 1) LRMax allows for early performance increase in Lifelong RL by efﬁciently transferring knowledge between tasks; 2) the Lipschitz bound of

Original: Equation 9 improves the sample complexity compared to RMax by providing a tighter upper bound on Q ∗ .
Revised: Equation 9 improves the sample complexity compared to RMax by providing a tighter upper bound on Q ∗ .

Original: Graphs are displayed with 95% conﬁdence intervals.
Revised: Graphs are displayed with 95% conﬁdence intervals.

Original: Information in line with the Machine
Revised: Information in line with the Machine

Original: Learning Reproducibility Check-list (Pineau, 2019) is documented in the Appendix, Section O.
Revised: Learning Reproducibility Check-list (Pineau, 2019) is documented in the Appendix, Section Q.

Original: We evaluate different variants of LRMax in a Lifelong RL experiment.
Revised: We evaluate different variants of LRMax in a Lifelong RL experiment.

Original: The RMax algorithm will be used as a no-transfer baseline.
Revised: The RMax algorithm will be used as a no-transfer baseline.

Original: LRMax( x ) denotes Algorithm 1 with prior D max = x . MaxQInit denotes the M AX QI NIT algorithm from Abel et al.
Revised: LRMax( x ) denotes Algorithm 1 with prior D max = x . MaxQInit denotes the M AX QI NIT algorithm from Abel et al.

Original: (2018), consisting in a state-of-the art PAC-MDP algorithm achieving transferin a non adversarial setting with PAC guarantees.
Revised: (2018), consisting in a state-of-the art PAC-MDP algorithm achieving transfer with PAC guarantees.

Original: Both LRMax and MaxQInit algorithms achieve value transfer by providing a tighter upper-bound on Q ∗ than 11 − γ .
Revised: Both LRMax and MaxQInit algorithms achieve value transfer by providing a tighter upper-bound on Q ∗ than 11 − γ .

Original: Computing both upper-bounds and taking the minimum results in combining the two approaches.
Revised: Computing both upper-bounds and taking the minimum results in combining the two approaches.

Original: We include such a combination inour study with the LRMaxQInit algorithm. Similarly, LRMaxQInit( x ) consists in the latter algorithm, beneﬁting from prior knowledge D max = x . The environment we used in all experiments is a variant of the “tight” environment used by Abel et al. This is a 11 × 11 grid-world, the initial state is in the centre, actions are the cardinal
Revised: We include such a combination in

Original: [Figure]
Revised: [Figure]

Original: v

Original: [Figure]

Original: l a t i v e D
Revised: l a t i v e D

Original: i s c
Revised: i s c

Original: [Figure]
Revised: [Figure]

Original: un t e dR
Revised: t e d

Original: [Figure]
Revised: [Figure]

Original: t u r n
Revised: t u r n

Original: [Figure]
Revised: [Figure]

Original: l a t i v e d
Revised: l a t i v e d

Original: i s c o un t e d r e t u r n
Revised: i s c o un t e d r e t u r n

Original: [Figure]
Revised: [Figure]

Original:  moves (Appendix, Section L).
Revised: our study with the LRMaxQInit algorithm. Similarly, LRMaxQInit( x ) consists in the latter algorithm, beneﬁting from prior knowledge D max = x . The environment we used in all experiments is a variant of the “tight” environment used by Abel et al. This is a 11 × 11 grid-world, the initial state is in the centre, actions are the cardinal moves (Appendix, Section M).

Original: The reward is zero everywhere except for the three goal cells in the upper-right corner.
Revised: The reward is zero everywhere except for the three goal cells in the upper-right corner.

Original: Each time a task is sampled, a new reward value is drawn from [0 . 8 , 1] for each of the three goal cells and a probability of slipping (performing a different action than the one selected) is picked in [0 , 0 . 1] .
Revised: Each time a task is sampled, a new reward value is drawn from [0 . 8 , 1] for each of the three goal cells and a probability of slipping (performing a different action than the one selected) is picked in [0 , 0 . 1] .

Original: Hence, tasks have different reward and transition functions.
Revised: Hence, tasks have different reward and transition functions.

Original: We sampletasks in sequence among a pool of 5 possible different sampled tasks.
Revised: We sampletasks in sequence among a pool of 5 possible different sampled tasks.

Original: Each is run for 2000 episodes of length 10.
Revised: Each is run for 2000 episodes of length 10.

Original: The operation is repeated 10 times to provide narrow conﬁdence intervals.
Revised: The operation is repeated 10 times to provide narrow conﬁdence intervals.

Original: We used n known = 10 , δ = 0 .
Revised: We used n known = 10 , δ = 0 .

Original: 05 and (cid:15) = 0 .
Revised: 05 and (cid:15) = 0 .

Original: 01 (discussion in Appendix, Section N).
Revised: 01 (discussion in Appendix, Section P).

Original: We drew tasks from a ﬁnite set of ﬁve MDPs.
Revised: We drew tasks from a ﬁnite set of ﬁve MDPs.

Original: This allows the application of MaxQInit and the subsequent comparison below.
Revised: This allows the application of MaxQInit and the subsequent comparison below.

Original: Note, however, that LRMax does not require the set of MDPs to be ﬁnite, which is a noticeable advantage in applicability.
Revised: Note, however, that LRMax does not require the set of MDPs to be ﬁnite, which is a noticeable advantage in applicability.

Revised: Other lifelong RL experiments are reported in the Appendix, Section N.

Original: The results are reported in Figure 1.
Revised: The results are reported in Figure 1.

Original: Figure 1a displays the discounted return for each task, averaged across episodes.
Revised: Figure 1a displays the discounted return for each task, averaged across episodes.

Original: Similarly, Figure 1b displays the discounted return for each episode, averaged across tasks (same color code as Figure 1a).
Revised: Similarly, Figure 1b displays the discounted return for each episode, averaged across tasks (same color code as Figure 1a).

Original: Figure 1c displays the discounted return for ﬁve speciﬁc instances, detailed below.
Revised: Figure 1c displays the discounted return for ﬁve speciﬁc instances, detailed below.

Original: To avoid inter-task disparities, all the aforementioned discounted returns are displayed relatively to an estimator of the optimal expected return for each task.
Revised: To avoid inter-task disparities, all the aforementioned discounted returns are displayed relatively to an estimator of the optimal expected return for each task.

Original: For readability purposes, Figures 1b and 1c display a moving average over 100 episodes.
Revised: For readability purposes, Figures 1b and 1c display a moving average over 100 episodes.

Original: Figure 1d reports the beneﬁts of various values of D max on the algorithmic properties.
Revised: Figure 1d reports the beneﬁts of various values of D max on the algorithmic properties.

Original: In Figure 1a, we ﬁrst observe that LRMax beneﬁts from the transfer method, as the average discounted return increases as more tasks are experienced.
Revised: In Figure 1a, we ﬁrst observe that LRMax beneﬁts from the transfer method, as the average discounted return increases as more tasks are experienced.

Original: Moreover, this advantage appears as early as the second task.
Revised: Moreover, this advantage appears as early as the second task.

Original: Conversely, the MaxQInit algorithm needs to wait for task 12 before beneﬁting from transfer.
Revised: Conversely, the MaxQInit algorithm needs to wait for task 12 before beneﬁting from transfer.

Original: As suggested in Section 4.3, various amounts of prior allow the LRMax transfer method to be more or less efﬁcient: a smaller known upper-bound D max on ˆ D M ¯ M causes a larger discounted return gain.
Revised: As suggested in Section 4.3, various amounts of prior allow the LRMax transfer method to be more or less efﬁcient: a smaller known upper-bound D max on ˆ D M ¯ M causes a larger discounted return gain.

Original: Combining both approaches in the LRMaxQInit algorithm outperforms all other methods.
Revised: Combining both approaches in the LRMaxQInit algorithm outperforms all other methods.

Original: Episode-wise, we observe in Figure 1b that the LRMax transfer method allows for faster convergence, hence decreases the sample complexity.
Revised: Episode-wise, we observe in Figure 1b that the LRMax transfer method allows for faster convergence, hence decreases the sample complexity.

Original: Interestingly, LRMax features three stages in the learning process.
Revised: Interestingly, LRMax features three stages in the learning process.

Original: The ﬁrst episodes are characterized by a direct exploitation of the transferred knowledge, causing these episodes to yield high payoff.
Revised: The ﬁrst episodes are characterized by a direct exploitation of the transferred knowledge, causing these episodes to yield high payoff.

Original: This is due to the combined facts that the Lipschitz bound of Equation 9 is larger on promising regions of S × A seen on previous tasks and the fact that LRMax acts greedily w.r.t.
Revised: This is due to the combined facts that the Lipschitz bound of Equation 9 is larger on promising regions of S × A seen on previous tasks and the fact that LRMax acts greedily w.r.t.

Original: that bound.
Revised: that bound.

Original: This high performance regime is followed by the exploration of unknown regions of S × A , in our case yielding low returns.
Revised: This high performance regime is followed by the exploration of unknown regions of S × A , in our case yielding low returns.

Original: Indeed, as promising regions are explored ﬁrst, the bound becomes tighter for the corresponding state-action pairs, enough for the Lipschitz bound of unknown pairs to become larger, thus driving the exploration towards low payoff regions.
Revised: Indeed, as promising regions are explored ﬁrst, the bound becomes tighter for the corresponding state-action pairs, enough for the Lipschitz bound of unknown pairs to become larger, thus driving the exploration towards low payoff regions.

Original: Such regions are quickly identiﬁed and never revisited thereafter.
Revised: Such regions are quickly identiﬁed and never revisited thereafter.

Original: Eventually,
Revised: Eventually,

Original: LRMax stops exploring and converges to the optimal policy.
Revised: LRMax stops exploring and converges to the optimal policy.

Original: Importantly, in all experiments, LRMax never features negative transfer as supported by the provability of the Lipschitz upper-bound with high probability.
Revised: Importantly, in all experiments, LRMax never features negative transfer as supported by the provability of the Lipschitz upper-bound with high probability.

Original: This is indeed demonstrated by the fact that it is at least as efﬁcient in learning as the no-transfer RMax baseline.
Revised: This is indeed demonstrated by the fact that it is at least as efﬁcient in learning as the no-transfer RMax baseline.

Original: Figure 1c displays the collected returns of RMax, LRMax(0.1), and MaxQInit for speciﬁc tasks.
Revised: Figure 1c displays the collected returns of RMax, LRMax(0.1), and MaxQInit for speciﬁc tasks.

Original: We observe that LRMax beneﬁts from the transfer as early as task 2, where the aforementioned 3-stages behavior is visible.
Revised: We observe that LRMax beneﬁts from the transfer as early as task 2, where the aforementioned 3-stages behavior is visible.

Original: Again, MaxQInit needs to wait for task 12 to leverage the transfer method.
Revised: Again, MaxQInit needs to wait for task 12 to leverage the transfer method.

Original: However, the bound it provides are tight enough to allow for almost zero exploration of the task.
Revised: However, the bound it provides are tight enough to allow for almost zero exploration of the task.

Original: In Figure 1d, we display the following quantities for various values of D max : ρ Lip , is the ratio of the time the Lipschitz bound was tighter than the RMax bound 11 − γ ; ρ Speed − up , is the relative gain of time steps before convergence when comparing LRMax to RMax.
Revised: In Figure 1d, we display the following quantities for various values of D max : ρ Lip , is the ratio of the time the Lipschitz bound was tighter than the RMax bound 11 − γ ; ρ Speed − up , is the relative gain of time steps before convergence when comparing LRMax to RMax.

Original: This quantity is estimated based on the last updates of the empirical model ¯ M ; ρ Return , is the relative total return gain on 2000 episodes of LRMax w.r.t.
Revised: This quantity is estimated based on the last updates of the empirical model ¯ M ; ρ Return , is the relative total return gain on 2000 episodes of LRMax w.r.t.

Original: RMax.
Revised: RMax.

Original: First, we observe an increase of ρ Lip as D max becomes tighter.
Revised: First, we observe an increase of ρ Lip as D max becomes tighter.

Original: This means that the Lipschitz bound of Equation 9 becomes effectively smaller than 11 − γ .
Revised: This means that the Lipschitz bound of Equation 9 becomes effectively smaller than 11 − γ .

Original: This phenomenon leads to faster convergence, indicated by ρ Speed − up .
Revised: This phenomenon leads to faster convergence, indicated by ρ Speed − up .

Original: Eventually, this increased convergence rate allows for a net total return gain, illustrated by the increase of ρ Return .
Revised: Eventually, this increased convergence rate allows for a net total return gain, illustrated by the increase of ρ Return .

Original: Overall, in this analysis, we have showed that LRMax beneﬁts from an enhanced sample complexity thanks to theused value transfer method.
Revised: Overall, in this analysis, we have showed that LRMax beneﬁts from an enhanced sample complexity thanks to the value transfer method.

Original: The knowledge of a prior D max further increases this beneﬁt.
Revised: The knowledge of a prior D max further increases this beneﬁt.

Original: The method is comparable to the MaxQInit method and has some advantages such as the early ﬁtness for use and the applicability to inﬁnite sets of tasks.
Revised: The method is comparable to the MaxQInit method and has some advantages such as the early ﬁtness for use and the applicability to inﬁnite sets of tasks.

Original: Moreover, the transfer is non-negative while preserving the PAC-MDP guarantees of the algorithm.
Revised: Moreover, the transfer is non-negative while preserving the PAC-MDP guarantees of the algorithm.

Original: Additionally to the analysis performed here, we show in the Appendix, Section M that, when provided with any prior knowledge D max , LRMax increasingly stops using this prior as the task is explored.
Revised: Additionally to the analysis performed here, we show in the Appendix, Section O that, when provided with any prior knowledge D max , LRMax increasingly stops using this prior as the task is explored.

Original: This conﬁrms the claim of section 4.3 that providing D max enables transfer even if it’s value is of little importance.
Revised: This conﬁrms the claim of section 4.3 that providing D max enables transfer even if it’s value is of little importance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have studied theoretically the Lipschitz continuity property of the optimal Q-function in the MDP space.
Revised: We have studied theoretically the Lipschitz continuity property of the optimal Q-function in the MDP space.

Original: This led to a local Lipschitz continuity result, establishing that thedistance between the optimal Q-functions of two  MDPs at the same state-action pair is upper bounded by a local (state-action dependent) distance between MDPs.
Revised: This led to a local Lipschitz continuity result, establishing that the optimal Q-functions of two close MDPs are themselves close to each other.

Original: Thislocal distance  can be computed by dynamic programming.
Revised: This distance between Q-functions can be computed by Dynamic Programming.

Original: A consequence of this result is a global Lipschitz continuity property of the optimal Q-function in the MDP space w.r.t.

Original: a pseudo metric between MDPs.

Original: We then proposed a value-transfer method using the local continuity property with the Lipschitz RMax algorithm, practically implementing this approach in the Lifelong RL setting.
Revised: We then proposed a value-transfer method using this continuity property with the Lipschitz RMax algorithm, practically implementing this approach in the Lifelong RL setting.

Original: The algorithm preserves PAC-MDP guarantees, accelerates the learning in subsequent tasks and performs non-negative transfer.
Revised: The algorithm preserves PAC-MDP guarantees, accelerates the learning in subsequent tasks and performs non-negative transfer.

Original: Potential improvements of the algorithm were discussed in the form of prior knowledge introduction on the maximum distance between models and online estimation with high probability of this distance.
Revised: Potential improvements of the algorithm were discussed in the form of prior knowledge introduction on the maximum distance between models and online estimation with high probability of this distance.

Original: We showcased the algorithm in lifelong RL experiments and demonstrated empirically its ability to accelerate learning.
Revised: We showcased the algorithm in lifelong RL experiments and demonstrated empirically its ability to accelerate learning.

Original: The results also conﬁrm that no negative transfer occurs, regardless of parameter settings.
Revised: The results also conﬁrm that no negative transfer occurs, regardless of parameter settings.

Original: It should be noted that our approach can directly extend other PAC-MDP algorithms (Szita and Szepesv´ari, 2010; Rao and Whiteson, 2012;
Revised: It should be noted that our approach can directly extend other PAC-MDP algorithms (Szita and Szepesv´ari, 2010; Rao and Whiteson, 2012;

Original: Pazis et al., 2016; Dann et al., 2017) to the Lifelong setting.
Revised: Pazis et al., 2016; Dann et al., 2017) to the Lifelong setting.

Revised: In hindsight, we believe this contribution provides a sound basis to non-negative value transfer via MDP similarity, a development that was lacking in the literature.

Revised: Key insights for the practitioner lie both in the theoretical analysis and in the practical derivation of a transfer scheme that achieves non-negative transfer with PAC guarantees.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: Omitted for anonymity.
Revised: Omitted for anonymity.

Original: A PPENDIX A A NEGATIVE TRANSFER EXAMPLE
Revised: A PPENDIX A A NEGATIVE TRANSFER EXAMPLE

Original: In their paper, Song et al.
Revised: In their paper, Song et al.

Original: (2016) propose two transfer methods based on the metric between MDPs they introduce, stemming from the bi-simulation metric introduced by Ferns et al.
Revised: (2016) propose two transfer methods based on the metric between MDPs they introduce, stemming from the bi-simulation metric introduced by Ferns et al.

Original: The intuition is that, for a new target task, the value function of the closest source task in terms of that metric is used as an initialisation.
Revised: The intuition is that, for a new target task, the value function of the closest source task in terms of that metric is used as an initialization.

Original: However, if no similar source task is available, using the closest task’s value function as an initialization can lead to negative transfer.
Revised: However, if no similar source task is available, using the closest task’s value function as an initialization can lead to negative transfer.

Original: We here understand negative transfer as the fact that it prevents a learning algorithm to converge to the optimal policy while interacting with a new task.
Revised: We here understand negative transfer as the fact that it prevents a learning algorithm to converge to the optimal policy while interacting with a new task.

Original: We make the hypothesis that the learning algorithm acts greedily w.r.t.
Revised: We make the hypothesis that the learning algorithm acts greedily w.r.t.

Original: the current Q-value function.
Revised: the current Q-value function.

Original: This is for example the behaviour of the RMax algorithm (Brafman and Tennenholtz, 2002).
Revised: This is for example the behavior of the RMax algorithm (Brafman and Tennenholtz, 2002).

Original: We now illustrate a negative transfer case with an example.
Revised: We now illustrate a negative transfer case with an example.

Original: Let us consider the 2-states MDP of Figure 2.
Revised: Let us consider the 2-states MDP of Figure 2.

Original: We assume that the transitions are deterministic and the initial state is
Revised: We assume that the transitions are deterministic and the initial state is

Original: [Figure]
Revised: [Figure]

Original: [Equation]

Original: [Figure]

Original: always s 0 .
Revised: always s 0 .

Original: In the ﬁrst MDP M 1 ∈ M , the reward is 0 everywhere except for R
Revised: In the ﬁrst MDP M 1 ∈ M , the reward is 0 everywhere except for R

Original: a 0 s 0 = 1 .
Revised: a 0 s 0 = 1 .

Original: In the second
Revised: In the second

Original: MDP M 2 ∈ M , the reward is 0 everywhere except for R a 1 s 1 = 1 .
Revised: MDP M 2 ∈ M , the reward is 0 everywhere except for R a 1 s 1 = 1 .

Original: With a discount factor γ = 0 .
Revised: With a discount factor γ = 0 .

Original: 9 , the value functions and Q-functions of both MDPs are summarized in Table 3 Using the weighted
Revised: 9 , the value functions and Q-functions of both MDPs are summarized in Table 3 Using the weighted

Original: V∗ M [Equation]
Revised: V

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Revised: V ∗ M 2 ( · )

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: a
Revised: M 2 (

Original: [Equation]
Revised: [Equation]

Original: V ∗ M 2 ( · ) Q ∗ M 2 ( · , a 0 ) 
Revised: a 0 ) Q

Original: Q ∗ M 2 ( · , a 1 ) s  8.
Revised: [Equation] M 2 ( · , a 1 ) s  8.

Original: s 8. transfer technique from M 1 to M 2 proposed by Song et al.
Revised: s 8. transfer technique from M 1 to M 2 proposed by Song et al.

Original: (Deﬁnition 4.1), the Q-function described below is used as an initialization for the exploration of M 2 .
Revised: (Deﬁnition 4.1), the Q-function described below is used as an initialization for the exploration of M 2 .

Original: [Equation]
Revised: [Equation]

Original: First, Q transfer Mdoes not respect the principle of “optimism under the face of uncertainty” that often results in sound and efﬁcient exploration (Strehl et al., 2009; Brafman and Tennenholtz, 2002; Sutton and Barto, 1998).
Revised: First, Q transfer Mdoes not respect the principle of “optimism under the face of uncertainty” that often results in sound and efﬁcient exploration (Strehl et al., 2009; Brafman and Tennenholtz, 2002; Sutton and Barto, 1998).

Original: Further, a greedy policy w.r.t.
Revised: Further, a greedy policy w.r.t.

Original: Q transfer Mwould never discover the state-action pair s 1 , a 1 in M 2 which is the maximum-reward pair.
Revised: Q transfer Mwould never discover the state-action pair s 1 , a 1 in M 2 which is the maximum-reward pair.

Original: Instead, the agent would go from s 0 to s 1 and perform self-loops thereafter.
Revised: Instead, the agent would go from s 0 to s 1 and perform self-loops thereafter.

Original: As a conclusion, this negative transfer example motivates the need for distance between MDPs not only to account for the best-source task to use for transfer but also to discourage the transfer when the distance is too high.
Revised: As a conclusion, this negative transfer example motivates the need for distance between MDPs not only to account for the best-source task to use for transfer but also to discourage the transfer when the distance is too high.

Original: The approach we develop in this paper used the distance to build optimistic upper-bounds on the Q-function.
Revised: The approach we develop in this paper used the distance to build optimistic upper-bounds on the Q-function.

Original: Those upper-bounds are simply of no use when the distance is too high which is equivalent as avoiding transfer.
Revised: Those upper-bounds are simply of no use when the distance is too high which is equivalent as avoiding transfer.

Original: B D ISCUSSION ON METRICS AND RELATED NOTIONS
Revised: B D ISCUSSION ON METRICS AND RELATED NOTIONS

Original: A metric on a set X is a function m : X × X
Revised: A metric on a set X is a function m : X × X

Original: → R which has the following properties for any x, y, z ∈ X :
Revised: → R which has the following properties for any x, y, z ∈ X :

Original: m ( x, y ) ≥ 0 , 2.
Revised: m ( x, y ) ≥ 0 , 2.

Original: m ( x, y ) = 

Revised: m ( x, y ) =

Original: ⇔ x = y , 3.
Revised: ⇔ x = y , 3.

Original: m ( x, y )
Revised: m ( x, y )

Original: = m ( y, x ) , 4.
Revised: = m ( y, x ) , 4.

Original: m ( x, z ) ≤ m ( x, y )
Revised: m ( x, z ) ≤ m ( x, y )

Original: + m ( y, z ) .
Revised: + m ( y, z ) .

Original: With only m ( x, x ) = 0 instead of property 2, m would be a pseudo-metric .
Revised: With only m ( x, x ) = 0 instead of property 2, m would be a pseudo-metric .

Original: Without property 3, one has a quasi-metric .
Revised: Without property 3, one has a quasi-metric .

Original: Without property 3 and 4, and when X is a set of probability measures, one has a divergence .
Revised: Without property 3 and 4, and when X is a set of probability measures, one has a divergence .

Original: In Deﬁnition 1, D ¯ MM,f ( s, a ) is indeed a pseudo-metric over MDPs since the choice of f can lead to a zero distance between different models.
Revised: In Deﬁnition 1, D ¯ MM,f ( s, a ) is indeed a pseudo-metric over MDPs since the choice of f can lead to a zero distance between different models.

Original: The local MDP dissimilarity between MDPs d ¯ MM ( s, a ) of Proposition 1 does not respect propertiesand 3, hence the name dissimilarity .
Revised: The local MDP dissimilarity between MDPs d ¯ MM ( s, a ) of Proposition 1 does not respect propertiesand 3, hence the name dissimilarity .

Original: The ∆ ¯ MM ( s, a ) (cid:44) min (cid:110)
Revised: The ∆ ¯ MM ( s, a ) (cid:44) min (cid:110)

Original: d ¯ MM ( s, a ) , d M ¯ M ( s, a ) (cid:111) quantity, however, regains property 3 and is hence a pseudo-metric.
Revised: d ¯ MM ( s, a ) , d M ¯ M ( s, a ) (cid:111) quantity, however, regains property 3 and is hence a pseudo-metric.

Original: An important consequence is that Propositionis “in the spirit” of a Lipschitz continuity theorem but cannot be called as such, hence the name pseudo-Lipschitz continuity .
Revised: An important consequence is that Propositionis “in the spirit” of a Lipschitz continuity theorem but cannot be called as such, hence the name pseudo-Lipschitz continuity .

Original: The same goes for the global dissimilarity d ¯ MM = 11 − γ max s,a ∈S×A (cid:104) D ¯ MM,γV ∗ ¯ M ( s, a ) (cid:105) .
Revised: The same goes for the global dissimilarity d ¯ MM = 11 − γ max s,a ∈S×A (cid:104) D ¯ MM,γV ∗ ¯ M ( s, a ) (cid:105) .

Original: However, using min (cid:110)
Revised: However, using min (cid:110)

Original: d ¯ MM , d M ¯ M (cid:111) allows to regain property 3 and makes this quantity a pseudo-metric again between MDPs.
Revised: d ¯ MM , d M ¯ M (cid:111) allows to regain property 3 and makes this quantity a pseudo-metric again between MDPs.

Original: C P ROOF OF P ROPOSITION
Revised: C P ROOF OF P ROPOSITION

Original: Lemma 1.
Revised: Lemma 1.

Original: Given two MDPs M and ¯ M , this equation on d is a ﬁxed-point equation admitting a unique solution which we call d ¯ MM
Revised: Given two MDPs M and ¯ M , this equation on d is a ﬁxed-point equation admitting a unique solution which we call d ¯ MM

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: The proof follows closely that in Puterman (2014) that proves that the Bellman operator over value functions is a contraction mapping.
Revised: The proof follows closely that in Puterman (2014) that proves that the Bellman operator over value functions is a contraction mapping.

Original: Let d 1 and d 2 be two functions from S × A to R and let L be the functional operator that maps any function d : S ×
Revised: Let d 1 and d 2 be two functions from S × A to R and let L be the functional operator that maps any function d : S ×

Original: A → R to
Revised: A → R to

Original: [Equation]
Revised: [Equation]

Original: Then Ld 1 ( s, a ) − Ld 2 ( s, a )
Revised: Then Ld 1 ( s, a ) − Ld 2 ( s, a )

Original: [Equation]
Revised: [Equation]

Original: But
Revised: But

Original: [Equation]
Revised: [Equation]

Original: And so [Equation] Ld [Equation]
Revised: And so (cid:107)

Original: Ld
Revised: Ld

Original: [Equation]
Revised: [Equation]

Original:  ≤ γ [Equation] d 1 − d [Equation]
Revised: − Ld 2 (cid:107) ∞ ≤ γ (cid:107) d 1 − d 2 (cid:107) ∞ .

Original: ∞ . Since γ < 1 , L is a contraction mapping in the metric space ( S × A , (cid:107) · (cid:107) ∞ ) .
Revised: Since γ < 1 , L is a contraction mapping in the metric space ( S × A , (cid:107) · (cid:107) ∞ ) .

Original: This metric space being complete and non-empty, it follows from Banach ﬁxed point theorem that d = Ld admits a single solution.
Revised: This metric space being complete and non-empty, it follows from Banach ﬁxed point theorem that d = Ld admits a single solution.

Original: Lemma 1 guarantees the existence of d ¯ MM .
Revised: Lemma 1 guarantees the existence of d ¯ MM .

Original: Proposition 1 states that for any two MDPs M and ¯ M and for all ( s, a ) ∈ S × A , | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ min (cid:110)
Revised: Proposition 1 states that for any two MDPs M and ¯ M and for all ( s, a ) ∈ S × A , | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ min (cid:110)

Original: d ¯ MM ( s, a ) ,
Revised: d ¯ MM ( s, a ) ,

Original: [Equation]
Revised: [Equation]

Original: M ¯ M ( s, a ) (cid:111) .
Revised: ¯ M ( s, a ) (cid:111) .

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: The proof is by induction.
Revised: The proof is by induction.

Original: The Value Iteration sequence of iterates ( Q nM ) n ∈ N for task M is:
Revised: The Value Iteration sequence of iterates ( Q nM ) n ∈ N for task M is:

Original: [Equation]
Revised: [Equation]

Original: It is obvious that Q 0 M ( s, a ) − Q 0¯ M ( s, a ) ≤ d ¯ MM ( s, a ) .
Revised: It is obvious that Q 0 M ( s, a ) − Q 0¯ M ( s, a ) ≤ d ¯ MM ( s, a ) .

Original: Suppose that | Q nM ( s, a ) − Q n ¯ M ( s, a ) | ≤ d ¯ MM ( s, a ) .
Revised: Suppose that | Q nM ( s, a ) − Q n ¯ M ( s, a ) | ≤ d ¯ MM ( s, a ) .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] Since Q ∗ M and Q ∗ ¯ M are respectively the limits of the ( Q nM ) n ∈ N and (cid:0) Q n ¯ M (cid:1) n ∈ N sequences, the result that | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ d ¯ MM ( s, a ) follows from passage to the limit.
Revised: Since Q ∗ M and Q ∗ ¯ M are respectively the limits of the ( Q nM ) n ∈ N and (cid:0) Q n ¯ M (cid:1) n ∈ N sequences, the result that | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ d ¯ MM ( s, a ) follows from passage to the limit.

Original: By symmetry, on also has | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ d M ¯ M ( s, a ) and thus | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ min (cid:110)
Revised: By symmetry, on also has | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ d M ¯ M ( s, a ) and thus | Q ∗ M ( s, a ) − Q ∗ ¯ M ( s, a ) | ≤ min (cid:110)

Original: d ¯ MM ( s, a ) , d M ¯ M ( s, a ) (cid:111) .
Revised: d ¯ MM ( s, a ) , d M ¯ M ( s, a ) (cid:111) .

Original: D S IMILAR RESULTS TO P ROPOSITION
Revised: D S IMILAR RESULTS TO P ROPOSITION

Original: Similar results to Proposition 1 can be derived with a similar proof as in Section C.
Revised: Similar results to Proposition 1 can be derived with a similar proof as in Section C.

Original: The ﬁrst result is for the value function and is stated below.
Revised: The ﬁrst result is for the value function and is stated below.

Original: Proposition (Local bound on the distance between value functions) .
Revised: Proposition (Local bound on the distance between value functions) .

Original: For any two MDPs M and ¯ M , for all s ∈ S ,
Revised: For any two MDPs M and ¯ M , for all s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: where the local MDP pseudo-metric ∆ ¯ MM ( s, a ) has the same deﬁnition as in Proposition 1.
Revised: where the local MDP pseudo-metric ∆ ¯ MM ( s, a ) has the same deﬁnition as in Proposition 1.

Original: Another result can be derived for any policy π that one wishes to evaluate in both MDPs.
Revised: Another result can be derived for any policy π that one wishes to evaluate in both MDPs.

Original: For the sake of generality, we state the result for any stochastic policy mapping states to distributions over actions.
Revised: For the sake of generality, we state the result for any stochastic policy mapping states to distributions over actions.

Original: A deterministic policy is a stochastic policy choosing the selected action with probabilityand the others with probability 0 .
Revised: A deterministic policy is a stochastic policy choosing the selected action with probabilityand the others with probability 0 .

Original: Proposition (Local bound on the distance between value and Q-value functions for any policy.) .
Revised: Proposition (Local bound on the distance between value and Q-value functions for any policy.) .

Original: For any two MDPs M and ¯ M , for a stochastic policy π , for all s, a ∈ S × A ,
Revised: For any two MDPs M and ¯ M , for a stochastic policy π , for all s, a ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Original: where d π, ¯ M M ( s ) is deﬁned with the following ﬁxed-point equation:
Revised: where d π, ¯ M M ( s ) is deﬁned with the following ﬁxed-point equation:

Original: [Equation]
Revised: [Equation]

Original: and ∆ π, ¯ M M
Revised: and ∆ π, ¯ M M

Original: [Equation]
Revised: [Equation]

Original: E G LOBAL PSEUDO -L IPSCHITZ CONTINUITY RESULT
Revised: E G LOBAL PSEUDO -L IPSCHITZ CONTINUITY RESULT

Original: Recall that Proposition 1 states that for any two MDPs M and ¯ M , for all ( s, a ) ∈ S ×A , | Q ∗ M ( s, a ) −
Revised: Recall that Proposition 1 states that for any two MDPs M and ¯ M , for all ( s, a ) ∈ S ×A , | Q ∗ M ( s, a ) −

Original: Q [Equation] ¯ M [Equation]
Revised: Q ∗ ¯ M ( s, a ) | ≤ min (cid:110)

Original: s, [Equation] | ≤ min
Revised: d ¯ MM , d M ¯

Original: [Equation]
Revised: [Equation]

Original: MM ,  dM ¯ 
Revised: , with d ¯ MM

Original: [Equation]
Revised: [Equation]

Original: , with d
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: MM
Revised: a

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is by induction and reuses the notations introduced in the proof of Proposition 1.
Revised: The proof is by induction and reuses the notations introduced in the proof of Proposition 1.

Original: It is immediate that
Revised: It is immediate that

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: the result holds for n = 0 .
Revised: the result holds for n = 0 .

Original: Let us suppose that
Revised: Let us suppose that

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: F P ROOF OF P ROPOSITION
Revised: F P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: The result is clear for all s, a / ∈ K since the Lipschitz bounds are provably greater than Q ∗ M .
Revised: The result is clear for all s, a / ∈ K since the Lipschitz bounds are provably greater than Q ∗ M .

Original: For s, a ∈ K , the result is by induction.
Revised: For s, a ∈ K , the result is by induction.

Original: Let us consider the Dynamic Programming (Bellman, 1957) sequences converging to Q ∗ M and U at rank n whose deﬁnitions follow:
Revised: Let us consider the Dynamic Programming (Bellman, 1957) sequences converging to Q ∗ M and U at rank n whose deﬁnitions follow:

Original: [Equation]
Revised: [Equation]

Original: Obviously
Revised: Obviously

Original: [Equation]
Revised: [Equation]

Original: Q ∗ M
Revised: Q ∗ M

Original: [Equation]
Revised: [Equation]

Original: 0 ( s
Revised: 0 ( s

Original: [Equation]
Revised: [Equation]

Original: a ) ≤ U 0 ( s, a ) .
Revised: a ) ≤ U 0 ( s, a ) .

Original: Suppose the property true at rank n and consider rank n + 1 :[Equation] n [Equation]
Revised: Suppose the property true at rank n and consider rank n + 1 :

Original: s

Original: [Equation]

Original: s

Original: [Equation]

Original: =

Original: [Equation]

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Which concludes the proof by induction.
Revised: Which concludes the proof by induction.

Original: The result holds by passage to the limit since the considered
Revised: The result holds by passage to the limit since the considered

Original: Dynamic Programming sequences converge to the true functions.
Revised: Dynamic Programming sequences converge to the true functions.

Original: G P ROOF OF P ROPOSITION
Revised: G P ROOF OF P ROPOSITION

Original: Consider two tasks M = (cid:104) T, R (cid:105) and ¯ M = (cid:104) ¯ T, ¯ R (cid:105) , with K and ¯ K the respective sets of state-action pairs where their learned models ˆ M = (cid:104)
Revised: Consider two tasks M = (cid:104) T, R (cid:105) and ¯ M = (cid:104) ¯ T, ¯ R (cid:105) , with K and ¯ K the respective sets of state-action pairs where their learned models ˆ M = (cid:104)

Original: ˆ T, ˆ R (cid:105) and ˆ ¯ M = (cid:104)
Revised: ˆ T, ˆ R (cid:105) and ˆ ¯ M = (cid:104)

Original: ˆ ¯ T, ˆ ¯ R (cid:105) are known with accuracy (cid:15) in
Revised: ˆ ¯ T, ˆ ¯ R (cid:105) are known with accuracy (cid:15) in

Original: L 1 -norm with probability at least 1 − δ , i.e. we have that,
Revised: L 1 -norm with probability at least 1 − δ , i.e. we have that,

Original: [Equation]
Revised: [Equation]

Original: and the same goes for ¯ M and its learned model ˆ¯ M .
Revised: and the same goes for ¯ M and its learned model ˆ¯ M .

Original: We state the result for each one of the three cases 1) s, a ∈ K ∩ ¯ K , 2) s, a ∈ K ∩ ¯ K c and 3) s, a ∈ K c ∩ ¯ K c , the case s,
Revised: We state the result for each one of the three cases 1) s, a ∈ K ∩ ¯ K , 2) s, a ∈ K ∩ ¯ K c and 3) s, a ∈ K c ∩ ¯ K c , the case s,

Original: a ∈ K c ∩ ¯ K being the symmetric of case 2).
Revised: a ∈ K c ∩ ¯ K being the symmetric of case 2).

Original: 1) If s, a ∈ K ∩ ¯ K , then properties 11 and 12 hold for both (cid:104) R, T (cid:105) with (cid:104) ˆ R, ˆ T (cid:105) and (cid:104)
Revised: 1) If s, a ∈ K ∩ ¯ K , then properties 11 and 12 hold for both (cid:104) R, T (cid:105) with (cid:104) ˆ R, ˆ T (cid:105) and (cid:104)

Original: ¯ R, ¯ T (cid:105) with (cid:104) ˆ¯ R, ˆ¯ T (cid:105) .
Revised: ¯ R, ¯ T (cid:105) with (cid:104) ˆ¯ R, ˆ¯ T (cid:105) .

Original: We have by deﬁnition:
Revised: We have by deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term of the RHS of Equation 13 respects the following sequence of inequalities with probability at least 1 − δ :
Revised: The ﬁrst term of the RHS of Equation 13 respects the following sequence of inequalities with probability at least 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: The second term of the RHS of Equation 13 respects the following sequence of inequalities with probability at least 1 − δ :
Revised: The second term of the RHS of Equation 13 respects the following sequence of inequalities with probability at least 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: Summation of Equations 14 and 15 revealsˆ D
Revised: Summation of Equations 14 and 15 reveals

Original: [Equation]
Revised: [Equation]

Original: ass
Revised: ass

Original: [Equation]

Original: − ˆ¯ T ass

Original: [Equation]
Revised: [Equation]

Original: | on the RHS of the inequality.
Revised: | on the RHS of the inequality.

Original: Remarking this, we can upper-bound the model pseudo-distance of
Revised: Remarking this, we can upper-bound the model pseudo-distance of

Original: Equation 13 by the expected quantity with probability at least 1 − δ , proving the Proposition for case 1):
Revised: Equation 13 by the expected quantity with probability at least 1 − δ , proving the Proposition for case 1):

Original: [Equation]
Revised: [Equation]

Original: If s, a ∈ K ∩ ¯ K c , then properties 11 and 12 hold for (cid:104) R, T (cid:105) with (cid:104) ˆ R, ˆ T (cid:105) only.
Revised: If s, a ∈ K ∩ ¯ K c , then properties 11 and 12 hold for (cid:104) R, T (cid:105) with (cid:104) ˆ R, ˆ T (cid:105) only.

Original: Similarly to the proof of case 1), we upper bound sequentially the two terms of the RHS of Equation 13.
Revised: Similarly to the proof of case 1), we upper bound sequentially the two terms of the RHS of Equation 13.

Original: With probability at least 1 − δ , we have the following:
Revised: With probability at least 1 − δ , we have the following:

Original: [Equation]
Revised: [Equation]

Original: Similarly, with probability at least 1 − δ , we have:
Revised: Similarly, with probability at least 1 − δ , we have:

Original: [Equation]
Revised: [Equation]

Original: Combining inequalities 16 and 17, we get the following with probability at least 1 − δ , noticing
Revised: Combining inequalities 16 and 17, we get the following with probability at least 1 − δ , noticing

Original: D M ¯ M γV ∗ ¯ M ( s, a ) on the LHS:
Revised: D M ¯ M γV ∗ ¯ M ( s, a ) on the LHS:

Original: [Equation]
Revised: [Equation]

Original: which is the expected result.
Revised: which is the expected result.

Original: If s, a ∈ K c ∩ ¯ K c , then properties 11 and 12 do not hold.
Revised: If s, a ∈ K c ∩ ¯ K c , then properties 11 and 12 do not hold.

Original: In such a case, the result
Revised: In such a case, the result

Original: [Equation]
Revised: [Equation]

Original: is straightforward by remarking that V ∗ ¯ M ( s ) ≤ ¯ V ( s ) with probability at least 1 − δ .
Revised: is straightforward by remarking that V ∗ ¯ M ( s ) ≤ ¯ V ( s ) with probability at least 1 − δ .

Original: H A NALYTICAL CALCULATION OF ˆ D M ¯ M IN P ROPOSITION
Revised: H A NALYTICAL CALCULATION OF ˆ D M ¯ M IN P ROPOSITION

Original: Consider two tasks M = (cid:104) T, R (cid:105) and ¯ M = (cid:104) ¯ T, ¯ R (cid:105) , with K and ¯ K the respective sets of state-action pairs where their learned models ˆ M = (cid:104)
Revised: Consider two tasks M = (cid:104) T, R (cid:105) and ¯ M = (cid:104) ¯ T, ¯ R (cid:105) , with K and ¯ K the respective sets of state-action pairs where their learned models ˆ M = (cid:104)

Original: ˆ T, ˆ R (cid:105) and ˆ¯ M = (cid:104) ˆ¯ T, ˆ¯ R (cid:105) are known with accuracy (cid:15) in
Revised: ˆ T, ˆ R (cid:105) and ˆ¯ M = (cid:104) ˆ¯ T, ˆ¯ R (cid:105) are known with accuracy (cid:15) in

Original: L 1 -norm with probability at least 1 − δ .
Revised: L 1 -norm with probability at least 1 − δ .

Original: We note V max , a known upper-bound on the maximum achievable value.
Revised: We note V max , a known upper-bound on the maximum achievable value.

Original: In the worst case where one does not have any information on the value of V max , one can always set V max = 11 − γ .
Revised: In the worst case where one does not have any information on the value of V max , one can always set V max = 11 − γ .

Original: We recall the deﬁnition of the upper bound on the pseudo-metric between models:
Revised: We recall the deﬁnition of the upper bound on the pseudo-metric between models:

Original: [Equation]
Revised: [Equation]

Original: with B = (cid:15) (cid:0) 1 +
Revised: with B = (cid:15) (cid:0) [Equation] +

Original: [Equation]
Revised: [Equation]

Original: (cid:1) and D M ¯ M f deﬁned as in Equation 13.
Revised: (cid:1) and D M ¯ M f deﬁned as in Equation 13.

Original: We detail the computation of ˆ D M ¯ M ( s, a ) for each cases 1) s, a ∈ K ∩ ¯ K , 2) s, a ∈ K ∩ ¯ K c (the s, a ∈ K c ∩ ¯ K is symmetric to this one), and 3) s, a ∈ K c ∩ ¯ K c .
Revised: We detail the computation of ˆ D M ¯ M ( s, a ) for each cases 1) s, a ∈ K ∩ ¯ K , 2) s, a ∈ K ∩ ¯ K c (the s, a ∈ K c ∩ ¯ K is symmetric to this one), and 3) s, a ∈ K c ∩ ¯ K c .

Original: Recall that we consider a ﬁnite, countable, state-action space
Revised: Recall that we consider a ﬁnite, countable, state-action space

Original: S × A .
Revised: S × A .

Original: 1) If s, a ∈ K ∩ ¯ K , we have
Revised: 1) If s, a ∈ K ∩ ¯ K , we have

Original: [Equation]
Revised: [Equation]

Original: Since s, a is a known state-action pair, everything is known and computable in this last equation.
Revised: Since s, a is a known state-action pair, everything is known and computable in this last equation.

Original: Note that max s (cid:48) ¯ V ( s (cid:48) ) can be tracked along the updates of ¯ V and thus its computation does not induce any additional complexity
Revised: Note that max s (cid:48) ¯ V ( s (cid:48) ) can be tracked along the updates of ¯ V and thus its computation does not induce any additional complexity

Original: If s, a ∈ K ∩ ¯ K c , we have 
Revised: If s, a ∈ K ∩ ¯ K c , we have ˆ

Original: [Equation]
Revised: [Equation]

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: Maximizing the max t ∈ [0 , 1] |S| term is maximizing a convex combination of ¯ V (whose values are all positive) whose terms are not independent (since the t s (cid:48) terms should sum to one).
Revised: Maximizing the max t ∈ [0 , 1] |S| term is maximizing a convex combination of ¯ V (whose values are all positive) whose terms are not independent (since the t s (cid:48) terms should sum to one).

Original: This is easily cast as a linear programming problem.
Revised: This is easily cast as a linear programming problem.

Original: A straightforward (simplex-like) resolution procedure consists in progressively adding mass on the terms that will maximize the convex combination as follows:
Revised: A straightforward (simplex-like) resolution procedure consists in progressively adding mass on the terms that will maximize the convex combination as follows:

Original: • t s (cid:48)
Revised: • t s (cid:48)

Original: = 0 , ∀ s (cid:48)
Revised: = 0 , ∀ s (cid:48)

Original: ∈ S
Revised: ∈ S

Original: • l = Sort states by decreasing value of ¯ V • While (cid:80) s ∈S t ( s ) <– s (cid:48)
Revised: • l = Sort states by decreasing value of ¯ V • While (cid:80) s ∈S t ( s ) <– s (cid:48)

Original: = pop ﬁrst state in l – Assign t ( s (cid:48) ) ← arg max t ∈ [0 , 1] | ˆ T ass (cid:48) − t | to s (cid:48)
Revised: = pop ﬁrst state in l – Assign t ( s (cid:48) ) ← arg max t ∈ [0 , 1] | ˆ T ass (cid:48) − t | to s (cid:48)

Original: (note that t s (cid:48) ∈ { 0 , 1 } ) – If (cid:80) s ∈S t s > 1 , then t s (cid:48) ← 1 − (cid:80) s
Revised: (note that t s (cid:48) ∈ { 0 , 1 } ) – If (cid:80) s ∈S t s > 1 , then t s (cid:48) ← 1 − (cid:80) s

Original: ∈S\ s (cid:48) t ( s )
Revised: ∈S\ s (cid:48) t ( s )

Original: This allows calculating the maximum over transition models.
Revised: This allows calculating the maximum over transition models.

Original: There is however a simpler computation that almost always yields the same result (when it does not, it provides an upper bound) and does not require the burden of the previous procedure.
Revised: There is however a simpler computation that almost always yields the same result (when it does not, it provides an upper bound) and does not require the burden of the previous procedure.

Original: Consider the subset of states for which ¯ V ( s (cid:48) )
Revised: Consider the subset of states for which ¯ V ( s (cid:48) )

Original: =
Revised: =

Original: max s ¯ V ( s ) (often these are states in ¯ K c ).
Revised: max s ¯ V ( s ) (often these are states in ¯ K c ).

Original: Among those states, let us suppose there exists s + unreachable from s, a , according to ˆ T , that is ˆ T ass + = 0 .
Revised: Among those states, let us suppose there exists s + unreachable from s, a , according to ˆ T , that is ˆ T ass + = 0 .

Original: If ¯ M has not been fully explored, as is often the case in RMax, there may be many such states.
Revised: If ¯ M has not been fully explored, as is often the case in RMax, there may be many such states.

Original: Then the distribution t with all its mass on s + is a maximizer of the max t ∈ [0 , 1] |S| term.
Revised: Then the distribution t with all its mass on s + is a maximizer of the max t ∈ [0 , 1] |S| term.

Original: Conversely, if such a state does not exist (that is, if for all such states ˆ T ass + > 0 ), then max s ¯ V ( s ) is an upper bound on the max t ∈ [0 , 1] |S| term.
Revised: Conversely, if such a state does not exist (that is, if for all such states ˆ T ass + > 0 ), then max s ¯ V ( s ) is an upper bound on the max t ∈ [0 , 1] |S| term.

Original: Therefore:
Revised: Therefore:

Original: [Equation]
Revised: [Equation]

Original: cases
Revised: cases

Original: [Equation]
Revised: [Equation]

Original: If s, a ∈ K c ∩ ¯ K c , the resolution is trivial and we have
Revised: If s, a ∈ K c ∩ ¯ K c , the resolution is trivial and we have

Original: [Equation]
Revised: [Equation]

Original: I P ROOF OF P ROPOSITION
Revised: I P ROOF OF P ROPOSITION

Original: Lemma 2.
Revised: Lemma 2.

Original: Given two tasks M and ¯ M
Revised: Given two tasks M and ¯ M

Original: [Equation]
Revised: [Equation]

Original: K the set of state-action pairs for which (cid:104) R
Revised: K the set of state-action pairs for which (cid:104) R

Original: [Equation]
Revised: [Equation]

Original: T (cid:105) is known with accuracy (cid:15) in L 1 -norm with probability at least 1 − δ
Revised: T (cid:105) is known with accuracy (cid:15) in L 1 -norm with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: If γ (1 + (cid:15) ) <
Revised: If γ (1 + (cid:15) ) <

Original: [Equation]
Revised: [Equation]

Original: this equation on ˆ d is a ﬁxed-point equation admitting a unique solution which we call ˆ d ¯ MM ˆ d ( s, a ) =
Revised: this equation on ˆ d is a ﬁxed-point equation admitting a unique solution which we call ˆ d ¯ MM ˆ d ( s, a ) =

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: The proof is similar to the proof of Lemma 1.
Revised: The proof is similar to the proof of Lemma 1.

Original: Let d 1 and d 2 be two functions from S × A to R and let L be the functional operator that maps any function d : S ×
Revised: Let d 1 and d 2 be two functions from S × A to R and let L be the functional operator that maps any function d : S ×

Original: A → R to
Revised: A → R to

Original: Ld : s, a (cid:55)→
Revised: Ld : s, a (cid:55)→

Original: [Equation]
Revised: [Equation]

Original: If s, a ∈ K , we have
Revised: If s, a ∈ K , we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: If s, a / ∈ K , we have

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] If s, a / ∈ K , we have [Equation]
Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In both cases, (cid:107) Ld 1 − Ld 2 (cid:107) ∞ ≤ γ (1 + (cid:15) ) (cid:107) d 1 − d 2 (cid:107) ∞ .
Revised: In both cases, (cid:107) Ld 1 − Ld 2 (cid:107) ∞ ≤ γ (1 + (cid:15) ) (cid:107) d 1 − d 2 (cid:107) ∞ .

Original: If γ (1 + (cid:15) ) < 1 , L is a contraction mapping in the metric space ( S × A , (cid:107) · (cid:107) ∞ ) .
Revised: If γ (1 + (cid:15) ) < 1 , L is a contraction mapping in the metric space ( S × A , (cid:107) · (cid:107) ∞ ) .

Original: This metric space being complete and non-empty, it follows from Banach ﬁxed point theorem that d = Ld admits a single solution.
Revised: This metric space being complete and non-empty, it follows from Banach ﬁxed point theorem that d = Ld admits a single solution.

Original: Proof of Proposition 5.
Revised: Proof of Proposition 5.

Original: The proof is done by induction, by calculating the values of d ¯ MM and ˆ d ¯ MM following the value iteration algorithm.
Revised: The proof is done by induction, by calculating the values of d ¯ MM and ˆ d ¯ MM following the value iteration algorithm.

Original: Those values can respectively be computed via the sequences of iterates ( d n ) n ∈ N and ( ˆ d n ) n ∈ N deﬁned as follows:
Revised: Those values can respectively be computed via the sequences of iterates ( d n ) n ∈ N and ( ˆ d n ) n ∈ N deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: and, ˆ
Revised: and, ˆ

Original: [Equation]

Original: ˆ

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: The proof at rank n = 0 is trivial.
Revised: The proof at rank n = 0 is trivial.

Original: Let us assume the proposition d n ≤ ˆ d n , ∀ s, a ∈ S ×
Revised: Let us assume the proposition d n ≤ ˆ d n , ∀ s, a ∈ S ×

Original: A true at rank n and consider rank n + 1 .
Revised: A true at rank n and consider rank n + 1 .

Original: There are two cases, depending on the fact that s, a is in K or not.
Revised: There are two cases, depending on the fact that s, a is in K or not.

Original: If s, a ∈ K , we have
Revised: If s, a ∈ K , we have

Original: [Equation]
Revised: [Equation]

Original: Using Proposition 4, we have that ˆ D M ¯ M ( s, a ) is an upper bound on D M ¯ M γV ∗ ¯ M ( s, a ) with probability at least 1 − δ .
Revised: Using Proposition 4, we have that ˆ D M ¯ M ( s, a ) is an upper bound on D M ¯ M γV ∗ ¯ M ( s, a ) with probability at least 1 − δ .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: This plus the fact that d n ≤ ˆ d n by induction hypothesis, we have that
Revised: This plus the fact that d n ≤ ˆ d n by induction hypothesis, we have that

Original: [Equation]
Revised: [Equation]

Original: Since Pr (cid:16) (cid:107) T − ˆ T (cid:107) 1 ≤ (cid:15) (cid:17) ≥ 1 − δ , we have with probability at least 1 − δ ,
Revised: Since Pr (cid:16) (cid:107) T − ˆ T (cid:107) 1 ≤ (cid:15) (cid:17) ≥ 1 − δ , we have with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof in this case.
Revised: which concludes the proof in this case.

Original: Conversely, if s, a / ∈ K , we have
Revised: Conversely, if s, a / ∈ K , we have

Original: [Equation]
Revised: [Equation]

Original: Using the same reasoning than in case s, a ∈ K , we have with probability higher than 1 − δ d n +[Equation] s
Revised: Using the same reasoning than in case s, a ∈ K , we have with probability higher than 1 − δ d n +

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof in the second case.
Revised: which concludes the proof in the second case.

Original: J P ROOF OF P ROPOSITION
Revised: J P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: We follow the proof of the computational complexity of RMax proposed by Strehl et al.
Revised: We follow the proof of the computational complexity of RMax proposed by Strehl et al.

Original: The cost of Lipschitz RMax is constant on most time steps since the action is greedily chosen w.r.t.
Revised: The cost of Lipschitz RMax is constant on most time steps since the action is greedily chosen w.r.t.

Original: the upper-bound on the optimal Q-value function which is a lookup table.
Revised: the upper-bound on the optimal Q-value function which is a lookup table.

Original: When updating a new state-action pair (labelling it as a known pair), the algorithm performs 2 N DP computations to update the Lipschitz bounds plus one DP computation to update the total-bound.
Revised: When updating a new state-action pair (labelling it as a known pair), the algorithm performs 2 N DP computations to update the Lipschitz bounds plus one DP computation to update the total-bound.

Original: The cost of one DP computation is given by (Strehl et al., 2009):
Revised: The cost of one DP computation is given by (Strehl et al., 2009):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The result comes out by remarking that at most SA state-action pairs are updated, each resulting in ( N + 1) DP computations.
Revised: The result comes out by remarking that at most SA state-action pairs are updated, each resulting in ( N + 1) DP computations.

Original: K P ROOF OF P ROPOSITION
Revised: K P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Consider a ﬁxed state-action pair s, a ∈ S × A .
Revised: Consider a ﬁxed state-action pair s, a ∈ S × A .

Original: For two sampled tasks M, ¯ M ∈ ˆ M 2 , we assume our algorithm to provide an upper-bound on D M ¯ M γV ∗ ¯ M ( s, a ) with probability at least 1 − δ .
Revised: For two sampled tasks M, ¯ M ∈ ˆ M 2 , we assume our algorithm to provide an upper-bound on D M ¯ M γV ∗ ¯ M ( s, a ) with probability at least 1 − δ .

Original: This assumption is actually guaranteed by Proposition 4 while running Algorithm 1.
Revised: This assumption is actually guaranteed by Proposition 4 while running Algorithm 1.

Original: With probability at least 1 − δ ,
Revised: With probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence, with probability at least 1 − δ ,
Revised: Hence, with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: In turn, the event of underestimating D max ( s, a ) occurs only if the two tasks, that we note M ∗ 1 , M ∗ 2 ∈
Revised: In turn, the event of underestimating D max ( s, a ) occurs only if the two tasks, that we note M ∗ 1 , M ∗ 2 ∈

Original: M 2 , maximizing M, ¯ M (cid:55)→ D M ¯ M γV ∗ ¯ M ( s, a ) , are not sampled, i.e. do not belong to ¯ M .
Revised: M 2 , maximizing M, ¯ M (cid:55)→ D M ¯ M γV ∗ ¯ M ( s, a ) , are not sampled, i.e. do not belong to ¯ M .

Original: M ∗ 1 and M ∗ 2 are not necessarily unique, but they could be.
Revised: M ∗ 1 and M ∗ 2 are not necessarily unique, but they could be.

Original: Since we aim at deriving a lower bound on the probability of sampling M ∗ 1 and M ∗ 2 , we consider the worst case where they are unique.
Revised: Since we aim at deriving a lower bound on the probability of sampling M ∗ 1 and M ∗ 2 , we consider the worst case where they are unique.

Original: The probability ˜ P of sampling one particular task, whose sampling probability is p , after i samples, is given by the cumulative distribution function of the geometric distribution and is p (1 − p )
Revised: The probability ˜ P of sampling one particular task, whose sampling probability is p , after i samples, is given by the cumulative distribution function of the geometric distribution and is p (1 − p )

Original: i − 1 .
Revised: i − 1 .

Original: Consequently, if the sampling probability p of this task is lower bounded by p min , the quantity p min (1 − p min )
Revised: Consequently, if the sampling probability p of this task is lower bounded by p min , the quantity p min (1 − p min )

Original: i −lower bounds ˜ P .
Revised: i −lower bounds ˜ P .

Original: Let us write X the random variable of the number of samples required for sampling either M ∗ 1 or M ∗ 2 for the ﬁrst time.
Revised: Let us write X the random variable of the number of samples required for sampling either M ∗ 1 or M ∗ 2 for the ﬁrst time.

Original: By considering that the sampling probability of either sampling
Revised: By considering that the sampling probability of either sampling

Original: M ∗ 1 or M ∗ 2 is lower bounded by 2 p min , we follow the same reasoning as for ˜ P and obtain that :
Revised: M ∗ 1 or M ∗ 2 is lower bounded by 2 p min , we follow the same reasoning as for ˜ P and obtain that :

Original: [Equation]
Revised: [Equation]

Original: Let us write Y the random variable of the number of samples required for sampling the remaining task for the ﬁrst time.
Revised: Let us write Y the random variable of the number of samples required for sampling the remaining task for the ﬁrst time.

Original: We have the following result using the geometric distribution for the conditional
Revised: We have the following result using the geometric distribution for the conditional

Original: Pr ( Y = k
Revised: Pr ( Y = k

Original: | X = i ) :
Revised: | X = i ) :

Original: [Equation]
Revised: [Equation]

Original: Pr ( Y = k ) is the probability of ﬁrst success at step k .
Revised: Pr ( Y = k ) is the probability of ﬁrst success at step k .

Original: For ˆ D max ( s, a ) to estimate D max ( s, a ) in m steps, we require that this success occurs any time during the ﬁrst m steps, so we have:
Revised: For ˆ D max ( s, a ) to estimate D max ( s, a ) in m steps, we require that this success occurs any time during the ﬁrst m steps, so we have:

Original: [Equation]
Revised: [Equation]

Original: Using Equation 19, we can deduce our result when remarking that necessarily p min ≤ 1 / 2 :[Equation] =
Revised: Using Equation 19, we can deduce our result when remarking that necessarily p min ≤ 1 / 2 :

Original: [Equation]
Revised: [Equation]

Original: min
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original:  min ) [Equation] − [Equation]
Revised: k = k − 1 (cid:88) i = (1 − p min ) k − i − 1 (1 − 2 p min )

Original:  p [Equation] min m 
Revised: i −≥ 2 p 2 min m − 2 (cid:88) k

Original: [Equation] (cid:88) 
Revised: = k (cid:88) i =

Original: [Equation]
Revised: (1 − p min ) k − i (1 − 2 p min )

Original: k (cid:88) i [Equation] (1 − 
Revised: i ≥ 2 p 2min m − 2 (cid:88) k

Original: [Equation] min ) [Equation] (1 − 
Revised: = (1 − p min ) k k (cid:88)

Original: [Equation] p min ) i ≥ [Equation]
Revised: i = (cid:18) 1 − 2 p min 1 − p min (cid:19)

Original:  m [Equation] (cid:88) [Equation]
Revised: i ≥ 2 p 2min m − 2 (cid:88) k

Original: [Equation] (cid:88) [Equation] (cid:18) 
Revised: = (1 − p min ) k 1 p (cid:18) 1 − p min − (1 − 2 p min ) k

Original: [Equation] − 
Revised: + (1 − p min ) k

Original: [Equation] p min 1 − p min (cid:19) i [Equation]
Revised: (cid:19) ≥ 2 p min m − 2 (cid:88) k

Original: [Equation] p (cid:18) 1 − p min  − (1 − 
Revised: = (cid:0) (1 − p min ) k +1 − (1 − 2 p min )

Original: [Equation] p min ) k + (1 − p min ) k (cid:19) ≥ 2 p min m − 
Revised: k +1 (cid:1) ≥ 2 p min (1 − p min ) 1 − (1 − p min ) m −1 − (1 − p min ) − 2 p min (1 − 2 p min ) 1 − (1 − 2 p min

Original: [Equation]
Revised: [Equation]

Original: (cid:88) k
Revised: [Figure] L D ISCUSSION ON AN UPPER BOUND ON DISTANCES BETWEEN MDP MODELS

Original: = (cid:0) (1 − p min ) k +1 − (1 − 2 p min ) k +1 (cid:1) ≥ 2 p min (1 − p min )1 − (1 − p min ) m − 1 1 − (1 − p min ) − 2 p min (1 − 2 p min )1 − (1 − 2 p min
Revised: Section 4.3 introduced the idea of exploiting prior knowledge on the maximum distance between two MDP models.

Original:  [Equation]
Revised: This idea begs for a more detailed discussion. Consider two MDPs M and ¯ M . By deﬁnition of the local model pseudo metric D M ¯ M γV ∗ ¯ M in Equation 1, the maximum possible distance is given by [Equation]

Revised: But this assumes any transition or reward model can deﬁne ¯ M .

Revised: In other words, the maximization is made on the whole set of possible MDPs.

Revised: To illustrate why this is too naive, consider a game within the Arcade Learning Environment (Bellemare et al., 2013).

Revised: We, as humans, have a strong bias concerning similarity between environments.

Revised: If the game changes, we still assume groups of pixels will move together on the screen as the result of game actions.

Revised: For instance, we generally discard possible new games ¯ M that “teleport” objects across the screen without physical considerations.

Revised: We also discard new games that allow transitions from a given screen to another screen full of static.

Revised: These examples illustrate why the knowledge of D max is very natural (and also why its precise value may be irrelevant).

Revised: The same observation can be made for the “tight” experiment of Section 5; the set of possible MDPs is restricted by some implicit assumptions that constrain the maximum distance between tasks.

Revised: For instance, in these experiments, all transitions move to a neighboring state and never “teleport” the agent to the other side of the gridworld.

Revised: Without the knowledge of D max , LRMax assumes such environments are possible and therefore transfer values very cautiously.

Revised: Overall, the experiments of Section 5 conﬁrm this important insight: safe transfer occurs slowly if no a priori is given on the maximum distance between MDPs.

Revised: On the contrary, the knowledge of D max allows a faster and more efﬁcient transfer between environments.

Original: L T HE “ TIGHT ” ENVIRONMENT USED IN EXPERIMENTS OF S ECTION
Revised: M T HE “ TIGHT ” ENVIRONMENT USED IN EXPERIMENTS OF S ECTION

Original: The tight environment is a 11 × 11 grid-world illustrated in Figure 4.
Revised: The tight environment is a 11 × 11 grid-world illustrated in Figure 20 cells

Original: The initial state of the agent is the central cell displayed with an “S”.
Revised: S G [Figure] A v e r a g

Revised: e R e l a t

Original: The actions are moving 1 cell in one of the four cardinal directions.
Revised: i v e D i s c o un t e d R e t u r n

Revised: [Figure]

Revised: N A DDITIONAL LIFELONG RL EXPERIMENTS

Original: The reward is 0 everywhere, except for executing an action in one of the three teal cells in the upper-right corner.
Revised: We ran additional experiments on the corridor grid-world environment represented in Figure 5. The initial state of the agent is the central cell labeled with the letter “S”. The actions are { left, right } and the goal is to reach the cell labeled with the letter “G” on the extreme right. A reward R > 0 is received when reaching the goal and 0 otherwise.

Original: Each time a task is sampled, a slipping probability of executing another action as the one selected is drawn in [0  , 1]  and the reward received in  each one of the teal cells is picked in [0 . 8 , 1 . 0] .
Revised: At each new task, a new value of R is sampled in [0 . 8 , 1] . The transition function is ﬁxed and deterministic. The key insight in this experiment is not to lose time exploring the left part of the corridor. We ran 20 episodes of 11 time steps for each one of the 20 sampled tasks.

Revised: Results are displayed in

Revised: Figure 6a and 6b, respectively for the average relative discounted return over episodes and over tasks.

Original: S
Revised: Similarly as in Section 5, we observe in Figure 6a that LRMax beneﬁts from the transfer method as early as the second task. The MaxQInit algorithm beneﬁts from the transfer from task number 12. Prior knowledge D max decreases the sample complexity of LRMax as reported earlier and the combination of LRMax with MaxQInit outperforms all other methods by providing a tighter upper-bound on the optimal Q-value function. This decrease of sample complexity is also observed in the episode-wise display of Figure 6b where the convergence happens more quickly on average for

Revised: LRMax and even more for MaxQInit.

Revised: This ﬁgure allows to see the three learning stages of LRMax reported in Section 5.

Revised: We also ran lifelong RL experiments in the maze grid-world of Figure 7.

Revised: The tasks consists in reaching the goal cell labeled with a “G” while the initial state of the agent is the central cell, labeled with an “S”.

Revised: Two walls conﬁgurations are possible, yielding two different tasks with probability 12 of being sampled in the lifelong RL setting.

Revised: The ﬁrst task corresponds to the case where orange walls are actually walls and green cells are normal white cells where the agent can go.

Revised: The second task is the converse, where green walls are walls and orange cells are normal white cells.

Revised: We runepisodes of length 15 time steps and sample a total of 30 different tasks.

Revised: Results can be found in

Revised: Figure 8.

Revised: In this experiment, we observe the increase of performance of LRMax as the value of D max decreases.

Revised: The three stages behavior of LRMax reported in Section 5 does not appear in this case.

Revised: We tested the performance of using the online estimation of the local model distances of Proposition 8 in the algorithm referred by LRMax in Figure 8.

Revised: Once enough tasks have been sampled, the estimate on the model local distance is used with high conﬁdence on its value and reﬁnes the upper-bound computed analytically in Equation 7.

Revised: Importantly, this instance of LRMax achieved the best result in this particular environment, demonstrating the usefulness of this result.

Revised: This method being similar

Revised: [Figure]

Revised: [Figure]

Revised: to the MaxQInit estimation of maximum Q-values, we unsurprisingly observe that both algorithms feature a similar performance in the maze environment.

Original: M P RIOR D max USE EXPERIMENT
Revised: O P RIOR D max USE EXPERIMENT

Original: Each time an s, a pair is updated, we compute the local distance upper bound ˆ D (Equation 7) for all ( s, a ) ∈ S × A .
Revised: Each time an s, a pair is updated, we compute the local distance upper bound ˆ D (Equation 7) for all ( s, a ) ∈ S × A .

Original: In this computation, one can leverage knowledge of D max to select min { ˆ D, D max } .
Revised: In this computation, one can leverage knowledge of D max to select min { ˆ D, D max } .

Original: We show that LRMax relies less and less on D max as knowledge on the current task increases.
Revised: We show that LRMax relies less and less on D max as knowledge on the current task increases.

Original: For this experiment, we used the two grid-worlds environments displayed in Figures 5 and 6.
Revised: For this experiment, we used the two grid-worlds environments displayed in Figures 9 and 10.

Original: The rewards collected with any actions performed in the teal cells of both tasks are deﬁned as:
Revised: The rewards collected with any actions performed in the teal cells of both tasks are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where ( s x , s y ) are the coordinates of the current state, ( g x , g y ) the coordinate of the goal cell labelled with a G and σ is a span parameter equal to 1 in the ﬁrst environment and 1 . 5 in the second environment.
Revised: where ( s x , s y ) are the coordinates of the current state, ( g x , g y ) the coordinate of the goal cell labelled with a G and σ is a span parameter equal to 1 in the ﬁrst environment and 1 . 5 in the second environment.

Original: The agent starts at the cell labelled with the S letter.
Revised: The agent starts at the cell labelled with the S letter.

Original: Black cells represent unreachablecells (walls).
Revised: Black cells represent unreachable

Original: We run LRMax twice on the two different maze grid-worlds and record for each model update the proportion of times D max is smaller than ˆ D in Figure 7 via the % use of D max .

Original: [Equation] [Figure]
Revised: [Figure]

Original:  probability is 10%.
Revised: Slipping probability is 10%.

Original: S G Slipping probability is 5%. 48 Computation Number      %
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: LRMax-prior19. LRMax-prior17. LRMax-prior15. LRMax-prior10.
Revised: cells (walls).

Original: LRMax-prior0.
Revised: We run LRMax twice on the two different maze grid-worlds and record for each model update the proportion of times D max is smaller than ˆ D in Figure 11 via the % use of D max .

Original: With maximum value D max = 19 , ˆ D is systematically lesser than D max , resulting in 0% use.
Revised: With maximum value D max = 19 , ˆ D is systematically lesser than D max , resulting in 0% use.

Original: Conversely, with minimum value D max = 0 , the use expectedly increases to 100%.
Revised: Conversely, with minimum value D max = 0 , the use expectedly increases to 100%.

Original: The in-between value of D max = 10 displays a linear decay of the use.
Revised: The in-between value of D max = 10 displays a linear decay of the use.

Original: This suggests that, at each update, ˆ D ≤ D max is only true for one more unique s, a pair, resulting in a constant decay of the use.
Revised: This suggests that, at each update, ˆ D ≤ D max is only true for one more unique s, a pair, resulting in a constant decay of the use.

Original: With fewer prior ( D max = 15 or 17 ), updating one single s, a pair allows ˆ D to drop under D max for more than one pair, resulting in less use of the prior knowledge.
Revised: With fewer prior ( D max = 15 or 17 ), updating one single s, a pair allows ˆ D to drop under D max for more than one pair, resulting in less use of the prior knowledge.

Original: The conclusion of this experiment if that D max is only useful at the beginning of the exploration, while LRMax relies more on its own bound ˆ D when partial knowledge of the task has been acquired.
Revised: The conclusion of this experiment if that D max is only useful at the beginning of the exploration, while LRMax relies more on its own bound ˆ D when partial knowledge of the task has been acquired.

Original: N D ISCUSSION ON RM AX PRECISION PARAMETERS (cid:15) , δ , n known
Revised: P D ISCUSSION ON RM AX PRECISION PARAMETERS (cid:15) , δ , n known

Original: We used n known = 10 , δ = 0 .
Revised: We used n known = 10 , δ = 0 .

Original: 05 and (cid:15) = 0 .
Revised: 05 and (cid:15) = 0 .

Original: Theoretically, n known should be a lot larger ( ≈ 10 5 ) in order to reach an accuracy (cid:15) = 0 .
Revised: Theoretically, n known should be a lot larger ( ≈ 10 5 ) in order to reach an accuracy (cid:15) = 0 .

Original: according to Strehl et al.
Revised: according to Strehl et al.

Original: However, it is common practice to assume such small values of n known are sufﬁcient to reach an acceptable model accuracy (cid:15) .
Revised: However, it is common practice to assume such small values of n known are sufﬁcient to reach an acceptable model accuracy (cid:15) .

Original: Interestingly, empirical validation did not conﬁrm this assumption for any RMax-based algorithm.
Revised: Interestingly, empirical validation did not conﬁrm this assumption for any RMax-based algorithm.

Original: We keep these values nonetheless for the sake of comparability between algorithms and consistency with the literature.
Revised: We keep these values nonetheless for the sake of comparability between algorithms and consistency with the literature.

Original: Despite such absence of accuracy guarantees, RMax-based algorithms still perform surprisingly well and are robust to model estimation uncertainties.
Revised: Despite such absence of accuracy guarantees, RMax-based algorithms still perform surprisingly well and are robust to model estimation uncertainties.

Original: O I NFORMATIONS ABOUT THE M ACHINE L EARNING REPRODUCIBILITY CHECKLIST
Revised: Q I NFORMATIONS ABOUT THE M ACHINE L EARNING REPRODUCIBILITY CHECKLIST

Original: For the experiments run in Section 5, the computing infrastructure used was a laptop using a single 64-bit CPU (model: Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz).
Revised: For the experiments run in Section 5, the computing infrastructure used was a laptop using a single 64-bit CPU (model: Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz).

Original: The collected samples sizes and number of evaluation runs for each experiment is summarized in Table 1. 
Revised: The collected samples sizes and number of evaluation runs for each experiment is summarized in Table 1. Task Number

Revised: [Table]

Revised: Number

Original: [Table]
Revised: [Table]

Revised: Number

Revised: [Table]

Revised: episodes

Revised: Maximumlengthofepisodes

Revised: Total number of collected transition samples (

Revised: [Table]

Revised: , a, r, s (cid:48) )

Original: “Tight” task of Figures 1a 1b and 1c
Revised: “Tight” task of Figures 1a 1b and 1c

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: “Tight” task of Figure 1d
Revised: “Tight” task of Figure 1d

Original: [Table]
Revised: [Table]

Revised: Corridor task

Revised: Section N

Revised: Maze task

Revised: Section N

Original:  The displayed conﬁdence intervals for any curve presented in the paper is the 95% conﬁdence interval (Neyman, 1937) on the displayed mean.
Revised: [Table] Section O [Table] The displayed conﬁdence intervals for any curve presented in the paper is the 95% conﬁdence interval (Neyman, 1937) on the displayed mean.

Original: No data were excluded neither pre-computed.
Revised: No data were excluded neither pre-computed.

Original: Hyper-parameters were determined to our appreciation, they may be sub-optimal but we found the results convincing enough to display interesting behaviours.
Revised: Hyper-parameters were determined to our appreciation, they may be sub-optimal but we found the results convincing enough to display interesting behaviors