Original: Operator Splitting Value Iteration
Revised: Operator Splitting Value Iteration

Original: Abstract
Revised: Abstract

Original: We introduce new planning and reinforcement learning algorithms for discounted
Revised: We introduce new planning and reinforcement learning algorithms for discounted

Original: MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function.
Revised: MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function.

Original: Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both
Revised: Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both

Original: Policy Evaluation and Control problems.
Revised: Policy Evaluation and Control problems.

Original: OS-VI achieves a much faster convergence rate when the model is accurate enough.
Revised: OS-VI achieves a much faster convergence rate when the model is accurate enough.

Original: We also introduce a sample-based version of the algorithm called OS-Dyna.
Revised: We also introduce a sample-based version of the algorithm called OS-Dyna.

Original: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of model approximation error.
Revised: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of model approximation error.

Original:  Introduction
Revised:  Introduction

Original: Consider a planning problem for a discounted MDP with dynamics P .
Revised: Consider a planning problem for a discounted MDP with dynamics P .

Original: Suppose that we have access to an approximate model ˆ P ≈ P as well.
Revised: Suppose that we have access to an approximate model ˆ P ≈ P as well.

Original: For example, P might be a high-ﬁdelity, but slow, simulator, and ˆ P is a lower-ﬁdelity, but fast, simulator.
Revised: For example, P might be a high-ﬁdelity, but slow, simulator, and ˆ P is a lower-ﬁdelity, but fast, simulator.

Original: Or in the context of model-based reinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which we can only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaply acquired.
Revised: Or in the context of model-based reinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which we can only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaply acquired.

Original: Can we use this approximate model ˆ P to accelerate the computation of the value function of a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?
Revised: Can we use this approximate model ˆ P to accelerate the computation of the value function of a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?

Original: The Value Iteration (VI) algorithm and its approximate variant are fundamental algorithms in Dynamic Programming that can ﬁnd the (approximate) value of a policy or the optimal value function.
Revised: The Value Iteration (VI) algorithm and its approximate variant are fundamental algorithms in Dynamic Programming that can ﬁnd the (approximate) value of a policy or the optimal value function.

Original: They are also the backbone of many reinforcement learning (RL) algorithms such as Temporal
Revised: They are also the backbone of many reinforcement learning (RL) algorithms such as Temporal

Original: Difference Learning [Sutton, 1988], Fitted Value Iteration [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári, 2008], and Deep Q Network [Mnih et al., 2015].
Revised: Difference Learning [Sutton, 1988], Fitted Value Iteration [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári, 2008], and Deep Q Network [Mnih et al., 2015].

Original: Value Iteration, however, can be slow when the discount factor is close to 1 , as its convergence rate is O ( γ k ) .
Revised: Value Iteration, however, can be slow when the discount factor is close to 1 , as its convergence rate is O ( γ k ) .

Original: Moreover, even though we could use VI using ˆ P instead of P to avoid expensive queries to P , the obtained value function would converge to a solution different from the value function of the original MDP.
Revised: Moreover, even though we could use VI using ˆ P instead of P to avoid expensive queries to P , the obtained value function would converge to a solution different from the value function of the original MDP.

Original: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that beneﬁts from an approximate model ˆ P to potentially accelerate the convergence of the value function sequence to the value function with respect to (w.r.t.)
Revised: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that beneﬁts from an approximate model ˆ P to potentially accelerate the convergence of the value function sequence to the value function with respect to (w.r.t.)

Original: the true model P (Section 3).
Revised: the true model P (Section 3).

Original: This algorithm is for both
Revised: This algorithm is for both

Original: PE (Section 3.1) and Control (Section 3.2) problems.
Revised: PE (Section 3.1) and Control (Section 3.2) problems.

Original: The acceleration is not uniform though, and depends on how close ˆ P is to P (Section 4).
Revised: The acceleration is not uniform though, and depends on how close ˆ P is to P (Section 4).

Original: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra, which is used to iteratively solve large linear systems of equations [Varga, 2000, Saad, 2003, Golub and Van Loan, 2013].
Revised: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra, which is used to iteratively solve large linear systems of equations [Varga, 2000, Saad, 2003, Golub and Van Loan, 2013].

Original: With a proper choice of splitting, one may change the convergence rate of 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Revised: With a proper choice of splitting, one may change the convergence rate of 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: linear systems solvers.
Revised: linear systems solvers.

Original: We show that the conventional VI for PE can be seen as a particular choice of splitting.
Revised: We show that the conventional VI for PE can be seen as a particular choice of splitting.

Original: This observation suggests that one may choose other forms of splitting as well in order to change the convergence rate.
Revised: This observation suggests that one may choose other forms of splitting as well in order to change the convergence rate.

Original: It turns out that we can choose a splitting that beneﬁts from having access to ˆ P (Section 2).
Revised: It turns out that we can choose a splitting that beneﬁts from having access to ˆ P (Section 2).

Original: The new splitting leads to OS-VI for PE.
Revised: The new splitting leads to OS-VI for PE.

Original: For the Control problem, the connection between solving linear system of equations and VI is not as straightforward anymore, as the former is linear, while the latter is not, but we can still get inspired from the splitting approach to design OS-VI for Control.
Revised: For the Control problem, the connection between solving linear system of equations and VI is not as straightforward anymore, as the former is linear, while the latter is not, but we can still get inspired from the splitting approach to design OS-VI for Control.

Original: The key step of such an algorithm is a new policy improvement step.
Revised: The key step of such an algorithm is a new policy improvement step.

Original: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate model ˆ
Revised: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate model ˆ

Original: P is learned using data.
Revised: P is learned using data.

Original: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].
Revised: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].

Original: OS-Dyna is a hybrid of model-free and model-based algorithms.
Revised: OS-Dyna is a hybrid of model-free and model-based algorithms.

Original: It uses the learned model in its inner planning loop, alike Dyna, but uses samples from the true model P in order to correct the effect of errors in the model.
Revised: It uses the learned model in its inner planning loop, alike Dyna, but uses samples from the true model P in order to correct the effect of errors in the model.

Original: Existing MBRL algorithms would converge to an incorrect solution if the approximate model ˆ P does not converge to the true model P .
Revised: Existing MBRL algorithms would converge to an incorrect solution if the approximate model ˆ P does not converge to the true model P .

Original: This would be the case whenever model approximation error exists.
Revised: This would be the case whenever model approximation error exists.

Original: On the other hand, OS-Dyna can still converge to the correct value function even when ˆ P does not converge to P .
Revised: On the other hand, OS-Dyna can still converge to the correct value function even when ˆ P does not converge to P .

Original: As far as we know, this is the ﬁrst model-based RL algorithm with such property.
Revised: As far as we know, this is the ﬁrst model-based RL algorithm with such property.

Original:  From value iteration to splitting-based linear system of equations solvers and back
Revised:  From value iteration to splitting-based linear system of equations solvers and back

Original: We brieﬂy describe the VI algorithm and the splitting methods for solving linear system of equations, and explain their connections.
Revised: We brieﬂy describe the VI algorithm and the splitting methods for solving linear system of equations, and explain their connections.

Original: We consider a discounted Markov Decision Process (MDP) ( X , A , R , P , γ )
Revised: We consider a discounted Markov Decision Process (MDP) ( X , A , R , P , γ )

Original: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].
Revised: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].

Original: We defer formal deﬁnitions to the supplementary material.
Revised: We defer formal deﬁnitions to the supplementary material.

Original: We only mention that for a policy π , we denote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π =
Revised: We only mention that for a policy π , we denote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π =

Original: V π ( R , P ) its state-value function.
Revised: V π ( R , P ) its state-value function.

Original: We also represent the optimal state-value function by V ∗ =
Revised: We also represent the optimal state-value function by V ∗ =

Original: V ∗ ( R , P ) and the optimal policy by π ∗ =
Revised: V ∗ ( R , P ) and the optimal policy by π ∗ =

Original: π ∗ ( R , P ) .
Revised: π ∗ ( R , P ) .

Original: The Bellman operator
Revised: The Bellman operator

Original: T π : B ( X ) → B ( X ) for policy π and the Bellman optimality operator T ∗ : B ( X ) → B ( X ) are( T
Revised: T π : B ( X ) → B ( X ) for policy π and the Bellman optimality operator T ∗ : B ( X ) → B ( X ) are( T

Original: [Equation]
Revised: [Equation]

Original: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ :
Revised: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ :

Original: V (cid:55)→ max π { r π + γ P π V } .
Revised: V (cid:55)→ max π { r π + γ P π V } .

Original: The greedy policy at state x ∈ X is
Revised: The greedy policy at state x ∈ X is

Original: [Equation]
Revised: [Equation]

Original: or more compactly, π g ( V ) ← argmax π T π V .
Revised: or more compactly, π g ( V ) ← argmax π T π V .

Original: We have T ∗ V
Revised: We have T ∗ V

Original: = T π g ( V ) V , that is, the effect of the
Revised: = T π g ( V ) V , that is, the effect of the

Original: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellman operator of the greedy policy w.r.t.
Revised: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellman operator of the greedy policy w.r.t.

Original: V to V .
Revised: V to V .

Original: 2.1 Value Iteration
Revised: 2.1 Value Iteration

Original: The value function V π and the optimal value function V ∗ are the ﬁxed points of the operators T π and T ∗ , respectively, and satisfy the Bellman equation.
Revised: The value function V π and the optimal value function V ∗ are the ﬁxed points of the operators T π and T ∗ , respectively, and satisfy the Bellman equation.

Original: For the PE problem, this means that
Revised: For the PE problem, this means that

Original: [Equation]
Revised: [Equation]

Original: There are several ways to compute the value function of a policy π or the optimal value function V ∗ , including the iterative methods such as VI and Policy Iteration (PI) algorithms, or solving a linear system of equations (for PE) or linear programming (for Control).
Revised: There are several ways to compute the value function of a policy π or the optimal value function V ∗ , including the iterative methods such as VI and Policy Iteration (PI) algorithms, or solving a linear system of equations (for PE) or linear programming (for Control).

Original: We focus on the VI algorithm in this work.
Revised: We focus on the VI algorithm in this work.

Original: VI repeatedly applies the Bellman operator to the most recent approximation of the value function:
Revised: VI repeatedly applies the Bellman operator to the most recent approximation of the value function:

Original: Given an initial value function V 0 , it generates a sequence ( V k ) k ≥ 0 as follows:
Revised: Given an initial value function V 0 , it generates a sequence ( V k ) k ≥ 0 as follows:

Original: [Equation]
Revised: [Equation]

Original: VI for Control can be written in an equivalent form: At iteration k , we ﬁrst compute the greedy policy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .
Revised: VI for Control can be written in an equivalent form: At iteration k , we ﬁrst compute the greedy policy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .

Original: Therefore, the policy improvement step is obtained through ﬁnding a policy that is greedy w.r.t.
Revised: Therefore, the policy improvement step is obtained through ﬁnding a policy that is greedy w.r.t.

Original: the last value function
Revised: the last value function

Original: V k − 1 , that is, the best policy if we only look one step ahead.
Revised: V k − 1 , that is, the best policy if we only look one step ahead.

Original: This form will be conductive for our later developments.
Revised: This form will be conductive for our later developments.

Original: As the Bellman operator is a γ -contraction w.r.t.
Revised: As the Bellman operator is a γ -contraction w.r.t.

Original: the supremum norm, the convergence rate of V k to V π (or V ∗ ) would be O ( γ k ) .
Revised: the supremum norm, the convergence rate of V k to V π (or V ∗ ) would be O ( γ k ) .

Original: This rate can be slow when γ is close to 1 .
Revised: This rate can be slow when γ is close to 1 .

Original: 2.2 Matrix splitting for solving linear system of equations
Revised: 2.2 Matrix splitting for solving linear system of equations

Original: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear system of equations (2.2).
Revised: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear system of equations (2.2).

Original: Consider the linear system Az = b , with A ∈ R d × d and z, b ∈ R d .
Revised: Consider the linear system Az = b , with A ∈ R d × d and z, b ∈ R d .

Original: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally, A , M , and N can be linear operators).
Revised: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally, A , M , and N can be linear operators).

Original: Therefore, z satisﬁes Mz =
Revised: Therefore, z satisﬁes Mz =

Original: Nz + b .
Revised: Nz + b .

Original: The splitting-based iterative approach deﬁnes the new approximation z k given the current z k − 1 by solving
Revised: The splitting-based iterative approach deﬁnes the new approximation z k given the current z k − 1 by solving

Original: [Equation]
Revised: [Equation]

Original: or equivalently,
Revised: or equivalently,

Original: [Equation]
Revised: [Equation]

Original: To analyze the convergence of this iterative method, consider the error e k (cid:44)
Revised: To analyze the convergence of this iterative method, consider the error e k (cid:44)

Original: z k − z .
Revised: z k − z .

Original: As z =
Revised: As z =

Original: M − 1 ( Nz + b ) , we have e k
Revised: M − 1 ( Nz + b ) , we have e k

Original: =
Revised: =

Original: M − 1 ( Nz k − 1 + b ) − M − 1 ( Nz + b )
Revised: M − 1 ( Nz k − 1 + b ) − M − 1 ( Nz + b )

Original: =
Revised: =

Original: M − 1 N ( z k − 1 − z ) , so the dynamics of the error is
Revised: M − 1 N ( z k − 1 − z ) , so the dynamics of the error is

Original: [Equation]
Revised: [Equation]

Original: Let G (cid:44) M − 1 N .
Revised: Let G (cid:44) M − 1 N .

Original: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as
Revised: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as

Original: [Equation]
Revised: [Equation]

Original: The errors are (norm-) convergent if (cid:107) G (cid:107)
Revised: The errors are (norm-) convergent if (cid:107) G (cid:107)

Original: =
Revised: =

Original: (cid:107) M − 1 N (cid:107) < 1 , for some choice of norm.
Revised: (cid:107) M − 1 N (cid:107) < 1 , for some choice of norm.

Original: More generally, the necessary and sufﬁcient condition for convergence is that the spectral radius ρ ( G ) , the maximum absolute value of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or
Revised: More generally, the necessary and sufﬁcient condition for convergence is that the spectral radius ρ ( G ) , the maximum absolute value of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or

Original: Theorem 11.2.1 of Golub and Van Loan [2013].
Revised: Theorem 11.2.1 of Golub and Van Loan [2013].

Original: The convergence is faster if the spectral radius (or norm) is closer to zero.
Revised: The convergence is faster if the spectral radius (or norm) is closer to zero.

Original: The success of this iterative procedure depends on how we choose M and N such that the norm (or spectral radius) is as small as possible.
Revised: The success of this iterative procedure depends on how we choose M and N such that the norm (or spectral radius) is as small as possible.

Original: Also we want to choose an M such that solving Mz k =
Revised: Also we want to choose an M such that solving Mz k =

Original: Nz k − 1 + b is not very expensive.
Revised: Nz k − 1 + b is not very expensive.

Original: For example, if M is an identity matrix I , we get that N
Revised: For example, if M is an identity matrix I , we get that N

Original: =
Revised: =

Original: I − A , and the iteration becomes z k
Revised: I − A , and the iteration becomes z k

Original: = ( I − A )
Revised: = ( I − A )

Original: z k − 1 + b .
Revised: z k − 1 + b .

Original: This iteration is convergent if ρ ( I − A ) < 1 .
Revised: This iteration is convergent if ρ ( I − A ) < 1 .

Original: Other commonly used choices lead to the Jacobi and Gauss-Seidel methods that are described in the supplementary material.
Revised: Other commonly used choices lead to the Jacobi and Gauss-Seidel methods that are described in the supplementary material.

Original: We are now ready to make the connection between splitting-based iterative methods and VI for PE.
Revised: We are now ready to make the connection between splitting-based iterative methods and VI for PE.

Original: If we choose A = I − γ P π , we see that equation
Revised: If we choose A = I − γ P π , we see that equation

Original: AV π = r π is indeed the Bellman equation for policy π (2.2).
Revised: AV π = r π is indeed the Bellman equation for policy π (2.2).

Original: The VI for PE, which is V k = γ P π V k − 1 + r π =
Revised: The VI for PE, which is V k = γ P π V k − 1 + r π =

Original: ( I − A )
Revised: ( I − A )

Original: V k − 1 + r π , corresponds to the choice of M
Revised: V k − 1 + r π , corresponds to the choice of M

Original: =
Revised: =

Original: I and N = γ P π .
Revised: I and N = γ P π .

Original: This brings up the question of whether it is possible to split A differently so that the resulting VI-like procedure has better convergence properties.
Revised: This brings up the question of whether it is possible to split A differently so that the resulting VI-like procedure has better convergence properties.

Original: We next suggest a particular choice.
Revised: We next suggest a particular choice.

Original:  Operator splitting value iteration algorithm
Revised:  Operator splitting value iteration algorithm

Original: We introduce the Operator Splitting Value Iteration (OS-VI) algorithm.
Revised: We introduce the Operator Splitting Value Iteration (OS-VI) algorithm.

Original: We start from the PE problem and develop the Control version based on that.
Revised: We start from the PE problem and develop the Control version based on that.

Original: We also present a visualization of how OS-VI works.
Revised: We also present a visualization of how OS-VI works.

Original: 3.1 OS-VI for policy evaluation
Revised: 3.1 OS-VI for policy evaluation

Original: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as
Revised: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as

Original: [Equation]
Revised: [Equation]

Original: Following the recipe of (2.4), the OS-VI algorithm for PE would be
Revised: Following the recipe of (2.4), the OS-VI algorithm for PE would be

Original: [Equation]
Revised: [Equation]

Original: starting from an initial V 0 .
Revised: starting from an initial V 0 .

Original: 3 To gain more intuition and prepare for further developments, we deﬁne a few notations.
Revised: 3 To gain more intuition and prepare for further developments, we deﬁne a few notations.

Original: We deﬁne the Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga (1928 – 2022) who has made signiﬁcant contributions to matrix analysis, as the mapping between the space of all bounded functions over X to the same space as
Revised: We deﬁne the Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga (1928 – 2022) who has made signiﬁcant contributions to matrix analysis, as the mapping between the space of all bounded functions over X to the same space as

Original: [Equation]
Revised: [Equation]

Original: Observe that (3.1) can be compactly written as
Revised: Observe that (3.1) can be compactly written as

Original: [Equation]
Revised: [Equation]

Original: It is not difﬁcult to see that S
Revised: It is not difﬁcult to see that S

Original: π V π = V π , i.e., the value function of a policy π is a ﬁxed-point of the
Revised: π V π = V π , i.e., the value function of a policy π is a ﬁxed-point of the

Original: Varga operator S π .
Revised: Varga operator S π .

Original: This and other properties of the Varga operator are shown in the supplementary material.
Revised: This and other properties of the Varga operator are shown in the supplementary material.

Original: Given any value function V , deﬁne an auxiliary reward function ¯ r V : X ×
Revised: Given any value function V , deﬁne an auxiliary reward function ¯ r V : X ×

Original: A → R as
Revised: A → R as

Original: [Equation]
Revised: [Equation]

Original: Similar to r π , we deﬁne the notation ¯ r πV :
Revised: Similar to r π , we deﬁne the notation ¯ r πV :

Original: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministic policy π (and similarly for a stochastic policy).
Revised: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministic policy π (and similarly for a stochastic policy).

Original: With this notation, the effect of applying S π to V is
Revised: With this notation, the effect of applying S π to V is

Original: [Equation]
Revised: [Equation]

Original: This is the value function of following π in an MDP with dynamics ˆ P and reward ¯ r V .
Revised: This is the value function of following π in an MDP with dynamics ˆ P and reward ¯ r V .

Original: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximate dynamics, and a reward function that consists of the original reward r and the correction term γ
Revised: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximate dynamics, and a reward function that consists of the original reward r and the correction term γ

Original: ( P − ˆ P ) V k − 1 .
Revised: ( P − ˆ P ) V k − 1 .

Original: The computation of this value function is a standard PE problem with the approximate model.
Revised: The computation of this value function is a standard PE problem with the approximate model.

Original: For instance, we may use another VI (PE) with dynamics ˆ P to solve it: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P
Revised: For instance, we may use another VI (PE) with dynamics ˆ P to solve it: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P

Original: π
Revised: π

Original: U i − 1 .
Revised: U i − 1 .

Original: This converges to S π V
Revised: This converges to S π V

Original: =
Revised: =

Original: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .
Revised: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .

Original: Note that aside from the computation of ¯ r πV , which requires querying P in order to compute the P π V term, this iterative process only uses the approximate model ˆ P , which is assumed to be cheap to access.
Revised: Note that aside from the computation of ¯ r πV , which requires querying P in order to compute the P π V term, this iterative process only uses the approximate model ˆ P , which is assumed to be cheap to access.

Original: What is the beneﬁt of this OS-VI procedure?
Revised: What is the beneﬁt of this OS-VI procedure?

Original: If the approximate model ˆ P is close to the true dynamics
Revised: If the approximate model ˆ P is close to the true dynamics

Original: P , this leads to a faster convergence of V k to V π , as shall be quantiﬁed soon.
Revised: P , this leads to a faster convergence of V k to V π , as shall be quantiﬁed soon.

Original: The faster convergence is in terms of the number of queries to P , which is assumed to be expensive.
Revised: The faster convergence is in terms of the number of queries to P , which is assumed to be expensive.

Original: To see this, consider the hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happens to perfectly match the reality.
Revised: To see this, consider the hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happens to perfectly match the reality.

Original: Then, S π V =
Revised: Then, S π V =

Original: ( I − γ P π ) − 1 ( r π + 0 V ) =
Revised: ( I − γ P π ) − 1 ( r π + 0 V ) =

Original: V π , and the value function for the original MDP is obtained in one iteration of OS-VI.
Revised: V π , and the value function for the original MDP is obtained in one iteration of OS-VI.

Original: Of course, we often can only hope for ˆ
Revised: Of course, we often can only hope for ˆ

Original: P ≈ P .
Revised: P ≈ P .

Original: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in more details, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ P is only a close approximation of P .
Revised: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in more details, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ P is only a close approximation of P .

Original: 3.2 OS-VI for control
Revised: 3.2 OS-VI for control

Original: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ← π g ( V k − 1 ) =
Revised: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ← π g ( V k − 1 ) =

Original: argmax π T π V k − 1 in its policy improvement step, and then uses one step of the Bellman operator w.r.t.
Revised: argmax π T π V k − 1 in its policy improvement step, and then uses one step of the Bellman operator w.r.t.

Original: the obtained policy π k to compute the new estimate of the value function V k ←
Revised: the obtained policy π k to compute the new estimate of the value function V k ←

Original: T π k V k − 1 , as described after (2.3).
Revised: T π k V k − 1 , as described after (2.3).

Original: The OS-VI algorithm for Control follows a similar structure with the difference that 1) the improved policy is the optimizer of the Varga operator, and not the Bellman operator, and 2) the new value function is obtained by applying the Varga operator of the newly obtained policy.
Revised: The OS-VI algorithm for Control follows a similar structure with the difference that 1) the improved policy is the optimizer of the Varga operator, and not the Bellman operator, and 2) the new value function is obtained by applying the Varga operator of the newly obtained policy.

Original: To be concrete, given a value function V , deﬁne the S -improved policy
Revised: To be concrete, given a value function V , deﬁne the S -improved policy

Original: [Equation]
Revised: [Equation]

Original: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .
Revised: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .

Original: We also deﬁne the Varga optimality operator S ∗ : B ( X ) → B ( X ) as
Revised: We also deﬁne the Varga optimality operator S ∗ : B ( X ) → B ( X ) as

Original: [Equation]
Revised: [Equation]

Original: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.
Revised: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.

Original: V applied to a value function V (compare it with T ∗ V =
Revised: V applied to a value function V (compare it with T ∗ V =

Original: T π g ( V ) V ).
Revised: T π g ( V ) V ).

Original: The OS-VI (Control) is then simply
Revised: The OS-VI (Control) is then simply

Original: [Equation]
Revised: [Equation]

Original: which in its expanded form, consists of the following two steps:
Revised: which in its expanded form, consists of the following two steps:

Original: [Equation]
Revised: [Equation]

Original: Comparing the S -improved policy (3.4) used in the policy improvement step (3.6) of OS-VI with the conventional greedy policy (2.1) is insightful.
Revised: Comparing the S -improved policy (3.4) used in the policy improvement step (3.6) of OS-VI with the conventional greedy policy (2.1) is insightful.

Original: The greedy policy is argmax π T π V .
Revised: The greedy policy is argmax π T π V .

Original: Expanding
Revised: Expanding

Original: T π V , we see that the greedy policy is the maximizer of r π + γ P π V .
Revised: T π V , we see that the greedy policy is the maximizer of r π + γ P π V .

Original: The function r π + γ P π V is a single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy, is in general different from the optimal policy, which maximizes V π .
Revised: The function r π + γ P π V is a single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy, is in general different from the optimal policy, which maximizes V π .

Original: On the other hand, the S -improved policy π V ( V ) solves a full MDP with an approximate model ˆ P and a reward function that has both the original reward r and the correction term γ
Revised: On the other hand, the S -improved policy π V ( V ) solves a full MDP with an approximate model ˆ P and a reward function that has both the original reward r and the correction term γ

Original: ( P − ˆ P ) V .
Revised: ( P − ˆ P ) V .

Original: In the special case that ˆ P = P , the correction term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.
Revised: In the special case that ˆ P = P , the correction term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.

Original: As often ˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.
Revised: As often ˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.

Original: The partial policy evaluation step (3.7) updates the value function to a value that is closer to the optimal value function.
Revised: The partial policy evaluation step (3.7) updates the value function to a value that is closer to the optimal value function.

Original: Remark.
Revised: Remark.

Original: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context of dynamic programming is not completely novel to this work.
Revised: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context of dynamic programming is not completely novel to this work.

Original: Kushner and Kleinman [1971] is one of the earliest paper that mentions the Jacobi and Gauss-Seidel procedures for computing the value function.
Revised: Kushner and Kleinman [1971] is one of the earliest paper that mentions the Jacobi and Gauss-Seidel procedures for computing the value function.

Original: Porteus [1975] proposes several transformations to the reward and probability transition matrix with the goal of improving the computational cost of solving the transformed MDP.
Revised: Porteus [1975] proposes several transformations to the reward and probability transition matrix with the goal of improving the computational cost of solving the transformed MDP.

Original: One of the transformations, called pre-inverse transform , has some similarities with the operator splitting of this work.
Revised: One of the transformations, called pre-inverse transform , has some similarities with the operator splitting of this work.

Original: The end result, however, is different.
Revised: The end result, however, is different.

Original: Bacon and Precup [2016] offer a matrix splitting perspective on planning with options.
Revised: Bacon and Precup [2016] offer a matrix splitting perspective on planning with options.

Original: The connection between multi-step models and matrix splitting is developed in Chapter 4 of Bacon [2018].
Revised: The connection between multi-step models and matrix splitting is developed in Chapter 4 of Bacon [2018].

Original: Refer to the supplementary material for more discussion.
Revised: Refer to the supplementary material for more discussion.

Original: 3.3 Visualizing how OS-VI works
Revised: 3.3 Visualizing how OS-VI works

Original: To present some intuition on how OS-VI works, we visualize the value function trajectories of several algorithms, including OS-VI, on a 2-state MDP, in Figure 1.
Revised: To present some intuition on how OS-VI works, we visualize the value function trajectories of several algorithms, including OS-VI, on a 2-state MDP, in Figure 1.

Original: We consider the policy evaluation for the dynamics P π = [ 0 . 9 0 .
Revised: We consider the policy evaluation for the dynamics P π = [ 0 . 9 0 .

Original: 9 ] with the reward
Revised: 9 ] with the reward

Original: r π = (cid:0) 1 − 0 .
Revised: r π = (cid:0) 1 − 0 .

Original: 5 (cid:1) and γ = 0 .
Revised: 5 (cid:1) and γ = 0 .

Original: We consider two approximate models: a relatively accurate ˆ P π accurate =
Revised: We consider two approximate models: a relatively accurate ˆ P π accurate =

Original: 95 ] , and an inaccurate ˆ P π inaccurate =
Revised: 95 ] , and an inaccurate ˆ P π inaccurate =

Original: In addition to OS-VI (PE), the ﬁrst algorithm is the conventional VI (PE), which repeatedly applies the Bellman operator according to the true model P π to the most recent approximation of the value function.
Revised: In addition to OS-VI (PE), the ﬁrst algorithm is the conventional VI (PE), which repeatedly applies the Bellman operator according to the true model P π to the most recent approximation of the value function.

Original: We use T π P to refer to its Bellman operator and to label the corresponding trajectory in the value space.
Revised: We use T π P to refer to its Bellman operator and to label the corresponding trajectory in the value space.

Original: This algorithm converges to the true value function V π .
Revised: This algorithm converges to the true value function V π .

Original: The second algorithm is VI (PE) that uses ˆ P π as the model.
Revised: The second algorithm is VI (PE) that uses ˆ P π as the model.

Original: This procedure is the basis of the Dyna architecture.
Revised: This procedure is the basis of the Dyna architecture.

Original: We use T π ˆ
Revised: We use T π ˆ

Original: P to refer to its Bellman operator and to label the corresponding trajectory in the value space.
Revised: P to refer to its Bellman operator and to label the corresponding trajectory in the value space.

Original: Due to the error of
Revised: Due to the error of

Original: T π ˆ
Revised: T π ˆ

Original: P compared to T π P , the algorithm converges to a wrong value function ˆ V π , as both ﬁgures show.
Revised: P compared to T π P , the algorithm converges to a wrong value function ˆ V π , as both ﬁgures show.

Original: We observe that even when the model is relatively accurate as in Figure 1a, the converged value function is quite wrong.
Revised: We observe that even when the model is relatively accurate as in Figure 1a, the converged value function is quite wrong.

Original: This illustrates one limitation of the conventional model-based RL algorithms where an inaccurate model may lead to signiﬁcantly inaccurate estimate of the value function.
Revised: This illustrates one limitation of the conventional model-based RL algorithms where an inaccurate model may lead to signiﬁcantly inaccurate estimate of the value function.

Original: V (0) −−−
Revised: V (0) −−−

Original: V ( 1 )
Revised: V ( 1 )

Original: Vˆ V π
Revised: Vˆ V π

Original: V π
Revised: V π

Original: T π P T π ˆ
Revised: T π P T π ˆ

Original: P ¯ T πk
Revised: P ¯ T πk

Original: S π (a) More accurate model ˆ P
Revised: S π (a) More accurate model ˆ P

Original: accurate
Revised: accurate

Original: V (0) −−−
Revised: V (0) −−−

Original: V ( 1 )
Revised: V ( 1 )

Original: Vˆ V π
Revised: Vˆ V π

Original: V π
Revised: V π

Original: T π P T π ˆ
Revised: T π P T π ˆ

Original: P ¯ T πk
Revised: P ¯ T πk

Original: S π (
Revised: S π (

Original: [Figure]
Revised: [Figure]

Original: The OS-VI algorithm repeatedly applies the Varga operator S π to the most recent approximation of the value function.
Revised: The OS-VI algorithm repeatedly applies the Varga operator S π to the most recent approximation of the value function.

Original: As discussed earlier, each computation of S π V k − 1 corresponds to solving an auxiliary MDPs ( X , A , ¯ r V k − 1 , ˆ P , γ ) .
Revised: As discussed earlier, each computation of S π V k − 1 corresponds to solving an auxiliary MDPs ( X , A , ¯ r V k − 1 , ˆ P , γ ) .

Original: We denote the Bellman operator of this auxiliary MDP by ¯ T πk .
Revised: We denote the Bellman operator of this auxiliary MDP by ¯ T πk .

Original: The ﬁgures show the trajectory generated by the iterative application of S π on the most recent value function as well as the trajectory for solving each auxiliary MDP, indicated by ¯ T πk .
Revised: The ﬁgures show the trajectory generated by the iterative application of S π on the most recent value function as well as the trajectory for solving each auxiliary MDP, indicated by ¯ T πk .

Original: We observe that the OS-VI algorithm converges to the correct value function despite using an incorrect model.
Revised: We observe that the OS-VI algorithm converges to the correct value function despite using an incorrect model.

Original: When the model is more accurate, very few iteration of OS-VI gets a value close to V π (two iterations in
Revised: When the model is more accurate, very few iteration of OS-VI gets a value close to V π (two iterations in

Original: Figure 1a)
Revised: Figure 1a)

Original: ; when the model is less accurate, a few more iterations are needed.
Revised: ; when the model is less accurate, a few more iterations are needed.

Original: Compared to VI, at least in these examples, the total number of iterations of OS-VI is signiﬁcantly smaller.
Revised: Compared to VI, at least in these examples, the total number of iterations of OS-VI is signiﬁcantly smaller.

Original: When the initial value function is V 0 = 0 , the result of the ﬁrst iteration of OS-VI is the same value function computed by the VI with the wrong model ˆ P π .
Revised: When the initial value function is V 0 = 0 , the result of the ﬁrst iteration of OS-VI is the same value function computed by the VI with the wrong model ˆ P π .

Original: This is because V 1 ← S π V 

Revised: This is because V 1 ← S π V

Original: = ( I − γ ˆ P π ) − 1 ¯ r πV 0 and ¯ r πV 0 =
Revised: = ( I − γ ˆ P π ) − 1 ¯ r πV 0 and ¯ r πV 0 =

Original: r π , so V 1 =
Revised: r π , so V 1 =

Original: ( I − γ ˆ P π ) − 1 r π , the same solution as the value function obtained using the approximate model ˆ P .
Revised: ( I − γ ˆ P π ) − 1 r π , the same solution as the value function obtained using the approximate model ˆ P .

Original: In these ﬁgures, this shows itself by the overlapping of the red arrows followed by T
Revised: In these ﬁgures, this shows itself by the overlapping of the red arrows followed by T

Original: π ˆ
Revised: π ˆ

Original: P and the ﬁrst segment of the orange arrows, which are generated by the repeated application of ¯ T π 1 .
Revised: P and the ﬁrst segment of the orange arrows, which are generated by the repeated application of ¯ T π 1 .

Original: In further iterations of OS-VI, the auxiliary MDPs change and the path followed by ¯ T πk ( k ≥ 2 ) deviates from the solution of the VI with the wrong model.
Revised: In further iterations of OS-VI, the auxiliary MDPs change and the path followed by ¯ T πk ( k ≥ 2 ) deviates from the solution of the VI with the wrong model.

Original:  Convergence analysis of operator splitting value iteration
Revised:  Convergence analysis of operator splitting value iteration

Original: In this section, we present the convergence analysis of OS-VI.
Revised: In this section, we present the convergence analysis of OS-VI.

Original: Our results show that OS-VI has an
Revised: Our results show that OS-VI has an

Original: O ( γ (cid:48) k ) convergence rate for an effective discount factor γ (cid:48) that depends on the error between ˆ P and
Revised: O ( γ (cid:48) k ) convergence rate for an effective discount factor γ (cid:48) that depends on the error between ˆ P and

Original: P .
Revised: P .

Original: For small enough error, γ (cid:48) < γ and OS-VI has a faster convergence rate compared to the classic
Revised: For small enough error, γ (cid:48) < γ and OS-VI has a faster convergence rate compared to the classic

Original: VI, Policy Iteration (PI), and Modiﬁed Policy Iteration (MPI), which all have O ( γ k ) behaviour.
Revised: VI, Policy Iteration (PI), and Modiﬁed Policy Iteration (MPI), which all have O ( γ k ) behaviour.

Original: We provide results for both the L ∞ and L p norms.
Revised: We provide results for both the L ∞ and L p norms.

Original: 4.1 Convergence of OS-VI for policy evaluation
Revised: 4.1 Convergence of OS-VI for policy evaluation

Original: We study the convergence behaviour of OS-VI (PE) in presence of error in value updates.
Revised: We study the convergence behaviour of OS-VI (PE) in presence of error in value updates.

Original: Speciﬁcally, we consider that at each iteration k , the update (3.2) has an error, i.e.,
Revised: Speciﬁcally, we consider that at each iteration k , the update (3.2) has an error, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The error (cid:15) value k encompasses various sources of errors that might occur in solving the auxiliary
Revised: The error (cid:15) value k encompasses various sources of errors that might occur in solving the auxiliary

Original: MDP ( X , A , ¯ r V k − 1 , ˆ P , γ ) .
Revised: MDP ( X , A , ¯ r V k − 1 , ˆ P , γ ) .

Original: One source is the function approximation error due to using a function approximator to represent V k , which is often required in large state spaces.
Revised: One source is the function approximation error due to using a function approximator to represent V k , which is often required in large state spaces.

Original: Another is the estimation (i.e., statistical) error caused due to having a ﬁnite number of samples, instead of direct access to P , in the RL setting.
Revised: Another is the estimation (i.e., statistical) error caused due to having a ﬁnite number of samples, instead of direct access to P , in the RL setting.

Original: Refer to Munos and Szepesvári [2008], Antos et al.
Revised: Refer to Munos and Szepesvári [2008], Antos et al.

Original: [2008], Farahmand et al.
Revised: [2008], Farahmand et al.

Original: Chen and Jiang [2019], Fan et al.
Revised: Chen and Jiang [2019], Fan et al.

Original: [2019] for the discussion of function approximation and estimation errors in the RL context.
Revised: [2019] for the discussion of function approximation and estimation errors in the RL context.

Original: In this work, we do not analyze how the number of samples, the function approximator, etc.
Revised: In this work, we do not analyze how the number of samples, the function approximator, etc.

Original: affect the errors (cid:15) value k .
Revised: affect the errors (cid:15) value k .

Original: We offer error propagation results similar to Munos [2007] (approximate VI), Munos [2003] (approximate PI), and Scherrer et al.
Revised: We offer error propagation results similar to Munos [2007] (approximate VI), Munos [2003] (approximate PI), and Scherrer et al.

Original: [2015] (approximate modiﬁed
Revised: [2015] (approximate modiﬁed

Original: PI) for approximate OS-VI.
Revised: PI) for approximate OS-VI.

Original: To study the convergence behaviour of OS-VI (PE), let G π
Revised: To study the convergence behaviour of OS-VI (PE), let G π

Original: = ( I − γ ˆ P π ) − 1 γ
Revised: = ( I − γ ˆ P π ) − 1 γ

Original: ( P π − ˆ P π ) .
Revised: ( P π − ˆ P π ) .

Original: We use the fact that S
Revised: We use the fact that S

Original: π V π = V π
Revised: π V π = V π

Original: and write
Revised: and write

Original: [Equation]
Revised: [Equation]

Original: Now, we have that
Revised: Now, we have that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that for any square matrix F with a matrix norm (cid:107) F (cid:107) p < 1 , it holds that (cid:107) ( I − F ) − 1 (cid:107) p ≤ 1 1 −(cid:107) F (cid:107) p
Revised: where we used the fact that for any square matrix F with a matrix norm (cid:107) F (cid:107) p < 1 , it holds that (cid:107) ( I − F ) − 1 (cid:107) p ≤ 1 1 −(cid:107) F (cid:107) p

Original: (see Lemma 2.3.3 of Golub and Van Loan 2013), and that the supremum norm of a stochastic matrix ˆ P π is 1 .
Revised: (see Lemma 2.3.3 of Golub and Van Loan 2013), and that the supremum norm of a stochastic matrix ˆ P π is 1 .

Original: Assuming that (cid:107) (cid:15) value k (cid:107) ∞ ≤ (cid:15) value for all k ≥ 1 and deﬁning effective discount factor γ (cid:48)
Revised: Assuming that (cid:107) (cid:15) value k (cid:107) ∞ ≤ (cid:15) value for all k ≥ 1 and deﬁning effective discount factor γ (cid:48)

Original: =
Revised: =

Original: γ 1 − γ (cid:107)P π − ˆ P π (cid:107) ∞ , the upper bounds (4.2) and (4.3) lead to (cid:107) V π − V k (cid:107) ∞
Revised: γ 1 − γ (cid:107)P π − ˆ P π (cid:107) ∞ , the upper bounds (4.2) and (4.3) lead to (cid:107) V π − V k (cid:107) ∞

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: (cid:48) (cid:15) value .
Revised: (cid:48) (cid:15) value .

Original: A few remarks are in order.
Revised: A few remarks are in order.

Original: First, whenever γ (cid:48) < γ , this is guaranteed to be faster than the convergence rate of the conventional VI, which is O ( γ k ) .
Revised: First, whenever γ (cid:48) < γ , this is guaranteed to be faster than the convergence rate of the conventional VI, which is O ( γ k ) .

Original: This happens if (cid:107)P π − ˆ P π (cid:107) ∞ < 1 − γ .
Revised: This happens if (cid:107)P π − ˆ P π (cid:107) ∞ < 1 − γ .

Original: If the model is very accurate, we obtain much faster rate than VI’s.
Revised: If the model is very accurate, we obtain much faster rate than VI’s.

Original: Since each iteration k corresponds to a query to the true model P , a faster rate entails that the algorithm requires fewer number of queries to the expensive model to reach the same level of accuracy.
Revised: Since each iteration k corresponds to a query to the true model P , a faster rate entails that the algorithm requires fewer number of queries to the expensive model to reach the same level of accuracy.

Original: Second, although the model error (cid:107)P π − ˆ P π (cid:107) ∞ is a reasonable choice to measure the distances between distributions (it is the maximum of the Total Variation distance between P π ( ·| x ) and ˆ P π ( ·| x ) over x , which itself can be upper bounded by their KL divergence; see the supplementary material), it is somewhat conservative as it takes the supremum over the state space.
Revised: Second, although the model error (cid:107)P π − ˆ P π (cid:107) ∞ is a reasonable choice to measure the distances between distributions (it is the maximum of the Total Variation distance between P π ( ·| x ) and ˆ P π ( ·| x ) over x , which itself can be upper bounded by their KL divergence; see the supplementary material), it is somewhat conservative as it takes the supremum over the state space.

Original: Likewise, requiring (cid:107) (cid:15) value k (cid:107) ∞ to be small is also conservative, as approximating S π V k − 1 using a function approximator given samples (RL setting) often leads to an L p -norm type of guarantee.
Revised: Likewise, requiring (cid:107) (cid:15) value k (cid:107) ∞ to be small is also conservative, as approximating S π V k − 1 using a function approximator given samples (RL setting) often leads to an L p -norm type of guarantee.

Original: We now provide a different analysis to address these issues.
Revised: We now provide a different analysis to address these issues.

Original: To present the L p -norm result, we need to deﬁne some notations.
Revised: To present the L p -norm result, we need to deﬁne some notations.

Original: First, we deﬁne the conditional discounted future-state distribution of policy π under ˆ P as the following probability distribution:
Revised: First, we deﬁne the conditional discounted future-state distribution of policy π under ˆ P as the following probability distribution:

Original: Given a measurable set B , we have ˆ η π ( B | x ) =
Revised: Given a measurable set B , we have ˆ η π ( B | x ) =

Original: [Equation]
Revised: [Equation]

Original: 1 − γ )
Revised: 1 − γ )

Original: [Equation]
Revised: [Equation]

Original: ∞ t =
Revised: ∞ t =

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: X t ∈ B | X 

Revised: X t ∈ B | X

Original: = x, π, ˆ P (cid:17) , where the chain ( X t ) t ≥ 0 starts from state x and evolves by following policy π under transitions ˆ P .
Revised: = x, π, ˆ P (cid:17) , where the chain ( X t ) t ≥ 0 starts from state x and evolves by following policy π under transitions ˆ P .

Original: For an arbitrary distribution ρ over the state space, we deﬁne the discounted future-state distribution concentration coefﬁcient of the approximate model as
Revised: For an arbitrary distribution ρ over the state space, we deﬁne the discounted future-state distribution concentration coefﬁcient of the approximate model as

Original: [Equation]
Revised: [Equation]

Original: Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.
Revised: Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.

Original: the distribution ρ .
Revised: the distribution ρ .

Original: It is assumed that for any x ∈ X , ˆ η π ( ·| x ) (cid:28) ρ , i.e., ˆ η π ( ·| x ) is absolutely continuous w.r.t.
Revised: It is assumed that for any x ∈ X , ˆ η π ( ·| x ) (cid:28) ρ , i.e., ˆ η π ( ·| x ) is absolutely continuous w.r.t.

Original: ρ (otherwise, the coefﬁcient would be set to inﬁnity).
Revised: ρ (otherwise, the coefﬁcient would be set to inﬁnity).

Original: This coefﬁcient measures how concentrated the distribution ˆ η π ( ·| x ) is compared to ρ .
Revised: This coefﬁcient measures how concentrated the distribution ˆ η π ( ·| x ) is compared to ρ .

Original: This is weighted according to the state distribution ρ .
Revised: This is weighted according to the state distribution ρ .

Original: Similar concentrability coefﬁcients, but not exactly this one, have appeared in the error propagation results [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al., 2015].
Revised: Similar concentrability coefﬁcients, but not exactly this one, have appeared in the error propagation results [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al., 2015].

Original: Finally, we deﬁne the weighted χ 2 -divergence of ˆ P π and P π as
Revised: Finally, we deﬁne the weighted χ 2 -divergence of ˆ P π and P π as

Original: [Equation]
Revised: [Equation]

Original: This notion of model error is less strict in requiring accurate approximation P in all states.
Revised: This notion of model error is less strict in requiring accurate approximation P in all states.

Original: Usually only a subset of the state space is important or even reachable in a problem.
Revised: Usually only a subset of the state space is important or even reachable in a problem.

Original: The above model error can focus on only speciﬁc areas of the state space through the choice of distribution ρ .
Revised: The above model error can focus on only speciﬁc areas of the state space through the choice of distribution ρ .

Original: We are now ready to present the main theorem for the approximate OS-VI (PE).
Revised: We are now ready to present the main theorem for the approximate OS-VI (PE).

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider the approximate OS-VI algorithm for PE (4.1) .
Revised: Consider the approximate OS-VI algorithm for PE (4.1) .

Original: Let (cid:107)·(cid:107) (cid:63) be either the supremum norm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over the state space ( (cid:63) = 4 , ρ ).
Revised: Let (cid:107)·(cid:107) (cid:63) be either the supremum norm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over the state space ( (cid:63) = 4 , ρ ).

Original: Assume that (cid:107) (cid:15) value k (cid:107) (cid:63) ≤ (cid:15) value for all k ≥ 1 .
Revised: Assume that (cid:107) (cid:15) value k (cid:107) (cid:63) ≤ (cid:15) value for all k ≥ 1 .

Original: Furthermore, deﬁne the effective discount factor as
Revised: Furthermore, deﬁne the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: For any k ≥ 0 , we have
Revised: For any k ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: 4.2 Convergence of OS-VI for control
Revised: 4.2 Convergence of OS-VI for control

Original: We turn to analyzing OS-VI for Control.
Revised: We turn to analyzing OS-VI for Control.

Original: We consider two types of errors: The ﬁrst is the error between the computed value function and the true optimal value function of the auxiliary MDP, i.e.,
Revised: We consider two types of errors: The ﬁrst is the error between the computed value function and the true optimal value function of the auxiliary MDP, i.e.,

Original: V k − S ∗ V k − 1 .
Revised: V k − S ∗ V k − 1 .

Original: The second is the suboptimality of obtained policy compared to the optimal policy of the auxiliary MDP, i.e., S π k V k − 1 − S ∗ V k − 1 .
Revised: The second is the suboptimality of obtained policy compared to the optimal policy of the auxiliary MDP, i.e., S π k V k − 1 − S ∗ V k − 1 .

Original: Concretely, we have
Revised: Concretely, we have

Original: [Equation]
Revised: [Equation]

Original: We have the following result for the approximate OS-VI (Control).
Revised: We have the following result for the approximate OS-VI (Control).

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .
Revised: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .

Original: Let (cid:107)·(cid:107) (cid:63) be either the supremum norm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over the state space ( (cid:63) = 4 , ρ ).
Revised: Let (cid:107)·(cid:107) (cid:63) be either the supremum norm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over the state space ( (cid:63) = 4 , ρ ).

Original: For any k ≥ 1 , let Π k
Revised: For any k ≥ 1 , let Π k

Original: =
Revised: =

Original: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 1 ≤ i < k } .
Revised: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 1 ≤ i < k } .

Original: Assume that (cid:107) (cid:15) value k
Revised: Assume that (cid:107) (cid:15) value k

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:15) value for all k ≥ 1 .
Revised: ≤ (cid:15) value for all k ≥ 1 .

Original: Furthermore, deﬁne the effective discount factor as
Revised: Furthermore, deﬁne the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: We can compare this result with the convergence result of VI.
Revised: We can compare this result with the convergence result of VI.

Original: For VI with the supremum norm, following the proof of Equation (2.2) by Munos [2007], we can show that (cid:107) V ∗ − V π k (cid:107) ∞ ≤
Revised: For VI with the supremum norm, following the proof of Equation (2.2) by Munos [2007], we can show that (cid:107) V ∗ − V π k (cid:107) ∞ ≤

Original: [Equation]
Revised: [Equation]

Original: , with (cid:107) V i − T ∗ V i − 1 (cid:107) ∞ ≤ (cid:15) value for all i < k (similar result for the L p -norm also holds, see Theorem 5.2 in Munos 2007).
Revised: , with (cid:107) V i − T ∗ V i − 1 (cid:107) ∞ ≤ (cid:15) value for all i < k (similar result for the L p -norm also holds, see Theorem 5.2 in Munos 2007).

Original: For the approximate VI, the initial error (cid:107) V ∗ − V 0 (cid:107) ∞ decays with the rate of O ( γ k ) .
Revised: For the approximate VI, the initial error (cid:107) V ∗ − V 0 (cid:107) ∞ decays with the rate of O ( γ k ) .

Original: This should be compared with O ( γ (cid:48) k ) rate of OS-VI.
Revised: This should be compared with O ( γ (cid:48) k ) rate of OS-VI.

Original: The effect of error at each step (cid:15) value is also similar: approximate VI has (1 − γ ) − 2 dependence while approximate OS-VI has (1 − γ (cid:48) )
Revised: The effect of error at each step (cid:15) value is also similar: approximate VI has (1 − γ ) − 2 dependence while approximate OS-VI has (1 − γ (cid:48) )

Original: − 2 .
Revised: − 2 .

Original: What is remarkable is that as opposed to γ , which is a ﬁxed parameter of the problem and can be close to 1 , γ (cid:48) can be made arbitrary close to zero when the approximate model ˆ P becomes more accurate.
Revised: What is remarkable is that as opposed to γ , which is a ﬁxed parameter of the problem and can be close to 1 , γ (cid:48) can be made arbitrary close to zero when the approximate model ˆ P becomes more accurate.

Original: The additional information given by ˆ P allows us to get much faster rate than VI.
Revised: The additional information given by ˆ P allows us to get much faster rate than VI.

Original: Of course, this requires the model to be accurate.
Revised: Of course, this requires the model to be accurate.

Original: An inaccurate model might be detrimental to the convergence rate, and may even lead to divergence.
Revised: An inaccurate model might be detrimental to the convergence rate, and may even lead to divergence.

Original: Similar conclusions can be made in comparing OS-VI with Policy Iteration and Modiﬁed Policy Iteration, as discussed in the supplementary material.
Revised: Similar conclusions can be made in comparing OS-VI with Policy Iteration and Modiﬁed Policy Iteration, as discussed in the supplementary material.

Original:  Operator splitting Dyna
Revised:  Operator splitting Dyna

Original: In the RL setting, we only have access to samples from P .
Revised: In the RL setting, we only have access to samples from P .

Original: MBRL algorithms, such as variants of the Dyna architecture, learn ˆ P from those samples, and use it to ﬁnd the value function or policy.
Revised: MBRL algorithms, such as variants of the Dyna architecture, learn ˆ P from those samples, and use it to ﬁnd the value function or policy.

Original: The learned model ˆ P is generally different from P due to the ﬁniteness of the samples as well as the possibility of model approximation error: the true dynamics P may not be representable with the function approximator used to represent ˆ P .
Revised: The learned model ˆ P is generally different from P due to the ﬁniteness of the samples as well as the possibility of model approximation error: the true dynamics P may not be representable with the function approximator used to represent ˆ P .

Original: This is another way to say that the world may be too big to be represented by our models.
Revised: This is another way to say that the world may be too big to be represented by our models.

Original: A MBRL algorithm that uses ˆ P in lieu of P does not ﬁnd the true value of the true MDP.
Revised: A MBRL algorithm that uses ˆ P in lieu of P does not ﬁnd the true value of the true MDP.

Original: Based on OS-VI, we propose OS-Dyna, as a hybrid model-based and model-free RL algorithm that takes advantage of both the true environment and the model in its updates, and can converge to the true value function despite using inaccurate ˆ P .
Revised: Based on OS-VI, we propose OS-Dyna, as a hybrid model-based and model-free RL algorithm, that takes advantage of both the true environment and the model in its updates and can converge to the true value function despite using inaccurate ˆ P .

Original: Learning ˆ P in OS-Dyna is similar to other MBRL algorithms [Moerland et al., 2022]: one can use various model learning approaches, either based on maximum likelihood estimate or a decision-aware model learning approach, to learn the model.
Revised: Learning ˆ P in OS-Dyna is similar to other MBRL algorithms [Moerland et al., 2022]: one can use various model learning approaches, either based on maximum likelihood estimate or a decision-aware model learning approach, to learn the model.

Original: Given a learned ˆ P , we can compute V k from the auxiliary reward function ¯ r k (cid:44) ¯ r V k − 1 by solving the PE or the Control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.
Revised: Given a learned ˆ P , we can compute V k from the auxiliary reward function ¯ r k (cid:44) ¯ r V k − 1 by solving the PE or the Control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.

Original: As V k is a function of ¯ r k , we focus on how ¯ r k should be estimated.
Revised: As V k is a function of ¯ r k , we focus on how ¯ r k should be estimated.

Original: The update rule of ¯ r k in OS-VI entails that for every ( x, a ) , we have
Revised: The update rule of ¯ r k in OS-VI entails that for every ( x, a ) , we have

Original: [Equation]
Revised: [Equation]

Original: We update our estimation of ¯ r using samples, as shall be discussed soon, and then the value function is updated to V π (¯ r, ˆ P ) (for PE) or V ∗ (¯ r, ˆ P ) (for Control) with most recent estimate of ¯ r .
Revised: We update our estimation of ¯ r using samples, as shall be discussed soon, and then the value function is updated to V π (¯ r, ˆ P ) (PE) or V ∗ (¯ r, ˆ P ) (Control) with most recent estimate of ¯ r .

Original: The challenge is that the above update rules need access to distribution P ( ·| x, a ) for every ( x, a ) , while we only have samples from P at some ( x, a ) pairs.
Revised: The challenge is that the above update rules need access to distribution P ( ·| x, a ) for every ( x, a ) , while we only have samples from P at some ( x, a ) pairs.

Original: Fortunately, this challenge has been tackled in developing sample-based algorithms based on the classic VI:
Revised: Fortunately, this challenge has been tackled in developing sample-based algorithms based on the classic VI:

Original: [Equation]
Revised: [Equation]

Original: where V k − 1 =
Revised: where V k − 1 =

Original: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a (cid:48)
Revised: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a (cid:48)

Original: Q k − 1 ( x, a ) in Control.
Revised: Q k − 1 ( x, a ) in Control.

Original: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Value Iteration and
Revised: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Value Iteration and

Original: Stochastic Approximation (SA)
Revised: Stochastic Approximation (SA)

Original: [Borkar, 2008].
Revised: [Borkar, 2008].

Original: In this paper we use SA to develop OS-Dyna [Equation] but we point out that other algorithms and techniques can also be applied to develop other versions of
Revised: In this paper we use SA to develop OS-Dyna, but we point out that other algorithms and techniques can also be applied to develop other versions of

Original: OS-Dyna.
Revised: OS-Dyna.

Original: The key step in SA is to use samples to form an unbiased estimate of the intended update value.
Revised: The key step in SA is to use samples to form an unbiased estimate of the intended update value.

Original: For a step in the true environment leading to ( X t , A t , R t , X (cid:48) t ) tuple, we can have the estimate
Revised: For a step in the true environment leading to ( X t , A t , R t , X (cid:48) t ) tuple, we can have the estimate

Original: [Equation]
Revised: [Equation]

Original: )] , where the expectation can also be estimated by samples from ˆ P ( ·| X t , A t ) .
Revised: )] , where the expectation can also be estimated by samples from ˆ P ( ·| X t , A t ) .

Original: This estimate Z of the update rule can then be used to update ¯ r .
Revised: This estimate Y t of the update rule can then be used to update ¯ r .

Original: As an example, for a ﬁnite state-action problem, the update rule is
Revised: As an example, for a ﬁnite state-action problem, the update rule is

Original: [Equation]
Revised: [Equation]

Original: where α t is the learning rate.
Revised: where α t is the learning rate.

Original: The ﬁnal procedure of OS-Dyna is presented in Algorithm 1.
Revised: The ﬁnal procedure of OS-Dyna is presented in Algorithm 1.

Original:  Experiments
Revised:  Experiments

Original: We evaluate both OS-VI and OS-Dyna in a ﬁnite MDP and compare them with existing methods.
Revised: We evaluate both OS-VI and OS-Dyna in a ﬁnite MDP and compare them with existing methods.

Original: Here we present the results for the Control problem on a modiﬁed cliffwalk environment in a 6 ×grid with 4 actions (UP, DOWN, LEFT, RIGHT).
Revised: Here we present the results for the Control problem on a modiﬁed cliffwalk environment in a 6 ×grid with 4 actions (UP, DOWN, LEFT, RIGHT).

Original: We postpone studying the PE problem, the results for other environments, and other relevant details to the supplementary material.
Revised: We postpone studying the PE problem, the results for other environments, and other relevant details to the supplementary material.

Original: Our convergence analysis shows that the convergence rates of our algorithms depend on the accuracy of ˆ P .
Revised: Our convergence analysis shows that the convergence rates of our algorithms depend on the accuracy of ˆ P .

Original: To test
Revised: To test

Original: OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothed model ˆ P of transitions P with smoothing parameter λ as
Revised: OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothed model ˆ P of transitions P with smoothing parameter λ as

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: m a li z e d || V k
Revised: m a li z e d || V k

Original: V * ||
Revised: V * ||

Original: [Figure]
Revised: [Figure]

Original: where U ( A ) for some set A is the uniform distribution over A .
Revised: where U ( A ) for some set A is the uniform distribution over A .

Original: Here, λ allows making adjustments to the amount of error introduced in ˆ P w.r.t.
Revised: Here, λ allows making adjustments to the amount of error introduced in ˆ P w.r.t.

Original: P .
Revised: P .

Original: If λ = 0 , ˆ P
Revised: If λ = 0 , ˆ P

Original: = P will be the accurate model, and if λ = 1 , ˆ P will be uniform over possible next states in P .
Revised: = P will be the accurate model, and if λ = 1 , ˆ P will be uniform over possible next states in P .

Original: The left plot in Figure 2 shows the convergence of OS-VI compared to VI and the solutions the model itself would lead to.
Revised: The left plot in Figure 2 shows the convergence of OS-VI compared to VI and the solutions the model itself would lead to.

Original: The plot shows normalized error of V π k w.r.t V ∗ , i.e., (cid:107) V
Revised: The plot shows normalized error of V π k w.r.t V ∗ , i.e., (cid:107) V

Original: π k − V ∗ (cid:107) 1 / (cid:107) V ∗ (cid:107) 1 .
Revised: π k − V ∗ (cid:107) 1 / (cid:107) V ∗ (cid:107) 1 .

Original: It can be seen that OS-VI has faster convergence with more accurate models and introduces acceleration compared to VI across different model errors.
Revised: It can be seen that OS-VI has faster convergence with more accurate models and introduces acceleration compared to VI across different model errors.

Original: Note that the convergence of OS-VI has been achieved despite the error in the model.
Revised: Note that the convergence of OS-VI has been achieved despite the error in the model.

Original: The dashed lines show how a fully model-based algorithm, which only uses ˆ P , would obtain a suboptimal solution.
Revised: The dashed lines show how a fully model-based algorithm, which only uses ˆ P , would obtain a suboptimal solution.

Original: We also compare OS-Dyna with Dyna and Q-Learning in the RL setting.
Revised: We also compare OS-Dyna with Dyna and Q-Learning in the RL setting.

Original: At each iteration t , the algorithms are given a sample ( X t , A t , R t , X (cid:48) t ) where X t , A t are selected uniformly at random.
Revised: At each iteration t , the algorithms are given a sample ( X t , A t , R t , X (cid:48) t ) where X t , A t are selected uniformly at random.

Original: For
Revised: For

Original: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.
Revised: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.

Original: If P MLE is the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ ) deﬁned in (6.1) as their models.
Revised: If P MLE is the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ ) deﬁned in (6.1) as their models.

Original: The learning rates are constant α for iterations t ≤ N and then decay in the form of α t = α/ ( t − N ) afterwards.
Revised: The learning rates are constant α for iterations t ≤ N and then decay in the form of α t = α/ ( t − N ) afterwards.

Original: We have ﬁne-tuned the learning rate schedule for each algorithm separately for the best results.
Revised: We have ﬁne-tuned the learning rate schedule for each algorithm separately for the best results.

Original: The right plot in Figure 2 shows the results for the RL setting.
Revised: The right plot in Figure 2 shows the results for the RL setting.

Original: We evaluate the expected return of the policy at iteration t in the initial state of the environment, i.e., V π t (0) .
Revised: We evaluate the expected return of the policy at iteration t in the initial state of the environment, i.e., V π t (0) .

Original: Again, OS-Dyna has converged to the optimal policy much faster than Q-Learning.
Revised: Again, OS-Dyna has converged to the optimal policy much faster than Q-Learning.

Original: Unlike OS-Dyna, Dyna has failed to ﬁnd the optimal policy in presence of model error.
Revised: Unlike OS-Dyna, Dyna has failed to ﬁnd the optimal policy in presence of model error.

Original: The results show that OS-Dyna can effectively converge faster than Q-Learning without introducing bias to the ﬁnal solution due to model error.
Revised: The results show that OS-Dyna can effectively converge faster than Q-Learning without introducing bias to the ﬁnal solution due to model error.

Original:  Conclusion
Revised:  Conclusion

Original: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can beneﬁt from an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to the true value function in terms of the number of queries to the true model P .
Revised: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can beneﬁt from an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to the true value function in terms of the number of queries to the true model P .

Original: With a small model error, its convergence rate is exponentially faster compared to well-known dynamical programming algorithms such as Value Iteration and Policy Iteration.
Revised: With a small model error, its convergence rate is exponentially faster compared to well-known dynamical programming algorithms such as Value Iteration and Policy Iteration.

Original: We also proposed OS-Dyna as a hybrid model-based/model-free algorithm that can bring in the beneﬁts of a model-based RL algorithm without converging to a biased solution, as Dyna or any other purely model-based RL algorithm does.
Revised: We also proposed OS-Dyna as a hybrid model-based/model-free algorithm that can bring in the beneﬁts of a model-based RL algorithm without converging to a biased solution, as Dyna or any other purely model-based RL algorithm does.

Original: This paper opens up several future directions.
Revised: This paper opens up several future directions.

Original: Empirically studying the algorithms on problems with large state spaces, for which a function approximator such as a DNN is required, is an obvious one.
Revised: Empirically studying the algorithms on problems with large state spaces, for which a function approximator such as a DNN is required, is an obvious one.

Original: This is postponed to a future work as our aim was to build the mathematical foundation and conducting experiments without worrying about challenges such as the optimization of a DNN.
Revised: This is postponed to a future work as our aim was to build the mathematical foundation and conducting experiments without worrying about challenges such as the optimization of a DNN.

Original: There are other algorithmic and theoretical directions to be pursued.
Revised: There are other algorithmic and theoretical directions to be pursued.

Original: One is exploring the space of splittings of I − γ P π .
Revised: One is exploring the space of splittings of I − γ P π .

Original: The other is whether we can design Operator Splitting variants of other DP algorithms such as Policy Iteration and Modiﬁed Policy Iteration, and study their convergence behaviour.
Revised: The other is whether we can design Operator Splitting variants of other DP algorithms such as Policy Iteration and Modiﬁed Policy Iteration, and study their convergence behaviour.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We would like to thank the anonymous reviewers for their comments that helped us to improve the clarity of the paper, as well as other members of the Adaptive Agents Lab who provided feedback on a draft of this paper.
Revised: We would like to thank the anonymous reviewers for their comments that helped us to improve the clarity of the paper, as well as other members of the Adaptive Agents Lab who provided feedback on a draft of this paper.

Original: AMF acknowledges the funding from the Canada CIFAR AI Chairs program, as well as the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program.
Revised: AMF acknowledges the funding from the Canada CIFAR AI Chairs program, as well as the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program.

Original: AR was partially supported by Borealis AI through the
Revised: AR was partially supported by Borealis AI through the

Original: Borealis AI Global Fellowship Award.
Revised: Borealis AI Global Fellowship Award.

Original: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
Revised: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.