Original: Operator Splitting Value Iteration
Revised: Operator Splitting Value Iteration

Original: Abstract
Revised: Abstract

Original: We introduce new planning and reinforcement learning algorithms for discounted
Revised: We introduce new planning and reinforcement learning algorithms for discounted

Original: MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function.
Revised: MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function.

Original: Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both
Revised: Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both

Original: Policy Evaluation and Control problems.
Revised: Policy Evaluation and Control problems.

Original: OS-VI achieves a much faster convergence rate when the model is accurate enough.
Revised: OS-VI achieves a much faster convergence rate when the model is accurate enough.

Original: We also introduce a sample-based version of the algorithm called OS-Dyna.
Revised: We also introduce a sample-based version of the algorithm called OS-Dyna.

Original: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of model approximation error.
Revised: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of model approximation error.

Original:  Introduction
Revised:  Introduction

Original: Consider a planning problem for a discounted MDP with dynamics P .
Revised: Consider a planning problem for a discounted MDP with dynamics P .

Original: Suppose that we have access to an approximate model ˆ P ≈ P as well.
Revised: Suppose that we have access to an approximate model ˆ P ≈ P as well.

Original: For example, P might be a high-fidelity, but slow, simulator, and ˆ P is a lower-fidelity, but fast, simulator.
Revised: For example, P might be a high-ﬁdelity, but slow, simulator, and ˆ P is a lower-ﬁdelity, but fast, simulator.

Original: Or in a different, but relevant, context of model-based reinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which we can only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaply acquired.
Revised: Or in the context of model-based reinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which we can only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaply acquired.

Original: Can we use this approximate model ˆ P to accelerate the computation of the value function of a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?
Revised: Can we use this approximate model ˆ P to accelerate the computation of the value function of a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?

Original: The Value Iteration (VI) algorithm, and its approximate variant, is a fundamental algorithm in Dynamic Programming that can find the  value of a policy or the optimal value function.
Revised: The Value Iteration (VI) algorithm and its approximate variant are fundamental algorithms in Dynamic Programming that can ﬁnd the (approximate) value of a policy or the optimal value function.

Original: It is also the backbone of many reinforcement learning (RL) algorithms such as TemporalDifference Learning [Sutton, 1988], Fitted Value Iteration
Revised: They are also the backbone of many reinforcement learning (RL) algorithms such as Temporal

Original:  [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári, 2008], and Deep Q Network [Mnih et al., 2015].
Revised: Difference Learning [Sutton, 1988], Fitted Value Iteration [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári, 2008], and Deep Q Network [Mnih et al., 2015].

Original: This algorithm, however, can be slow when the discount factor is close to 1 , as its convergence rate is O ( γ k ) .
Revised: Value Iteration, however, can be slow when the discount factor is close to 1 , as its convergence rate is O ( γ k ) .

Original: Moreover, even though we could use VI using ˆ P instead of P , effectively avoiding any need for expensive queries to P , the obtained value function would converge to a solution different from the value function of the original MDP.
Revised: Moreover, even though we could use VI using ˆ P instead of P to avoid expensive queries to P , the obtained value function would converge to a solution different from the value function of the original MDP.

Original: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that benefits from an approximate model ˆ P to potentially accelerate the convergence of the value function sequence to the value function with respect to (w.r.t.)
Revised: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that beneﬁts from an approximate model ˆ P to potentially accelerate the convergence of the value function sequence to the value function with respect to (w.r.t.)

Original: the true model P (Section 3).
Revised: the true model P (Section 3).

Original: This algorithm is for both
Revised: This algorithm is for both

Original: PE (Section 3.1) and Control (Section 3.2) problems.
Revised: PE (Section 3.1) and Control (Section 3.2) problems.

Original: The acceleration is not uniform though, and depends on how close ˆ P is to P (Section 4).
Revised: The acceleration is not uniform though, and depends on how close ˆ P is to P (Section 4).

Original: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra, which is used for iterative solution of large linear systems of equations [Varga, 2000, Saad, 2003, Golub and Van Loan, 2013].
Revised: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra, which is used to iteratively solve large linear systems of equations [Varga, 2000, Saad, 2003, Golub and Van Loan, 2013].

Original: With a proper choice of splitting, one may change the convergence  36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Revised: With a proper choice of splitting, one may change the convergence rate of 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: rate of linear system of equation solvers.
Revised: linear systems solvers.

Original: We show that the conventional VI for PE can be seen as a particular choice of splitting.
Revised: We show that the conventional VI for PE can be seen as a particular choice of splitting.

Original: This observation suggests that one may choose other forms of splitting  in order to change the convergence rate.
Revised: This observation suggests that one may choose other forms of splitting as well in order to change the convergence rate.

Original: It turns out that we can choose a splitting that benefits from having access to ˆ P (Section 2).
Revised: It turns out that we can choose a splitting that beneﬁts from having access to ˆ P (Section 2).

Original: The new splitting leads to OS-VI for PE.
Revised: The new splitting leads to OS-VI for PE.

Original: For the Control problem, the connection between solving linear systems of equations and VI is not as straightforward anymore, as the former is linear, while the latter is not, but we can still get inspired from the splitting approach to design OS-VI for Control.
Revised: For the Control problem, the connection between solving linear system of equations and VI is not as straightforward anymore, as the former is linear, while the latter is not, but we can still get inspired from the splitting approach to design OS-VI for Control.

Original: The key step of such an algorithm is a new policy improvement step.
Revised: The key step of such an algorithm is a new policy improvement step.

Original: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate model ˆ
Revised: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate model ˆ

Original: P is learned using data.
Revised: P is learned using data.

Original: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].
Revised: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].

Original: OS-Dyna is a hybrid of model-free and model-based algorithms.
Revised: OS-Dyna is a hybrid of model-free and model-based algorithms.

Original: It uses the learned model in its inner planning loop,  but uses samples from the true model P in order to correct the effect of errors in the model.
Revised: It uses the learned model in its inner planning loop, alike Dyna, but uses samples from the true model P in order to correct the effect of errors in the model.

Original: Existing MBRL algorithms would converge to an incorrect solution if the approximate model ˆ P does not converge to the true model P .
Revised: Existing MBRL algorithms would converge to an incorrect solution if the approximate model ˆ P does not converge to the true model P .

Original: This would be the case whenever model approximation error exists.
Revised: This would be the case whenever model approximation error exists.

Original: On the other hand, OS-Dyna can still converge to the correct value function even when ˆ P does not converge to P .
Revised: On the other hand, OS-Dyna can still converge to the correct value function even when ˆ P does not converge to P .

Original: As far as we know, this is the first model-based RL algorithm with such property.
Revised: As far as we know, this is the ﬁrst model-based RL algorithm with such property.

Original:  From value iteration to splitting-based linear system of equations solvers and back
Revised:  From value iteration to splitting-based linear system of equations solvers and back

Original: We briefly describe the VI algorithm and the splitting methods for solving linear systems of equations, and explain their connections.
Revised: We brieﬂy describe the VI algorithm and the splitting methods for solving linear system of equations, and explain their connections.

Original: We consider a discounted Markov Decision Process (MDP) ( X , A , R , P , γ )
Revised: We consider a discounted Markov Decision Process (MDP) ( X , A , R , P , γ )

Original: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].
Revised: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].

Original: We defer formal definitions to the supplementary material.
Revised: We defer formal deﬁnitions to the supplementary material.

Original: We only mention that for a policy π , we denote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π =
Revised: We only mention that for a policy π , we denote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π =

Original: V π ( R , P ) its state-value function.
Revised: V π ( R , P ) its state-value function.

Original: We also represent the optimal state-value function by V ∗ =
Revised: We also represent the optimal state-value function by V ∗ =

Original: V ∗ ( R , P ) and the optimal policy by π ∗ =
Revised: V ∗ ( R , P ) and the optimal policy by π ∗ =

Original: π ∗ ( R , P ) .
Revised: π ∗ ( R , P ) .

Original: The Bellman operator
Revised: The Bellman operator

Original: T π : B ( X ) → B ( X ) for policy π and the Bellman optimality operator T ∗ : B ( X ) → B ( X ) are( T
Revised: T π : B ( X ) → B ( X ) for policy π and the Bellman optimality operator T ∗ : B ( X ) → B ( X ) are( T

Original: [Equation]
Revised: [Equation]

Original: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ :
Revised: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ :

Original: V (cid:55)→ max π { r π + γ P π V } .
Revised: V (cid:55)→ max π { r π + γ P π V } .

Original: The greedy policy at state x ∈ X is
Revised: The greedy policy at state x ∈ X is

Original: [Equation]
Revised: [Equation]

Original: or more compactly, π g ( V ) ← argmax π T π V .
Revised: or more compactly, π g ( V ) ← argmax π T π V .

Original: We have T ∗ V
Revised: We have T ∗ V

Original: = T π g ( V ) V , that is, the effect of the
Revised: = T π g ( V ) V , that is, the effect of the

Original: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellman operator of the greedy policy w.r.t.
Revised: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellman operator of the greedy policy w.r.t.

Original: V to V .
Revised: V to V .

Original: 2.1 Value Iteration
Revised: 2.1 Value Iteration

Original: The value function V π and the optimal value function V ∗ are the fixed points of the operators T π and T ∗ , respectively, and satisfy the Bellman equation.
Revised: The value function V π and the optimal value function V ∗ are the ﬁxed points of the operators T π and T ∗ , respectively, and satisfy the Bellman equation.

Original: For the PE problem, this means that
Revised: For the PE problem, this means that

Original: [Equation]
Revised: [Equation]

Original: There are several ways to compute the value function of a policy π or the optimal value function V ∗ , including the iterative methods such as Value Iteration (VI) and Policy Iteration (PI) algorithms, or solving a linear systems of equations (for PE) or linear programming (for Control).
Revised: There are several ways to compute the value function of a policy π or the optimal value function V ∗ , including the iterative methods such as VI and Policy Iteration (PI) algorithms, or solving a linear system of equations (for PE) or linear programming (for Control).

Original: We focus on the VI algorithm in this work.
Revised: We focus on the VI algorithm in this work.

Original: VI repeatedly applies the Bellman operator to the most recent approximation of the value function:
Revised: VI repeatedly applies the Bellman operator to the most recent approximation of the value function:

Original: Given an initial value function V 0 , it generates a sequence ( V k ) k ≥ 0 as follows:
Revised: Given an initial value function V 0 , it generates a sequence ( V k ) k ≥ 0 as follows:

Original: [Equation]
Revised: [Equation]

Original: VI for Control can be written in an equivalent form: At iteration k , we first compute the greedy policy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .
Revised: VI for Control can be written in an equivalent form: At iteration k , we ﬁrst compute the greedy policy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .

Original: Therefore, the policy improvement step is obtained through finding a policy that is greedy w.r.t.
Revised: Therefore, the policy improvement step is obtained through ﬁnding a policy that is greedy w.r.t.

Original: the last value function
Revised: the last value function

Original: V k − 1 , that is, the best policy if we only look one step ahead.
Revised: V k − 1 , that is, the best policy if we only look one step ahead.

Original: This form will be conductive for our later developments.
Revised: This form will be conductive for our later developments.

Original: As the Bellman operator is a γ -contraction w.r.t.
Revised: As the Bellman operator is a γ -contraction w.r.t.

Original: the supremum norm, the convergence rate of V k to V π (or V ∗ ) would be O ( γ k ) .
Revised: the supremum norm, the convergence rate of V k to V π (or V ∗ ) would be O ( γ k ) .

Original: This rate can be slow when γ isvery close to 1 .
Revised: This rate can be slow when γ is close to 1 .

Original: 2.2 Matrix splitting for solving linear systems of equations
Revised: 2.2 Matrix splitting for solving linear system of equations

Original: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear system of equations (2.1).
Revised: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear system of equations (2.2).

Original: Consider the linear systems of equations Az = b , with A ∈ R d × d and z, b ∈ R d .
Revised: Consider the linear system Az = b , with A ∈ R d × d and z, b ∈ R d .

Original: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally, A , M , and N can be linear operators).
Revised: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally, A , M , and N can be linear operators).

Original: Therefore, z satisfies Mz =
Revised: Therefore, z satisﬁes Mz =

Original: Nz + b .
Revised: Nz + b .

Original: The splitting-based iterative approach defines the new approximation z k given the current z k − 1 by solving
Revised: The splitting-based iterative approach deﬁnes the new approximation z k given the current z k − 1 by solving

Original: [Equation]
Revised: [Equation]

Original: or equivalently
Revised: or equivalently,

Original: [Equation]
Revised: [Equation]

Original: To analyze the convergence of this iterative method, consider the error e k ≜ z k − z .
Revised: To analyze the convergence of this iterative method, consider the error e k (cid:44) z k − z .

Original: As Mz =
Revised: As z =

Revised: M − 1 ( Nz + b ) , we have e k

Original:  Nz + b  ,  the dynamics of the error is
Revised: = M − 1 ( Nz k − 1 + b ) − M − 1 ( Nz + b ) = M − 1 N ( z k − 1 − z ) , so the dynamics of the error is

Original: [Equation]
Revised: [Equation]

Original: Let G ≜ M − 1 N .
Revised: Let G (cid:44) M − 1 N .

Original: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as
Revised: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as

Original: [Equation]
Revised: [Equation]

Original: The errors are (norm-) convergent if ∥ G ∥ =
Revised: The errors are (norm-) convergent if (cid:107) G (cid:107) =

Original: ∥ M − 1 N ∥ < 1 , for some choice of norm.
Revised: (cid:107) M − 1 N (cid:107) < 1 , for some choice of norm.

Original: More generally, the necessary and sufficient condition for convergence is that the spectral radius ρ ( G ) , the maximum absolute value of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or
Revised: More generally, the necessary and sufﬁcient condition for convergence is that the spectral radius ρ ( G ) , the maximum absolute value of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or

Original: Theorem 11.2.1 of Golub and Van Loan [2013].
Revised: Theorem 11.2.1 of Golub and Van Loan [2013].

Original: The convergence is faster if the spectral radius (or norm) is closer to zero.
Revised: The convergence is faster if the spectral radius (or norm) is closer to zero.

Original: The success of this iterative procedure depends on how we choose M and N such that the norm (or spectral radius) is as small as possible.
Revised: The success of this iterative procedure depends on how we choose M and N such that the norm (or spectral radius) is as small as possible.

Original: Also we want to choose an M such that solving Mz k =
Revised: Also we want to choose an M such that solving Mz k =

Original: Nz k − 1 + b is not very expensive.
Revised: Nz k − 1 + b is not very expensive.

Original: For example, if M is an identity matrix I , we get that N
Revised: For example, if M is an identity matrix I , we get that N

Original: =
Revised: =

Original: I − A , and the iteration becomes z k
Revised: I − A , and the iteration becomes z k

Original: = ( I − A )
Revised: = ( I − A )

Original: z k − 1 + b .
Revised: z k − 1 + b .

Original: This iteration is convergent if ρ ( I − A ) < 1 .
Revised: This iteration is convergent if ρ ( I − A ) < 1 .

Original: Other commonly used choices lead to the Jacobi and Gauss-Seidel methods that are described in the supplementary material.
Revised: Other commonly used choices lead to the Jacobi and Gauss-Seidel methods that are described in the supplementary material.

Original: We are now ready to make the connection between splitting-based iterative methods and VI for PE.
Revised: We are now ready to make the connection between splitting-based iterative methods and VI for PE.

Original: If we choose A = I − γ P π , we see that equation
Revised: If we choose A = I − γ P π , we see that equation

Original: AV π = r π is indeed the Bellman equation for policy π (2.1).
Revised: AV π = r π is indeed the Bellman equation for policy π (2.2).

Original: The VI for PE, which is V k = γ P π V k − 1 + r π =
Revised: The VI for PE, which is V k = γ P π V k − 1 + r π =

Original: ( I − A )
Revised: ( I − A )

Original: V k − 1 + r π , corresponds to the choice of M
Revised: V k − 1 + r π , corresponds to the choice of M

Original: =
Revised: =

Original: I and N = γ P π .
Revised: I and N = γ P π .

Original: This brings up the question of whether it is possible to split A to other choices of M and N so that the resulting VI-like procedure has better convergence properties.
Revised: This brings up the question of whether it is possible to split A differently so that the resulting VI-like procedure has better convergence properties.

Original: We  suggest a particular choice in the next section.
Revised: We next suggest a particular choice.

Original:  Operator splitting value iteration algorithm
Revised:  Operator splitting value iteration algorithm

Original: This section introduces the Operator Splitting Value Iteration (OS-VI) algorithm.
Revised: We introduce the Operator Splitting Value Iteration (OS-VI) algorithm.

Original: We start from the PE problem and introduce the Control version based on that.
Revised: We start from the PE problem and develop the Control version based on that.

Revised: We also present a visualization of how OS-VI works.

Original: 3.1 OS-VI for policy evaluation
Revised: 3.1 OS-VI for policy evaluation

Original: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as
Revised: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as

Original: [Equation]
Revised: [Equation]

Original: Following the recipe of (2.3), the OS-VI algorithm for PE would be
Revised: Following the recipe of (2.4), the OS-VI algorithm for PE would be

Original: [Equation]
Revised: [Equation]

Original: starting from an initial V 0 .
Revised: starting from an initial V 0 .

Original: 3 To gain more intuition and prepare for further developments, we define a few notations.
Revised: 3 To gain more intuition and prepare for further developments, we deﬁne a few notations.

Original: We define the Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga (1928 – 2022) who has made significant contributions to matrix analysis, as the mapping between the space of all bounded functions over X to the same space as
Revised: We deﬁne the Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga (1928 – 2022) who has made signiﬁcant contributions to matrix analysis, as the mapping between the space of all bounded functions over X to the same space as

Original: [Equation]
Revised: [Equation]

Original: Observe that (3.1) can be compactly written as
Revised: Observe that (3.1) can be compactly written as

Original: [Equation]
Revised: [Equation]

Original: It is not difficult to see that S
Revised: It is not difﬁcult to see that S

Original: π V π = V π , i.e., the value function of a policy π is a fixed-point of the
Revised: π V π = V π , i.e., the value function of a policy π is a ﬁxed-point of the

Original: Varga operator S π .
Revised: Varga operator S π .

Original: This and other properties of the Varga operator are  in the supplementary material.
Revised: This and other properties of the Varga operator are shown in the supplementary material.

Original: Given any value function V , define an auxiliary reward function ¯ r V : X ×
Revised: Given any value function V , deﬁne an auxiliary reward function ¯ r V : X ×

Original: A → R as
Revised: A → R as

Original: [Equation]
Revised: [Equation]

Original: Similar tothe notation for r π , we define ¯ r πV :
Revised: Similar to r π , we deﬁne the notation ¯ r πV :

Original: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministic policy π (and similarly for a stochastic policy).
Revised: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministic policy π (and similarly for a stochastic policy).

Original: With this notation, 
Revised: With this notation, the effect of applying S π to V is

Original: [Equation]
Revised: [Equation]

Original: This is the value function of following π in an MDP with dynamics ˆ P and reward ¯ r V .
Revised: This is the value function of following π in an MDP with dynamics ˆ P and reward ¯ r V .

Original: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximate dynamics, and a reward function that consists of the original reward r and the correction term γ
Revised: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximate dynamics, and a reward function that consists of the original reward r and the correction term γ

Original: ( P − ˆ P ) V k − 1 .
Revised: ( P − ˆ P ) V k − 1 .

Original: The computation of this value function is a standard PE problem with the approximate model.
Revised: The computation of this value function is a standard PE problem with the approximate model.

Original: For instance, we may use another VI (PE) with dynamics ˆ P to solve it: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P
Revised: For instance, we may use another VI (PE) with dynamics ˆ P to solve it: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P

Original: π
Revised: π

Original: U i − 1 .
Revised: U i − 1 .

Original: This converges to S π V
Revised: This converges to S π V

Original: =
Revised: =

Original: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .
Revised: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .

Original: Note that aside from the computation of ¯ r πV , which requires querying P in order to compute the P π V term, this iterative process only uses the approximate model ˆ P , which is assumed to be cheap to access.
Revised: Note that aside from the computation of ¯ r πV , which requires querying P in order to compute the P π V term, this iterative process only uses the approximate model ˆ P , which is assumed to be cheap to access.

Original: What is the benefit of this OS-VI procedure?
Revised: What is the beneﬁt of this OS-VI procedure?

Original: If the approximate model ˆ P is close to the true dynamics
Revised: If the approximate model ˆ P is close to the true dynamics

Original: P , this leads to a faster convergence of V k to V π , as shall be quantified soon.
Revised: P , this leads to a faster convergence of V k to V π , as shall be quantiﬁed soon.

Original: The faster convergence is in terms of the number of queries to P , which is assumed to be expensive.
Revised: The faster convergence is in terms of the number of queries to P , which is assumed to be expensive.

Original: To see this, consider the hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happens to perfectly match the reality.
Revised: To see this, consider the hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happens to perfectly match the reality.

Original: Then, S π V =
Revised: Then, S π V =

Original: ( I − γ P π ) − 1 ( r π + 0 V ) =
Revised: ( I − γ P π ) − 1 ( r π + 0 V ) =

Original: V π , and the value function for the original MDP is obtained in one iteration of OS-VI.
Revised: V π , and the value function for the original MDP is obtained in one iteration of OS-VI.

Original: Of course, we often can only hope for ˆ
Revised: Of course, we often can only hope for ˆ

Original: P ≈ P .
Revised: P ≈ P .

Original: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in more details, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ P is only a close approximation of P .
Revised: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in more details, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ P is only a close approximation of P .

Original: 3.2 OS-VI for control
Revised: 3.2 OS-VI for control

Original: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ← π g ( V k − 1 ) =
Revised: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ← π g ( V k − 1 ) =

Original: argmax π T π V k − 1 in its policy improvement step, and  uses one step of the Bellman operator w.r.t.
Revised: argmax π T π V k − 1 in its policy improvement step, and then uses one step of the Bellman operator w.r.t.

Original: the obtained policy π k to compute the new estimate of the value function V k ← T π k V k − 1 , as described after (2.2).
Revised: the obtained policy π k to compute the new estimate of the value function V k ← T π k V k − 1 , as described after (2.3).

Original: The OS-VI algorithm for Control follows a similar structure with the difference that 1) the improved policy is the optimizer of the Varga operator, and not the Bellman operator, and 2) the new value function is obtained by applying the Varga operator of the newly obtained policy.
Revised: The OS-VI algorithm for Control follows a similar structure with the difference that 1) the improved policy is the optimizer of the Varga operator, and not the Bellman operator, and 2) the new value function is obtained by applying the Varga operator of the newly obtained policy.

Original: To be concrete, given a value function V , define the S -improved policy
Revised: To be concrete, given a value function V , deﬁne the S -improved policy

Original: [Equation]
Revised: [Equation]

Original: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .
Revised: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .

Original: We also define the Varga optimal operator S ∗ : B ( X ) → B ( X ) as
Revised: We also deﬁne the Varga optimality operator S ∗ : B ( X ) → B ( X ) as

Original: [Equation]
Revised: [Equation]

Original: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.
Revised: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.

Original: V applied to a value function V (compare it with T ∗ V =
Revised: V applied to a value function V (compare it with T ∗ V =

Original: T π g ( V ) V ).
Revised: T π g ( V ) V ).

Original: The OS-VI (Control) is then simply
Revised: The OS-VI (Control) is then simply

Original: [Equation]
Revised: [Equation]

Original: which in its expanded form, consists of the following two steps:
Revised: which in its expanded form, consists of the following two steps:

Original: [Equation]
Revised: [Equation]

Original: Comparing the S -improved policy [Equation] 4) used in the policy improvement step (3.6) of OS-VI with the conventional greedy policy  is insightful.
Revised: Comparing the S -improved policy (3.4) used in the policy improvement step (3.6) of OS-VI with the conventional greedy policy (2.1) is insightful.

Original: The greedy policy is argmax π T π V .
Revised: The greedy policy is argmax π T π V .

Original: Expanding T π V , we see that the greedy policy is the maximizer of r π + γ P π V .
Revised: Expanding T π V , we see that the greedy policy is the maximizer of r π + γ P π V .

Original: The function r π + γ P π V isonly a single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy, is in general different from the optimal policy, which maximizes V π .
Revised: The function r π + γ P π V is a single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy, is in general different from the optimal policy, which maximizes V π .

Original: On the other hand, the S -improved policy π V ( V ) solves a full MDP with an approximate model ˆ P and the reward function that has both the original reward r and the correction term γ
Revised: On the other hand, the S -improved policy π V ( V ) solves a full MDP with an approximate model ˆ P and a reward function that has both the original reward r and the correction term γ

Original: ( P − ˆ P ) V .
Revised: ( P − ˆ P ) V .

Original: In the special case that ˆ P = P , the correction term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.
Revised: In the special case that ˆ P = P , the correction term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.

Original: As often ˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.
Revised: As often ˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.

Original: The partial policy evaluation step (3.7) updates the value function to a value that is closer to the optimal value function.
Revised: The partial policy evaluation step (3.7) updates the value function to a value that is closer to the optimal value function.

Original: Remark.
Revised: Remark.

Original: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context of dynamic programming is not completely novel to this work.
Revised: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context of dynamic programming is not completely novel to this work.

Original: Kushner and Kleinman [1971] is one of the earliest paper that mentions the Jacobi and Gauss-Seidel procedures for computing the value function.
Revised: Kushner and Kleinman [1971] is one of the earliest paper that mentions the Jacobi and Gauss-Seidel procedures for computing the value function.

Original: Porteus [1975] proposes several transformations to the reward and probability transition matrix with the goal of improving the computational cost of solving the transformed MDP.
Revised: Porteus [1975] proposes several transformations to the reward and probability transition matrix with the goal of improving the computational cost of solving the transformed MDP.

Original: One of the transformations, called pre-inverse transform , has some similarities with the operator splitting of this work.
Revised: One of the transformations, called pre-inverse transform , has some similarities with the operator splitting of this work.

Original: The end result, however, is different.
Revised: The end result, however, is different.

Original: Bacon and Precup [2016] provide a matrix splitting perspective on planning with options.
Revised: Bacon and Precup [2016] offer a matrix splitting perspective on planning with options.

Original: The connection between multi-step models and matrix splitting is developed in Chapter 4 of Bacon [2018].
Revised: The connection between multi-step models and matrix splitting is developed in Chapter 4 of Bacon [2018].

Original: Refer to the supplementary material for more discussion.
Revised: Refer to the supplementary material for more discussion.

Original: 3.3 Visualizing how OS-VI works
Revised: 3.3 Visualizing how OS-VI works

Original: We visualize the value function trajectories of several algorithms, including OS-VI, on a 2-state MDP in order to provide some intuition on how OS-VI works.
Revised: To present some intuition on how OS-VI works, we visualize the value function trajectories of several algorithms, including OS-VI, on a 2-state MDP, in Figure 1.

Original: We consider the policy evaluation for the dynamics P π = [ 0 . 9 0 .
Revised: We consider the policy evaluation for the dynamics P π = [ 0 . 9 0 .

Original: 9 ] with the reward
Revised: 9 ] with the reward

Original: r π = (cid:0) 1 − 0 .
Revised: r π = (cid:0) 1 − 0 .

Original: 5 (cid:1) and γ = 0 .
Revised: 5 (cid:1) and γ = 0 .

Original: We consider two approximate models: a relatively accurateone ˆ P π accurate =
Revised: We consider two approximate models: a relatively accurate ˆ P π accurate =

Original: 95 ] , and an inaccurateone ˆ P π inaccurate =
Revised: 95 ] , and an inaccurate ˆ P π inaccurate =

Original: The results are shown in Figure 1. The first algorithm is the conventional VI (PE), which repeatedly applies the Bellman operator according to the true model P π to the most recent approximation of the value function.
Revised: In addition to OS-VI (PE), the ﬁrst algorithm is the conventional VI (PE), which repeatedly applies the Bellman operator according to the true model P π to the most recent approximation of the value function.

Original: We use T π P to refer to its Bellman operator and to label the corresponding trajectory in the value space.
Revised: We use T π P to refer to its Bellman operator and to label the corresponding trajectory in the value space.

Original: This algorithm converges to the true value function V π .
Revised: This algorithm converges to the true value function V π .

Original: The second algorithm is VI (PE) that uses ˆ P π as the model.
Revised: The second algorithm is VI (PE) that uses ˆ P π as the model.

Original: This procedure is the basis of the Dyna architecture.
Revised: This procedure is the basis of the Dyna architecture.

Original: We use T π ˆ
Revised: We use T π ˆ

Original: P to refer to its Bellman operator and to label the corresponding trajectory in the value space.
Revised: P to refer to its Bellman operator and to label the corresponding trajectory in the value space.

Original: Due to the error of T π ˆ P compared to T π P , the algorithm converges to a wrong value function ˆ V π , as both figures show.
Revised: Due to the error of T π ˆ P compared to T π P , the algorithm converges to a wrong value function ˆ V π , as both ﬁgures show.

Original: We observe that even when the model is relatively accurate as in Figure 1a, the converged value function is quite wrong.
Revised: We observe that even when the model is relatively accurate as in Figure 1a, the converged value function is quite wrong.

Original: This illustrates one limitation of the conventional model-based RL algorithms where an inaccurate model may lead to significantly inaccurate estimate of the value function.
Revised: This illustrates one limitation of the conventional model-based RL algorithms where an inaccurate model may lead to signiﬁcantly inaccurate estimate of the value function.

Original: The OS-VI algorithm repeatedly applies the Varga operator S π to the most recent approximation of the value function.

Original: As discussed earlier, each computation of S π V k − 1 corresponds to solving an

Original: V (0) −−−
Revised: V (0) −−−

Original: V ( 1 )
Revised: V ( 1 )

Original: Vˆ V π
Revised: Vˆ V π

Original: V π
Revised: V π

Original: T π P T π ˆ
Revised: T π P T π ˆ

Original: P ¯ T πk
Revised: P ¯ T πk

Original: S π (a) More accurate model ˆ P
Revised: S π (a) More accurate model ˆ P

Original: accurate
Revised: accurate

Original: V (0) −−−
Revised: V (0) −−−

Original: V ( 1 )
Revised: V ( 1 )

Original: Vˆ V π
Revised: Vˆ V π

Original: V π
Revised: V π

Original: T π P T π ˆ
Revised: T π P T π ˆ

Original: P ¯ T πk
Revised: P ¯ T πk

Original: S π 
Revised: S π (

Original: [Figure]
Revised: [Figure]

Revised: The OS-VI algorithm repeatedly applies the Varga operator S π to the most recent approximation of the value function.

Original:  auxiliary MDPs ( X , A , ¯ r V k − 1 , ˆ P , γ ) .
Revised: As discussed earlier, each computation of S π V k − 1 corresponds to solving an auxiliary MDPs ( X , A , ¯ r V k − 1 , ˆ P , γ ) .

Original: We denote the Bellman operator of this auxiliary MDP by ¯ T π k .
Revised: We denote the Bellman operator of this auxiliary MDP by ¯ T πk .

Original: The figures show the trajectory generated by the iterative application of S π on the most recent value function as well as the trajectory for solving each auxiliary MDP, indicated by ¯ T πk .
Revised: The ﬁgures show the trajectory generated by the iterative application of S π on the most recent value function as well as the trajectory for solving each auxiliary MDP, indicated by ¯ T πk .

Original: We observe that the OS-VI algorithm converges to the correct value function despite using an incorrect model.
Revised: We observe that the OS-VI algorithm converges to the correct value function despite using an incorrect model.

Original: When the model is more accurate, very few iteration of OS-VI gets a value close to V π (two iterations in
Revised: When the model is more accurate, very few iteration of OS-VI gets a value close to V π (two iterations in

Original: Figure 1a)
Revised: Figure 1a)

Original: ; when the model is less accurate, a few more iterations are needed.
Revised: ; when the model is less accurate, a few more iterations are needed.

Original: Compared to VI, at least in these examples, the total number of iterations of OS-VI is significantly fewer.
Revised: Compared to VI, at least in these examples, the total number of iterations of OS-VI is signiﬁcantly smaller.

Original: When the initial value function is V 0 = 0 , the result of the first iteration of OS-VI is the same value function computed by the VI with the wrong model ˆ P π .
Revised: When the initial value function is V 0 = 0 , the result of the ﬁrst iteration of OS-VI is the same value function computed by the VI with the wrong model ˆ P π .

Original: This is because V 1 ← S π V 

Revised: This is because V 1 ← S π V

Original: = ( I − γ ˆ P π ) − 1 ¯ r πV 0 and ¯ r πV 0 =
Revised: = ( I − γ ˆ P π ) − 1 ¯ r πV 0 and ¯ r πV 0 =

Original: r π , so V 1 =
Revised: r π , so V 1 =

Original: ( I − γ ˆ P π ) − 1 r π . This is the same solution as the value function obtained using the approximate model ˆ P .
Revised: ( I − γ ˆ P π ) − 1 r π , the same solution as the value function obtained using the approximate model ˆ P .

Original: In these figures, this shows itself by the overlapping of the red arrows followed by T π ˆ P and the first segment of the orange arrows, which are generated by the repeated application of ¯ T π 1 .
Revised: In these ﬁgures, this shows itself by the overlapping of the red arrows followed by T π ˆ P and the ﬁrst segment of the orange arrows, which are generated by the repeated application of ¯ T π 1 .

Original: In further iterations of OS-VI, the auxiliary MDPs change and the path followed by ¯ T πk ( k ≥ 2 ) deviates from the solution of the VI with the wrong model.
Revised: In further iterations of OS-VI, the auxiliary MDPs change and the path followed by ¯ T πk ( k ≥ 2 ) deviates from the solution of the VI with the wrong model.

Original:  Convergence analysis of operator splitting value iteration
Revised:  Convergence analysis of operator splitting value iteration

Original: In this section, we present the convergence analysis of OS-VI.
Revised: In this section, we present the convergence analysis of OS-VI.

Original: Our results show that OS-VI has an
Revised: Our results show that OS-VI has an

Original: O ( γ ′ k ) convergence rate for an effective discount factor γ ′ that depends on the error between ˆ P and
Revised: O ( γ (cid:48) k ) convergence rate for an effective discount factor γ (cid:48) that depends on the error between ˆ P and

Original: P .
Revised: P .

Original: For small enough error, γ ′ < γ and OS-VI has a faster convergence rate compared to the classic
Revised: For small enough error, γ (cid:48) < γ and OS-VI has a faster convergence rate compared to the classic

Original: VI, Policy Iteration (PI), and Modified Policy Iteration (MPI), which all have O ( γ k ) behaviour.
Revised: VI, Policy Iteration (PI), and Modiﬁed Policy Iteration (MPI), which all have O ( γ k ) behaviour.

Original: We provide results for both the L ∞ and L p norms.
Revised: We provide results for both the L ∞ and L p norms.

Original: 4.1 Convergence of OS-VI for policy evaluation
Revised: 4.1 Convergence of OS-VI for policy evaluation

Original: We study the convergence behaviour of OS-VI (PE) in presence of error in value updates.
Revised: We study the convergence behaviour of OS-VI (PE) in presence of error in value updates.

Original: Specifically, we consider that in each iteration k , the update (3.2) has an error, i.e.,
Revised: Speciﬁcally, we consider that at each iteration k , the update (3.2) has an error, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The error ϵ value k encompasses various sources of errors.
Revised: The error (cid:15) value k encompasses various sources of errors that might occur in solving the auxiliary

Revised: MDP ( X , A , ¯ r V k − 1 , ˆ P , γ ) .

Original: One source is the function approximation error due to using a function approximator to represent V k , which is often required in large state-action spaces.
Revised: One source is the function approximation error due to using a function approximator to represent V k , which is often required in large state spaces.

Original: Another is the estimation (i.e., statistical) error caused due to having a finite number of samples, instead of direct access to P , in the RL setting.
Revised: Another is the estimation (i.e., statistical) error caused due to having a ﬁnite number of samples, instead of direct access to P , in the RL setting.

Original: Refer toGyörfi et al. [2002], Steinwart and Christmann [2008] for the discussion of function approximation and estimation errors in the supervised learning context and to Munos and Szepesvári [2008], Antos et al.
Revised: Refer to Munos and Szepesvári [2008], Antos et al.

Original: [2008], Farahmand et al.
Revised: [2008], Farahmand et al.

Original: [2016], Chen and Jiang [2019], Fan et al.
Revised: Chen and Jiang [2019], Fan et al.

Original: [2019] for their analysis in the RL context.
Revised: [2019] for the discussion of function approximation and estimation errors in the RL context.

Original: In this work, we do not analyze how the number of samples, the function approximator, etc.
Revised: In this work, we do not analyze how the number of samples, the function approximator, etc.

Original: affect the errors ϵ value k .
Revised: affect the errors (cid:15) value k .

Original: The result of this section provides an error propagation result similar to Munos [2007] (approximate 
Revised: We offer error propagation results similar to Munos [2007] (approximate VI), Munos [2003] (approximate PI), and Scherrer et al. [2015] (approximate modiﬁed

Original: VI), Munos
Revised: PI) for approximate OS-VI.

Original: To study  convergence behaviour of OS-VI (PE), let G π
Revised: To study the convergence behaviour of OS-VI (PE), let G π

Original: = ( I − γ ˆ P π ) − 1 γ
Revised: = ( I − γ ˆ P π ) − 1 γ

Original: ( P π − ˆ P π ) .
Revised: ( P π − ˆ P π ) .

Original: We use the fact that S
Revised: We use the fact that S

Original: π V π = V π
Revised: π V π = V π

Original: and write
Revised: and write

Original: [Equation]
Revised: [Equation]

Original: Now, we have that
Revised: Now, we have that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that for any square matrix F with a matrix norm ∥ F ∥ p < 1 , it holds that ∥ ( I − F ) − 1 ∥ p ≤ [Equation]
Revised: where we used the fact that for any square matrix F with a matrix norm (cid:107) F (cid:107) p < 1 , it holds that (cid:107) ( I − F ) − 1 (cid:107) p ≤ 1 1 −(cid:107) F (cid:107) p

Original: (e.g., Lemma 2.3.3 of Golub and Van Loan 2013), and that the supremum norm of a stochastic matrix ˆ P π is 1 .
Revised: (see Lemma 2.3.3 of Golub and Van Loan 2013), and that the supremum norm of a stochastic matrix ˆ P π is 1 .

Original: Assuming that ∥ ϵ value k ∥ ∞ ≤ ϵ value for all k ≥ 1 and defining effective discount factor γ ′ =
Revised: Assuming that (cid:107) (cid:15) value k (cid:107) ∞ ≤ (cid:15) value for all k ≥ 1 and deﬁning effective discount factor γ (cid:48)

Original:  γ 
Revised: = γ 1 − γ (cid:107)P π − ˆ P π (cid:107) ∞ , the upper bounds (4.2) and (4.3) lead to (cid:107) V π − V k (cid:107) ∞ [Equation] (cid:48)

Original: [Equation]
Revised: [Equation]

Original: − ˆ P π ∥ ∞ , the upper bounds (4.3) and (4.2) lead to
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: (cid:48)

Revised: [Equation]

Revised: (cid:48) (cid:15) value .

Original: A few remarks are in order.
Revised: A few remarks are in order.

Original: First, whenever γ ′ < γ , this is guaranteed to be faster than the convergence rate of the conventional VI, which is O ( γ k ) .
Revised: First, whenever γ (cid:48) < γ , this is guaranteed to be faster than the convergence rate of the conventional VI, which is O ( γ k ) .

Original: This is the case if ∥P π − ˆ P π ∥ ∞ < 1 − γ .
Revised: This happens if (cid:107)P π − ˆ P π (cid:107) ∞ < 1 − γ .

Original: If the model is very accurate, we obtain much faster rate than VI’s.
Revised: If the model is very accurate, we obtain much faster rate than VI’s.

Original: Since each iteration k corresponds to a query to the true model P , a faster rate entails that the algorithm requires less number of queries to the expensive model to reach the same level of accuracy.
Revised: Since each iteration k corresponds to a query to the true model P , a faster rate entails that the algorithm requires fewer number of queries to the expensive model to reach the same level of accuracy.

Original: Second, although the model error ∥P π − ˆ P π ∥ ∞ is a reasonable choice to measure the distances between distributions (it is the maximum of the Total Variation distance between P π ( ·| x ) and ˆ P π ( ·| x ) over x , which itself can be upper bounded by their KL divergence; see the supplementary material), it is somewhat conservative as it takes the supremum over the state space.
Revised: Second, although the model error (cid:107)P π − ˆ P π (cid:107) ∞ is a reasonable choice to measure the distances between distributions (it is the maximum of the Total Variation distance between P π ( ·| x ) and ˆ P π ( ·| x ) over x , which itself can be upper bounded by their KL divergence; see the supplementary material), it is somewhat conservative as it takes the supremum over the state space.

Original: Likewise, requiring ∥ ϵ value k ∥ ∞ to be small is also conservative, as approximating S π V k − 1 using a function approximator given samples (RL setting) often leads to an L p -norm type of guarantee.
Revised: Likewise, requiring (cid:107) (cid:15) value k (cid:107) ∞ to be small is also conservative, as approximating S π V k − 1 using a function approximator given samples (RL setting) often leads to an L p -norm type of guarantee.

Original: We now provide a different analysis to address these issues.
Revised: We now provide a different analysis to address these issues.

Original: To present the L p -norm result, we need to define some notations.
Revised: To present the L p -norm result, we need to deﬁne some notations.

Original: First, we define the conditional discounted future-state distribution of policy π under ˆ P as the following probability distribution:
Revised: First, we deﬁne the conditional discounted future-state distribution of policy π under ˆ P as the following probability distribution:

Original: Given a measurable set B , we have ˆ η π ( B | x ) = 
Revised: Given a measurable set B , we have ˆ η π [Equation] B | x ) = (1 − γ )

Original: [Equation]
Revised: [Equation]

Revised: ∞ t =

Original: [Equation]
Revised: [Equation]

Revised: · P

Original: [Equation]
Revised: [Equation]

Original:  ∈ B | X 

Revised: X t ∈ B | X

Original: = x, π, ˆ P (cid:17) , where the chain ( X t ) t ≥ 0 starts from state x and evolves by following policy π under transitions ˆ P .
Revised: = x, π, ˆ P (cid:17) , where the chain ( X t ) t ≥ 0 starts from state x and evolves by following policy π under transitions ˆ P .

Original: For an arbitrary distribution ρ over the state space, we define theapproximate discounted future-state distribution concentration coefficient as[Equation]
Revised: For an arbitrary distribution ρ over the state space, we deﬁne the discounted future-state distribution concentration coefﬁcient of the approximate model as

Original: [Equation]
Revised: [Equation]

Original: Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.
Revised: Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.

Original: the distribution ρ , and it is assumed that for any x ∈ X , ˆ η π ( ·| x ) ≪ ρ , i.e., ˆ η π ( ·| x ) is absolutely continuous w.r.t.
Revised: the distribution ρ . It is assumed that for any x ∈ X , ˆ η π ( ·| x ) (cid:28) ρ , i.e., ˆ η π ( ·| x ) is absolutely continuous w.r.t.

Original: ρ (otherwise, the coefficient would be set to infinity).
Revised: ρ (otherwise, the coefﬁcient would be set to inﬁnity).

Original: This coefficient measures how concentrated the distribution ˆ η π ( ·| x ) is compared to ρ .
Revised: This coefﬁcient measures how concentrated the distribution ˆ η π ( ·| x ) is compared to ρ .

Original: This is weighted according to theinitial state distribution ρ .
Revised: This is weighted according to the state distribution ρ .

Original: Similar concentrability coefficients, but not exactly this one, have appeared in the error propagation results [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al., 2015].
Revised: Similar concentrability coefﬁcients, but not exactly this one, have appeared in the error propagation results [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al., 2015].

Original: Finally, we define the weighted χ 2 -divergence of ˆ P π and P π as 
Revised: Finally, we deﬁne the weighted χ 2 -divergence of ˆ P π and P π as χ [Equation] P

Original: [Equation]
Revised: [Equation]

Original: This notion of model error is less strict in requiring accurate approximation P in all states.
Revised: This notion of model error is less strict in requiring accurate approximation P in all states.

Original: Usually only a subset of  state space is important or even reachable in a problem.
Revised: Usually only a subset of the state space is important or even reachable in a problem.

Original: The above model error canonly focus on specific areas of the state space through the choice of distribution ρ .
Revised: The above model error can focus on only speciﬁc areas of the state space through the choice of distribution ρ .

Original: We are now ready to to provide the main theorem for the approximate OS-VI (PE).
Revised: We are now ready to present the main theorem for the approximate OS-VI (PE).

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider the approximate OS-VI algorithm for PE (4.1) .
Revised: Consider the approximate OS-VI algorithm for PE (4.1) .

Original: Let ∥·∥ ⋆ be either the supremum norm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over the state space ( ⋆ = 4 , ρ ).
Revised: Let (cid:107)·(cid:107) (cid:63) be either the supremum norm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over the state space ( (cid:63) = 4 , ρ ).

Original: Assume that ∥ ϵ value k ∥ ⋆ ≤ ϵ value for all k ≥ 1 .
Revised: Assume that (cid:107) (cid:15) value k (cid:107) (cid:63) ≤ (cid:15) value for all k ≥ 1 .

Original: Furthermore, define the effective discount factor as
Revised: Furthermore, deﬁne the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: For any k ≥ 0 , we have
Revised: For any k ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: 4.2 Convergence of OS-VI for control
Revised: 4.2 Convergence of OS-VI for control

Original: We turn to analyzing OS-VI for Control.
Revised: We turn to analyzing OS-VI for Control.

Original: We consider two types of errors: The first is the error between the computed value function and the true optimal value function of the auxiliary MDP, i.e.,
Revised: We consider two types of errors: The ﬁrst is the error between the computed value function and the true optimal value function of the auxiliary MDP, i.e.,

Original: V k − S ∗ V k − 1 .
Revised: V k − S ∗ V k − 1 .

Original: The second is the suboptimality of obtained policy compared to the optimal policy of the auxiliary MDP, i.e., S π k V k − 1 − S ∗ V k − 1 .
Revised: The second is the suboptimality of obtained policy compared to the optimal policy of the auxiliary MDP, i.e., S π k V k − 1 − S ∗ V k − 1 .

Original: Concretely, we have
Revised: Concretely, we have

Original: [Equation]
Revised: [Equation]

Original: We have the following result for  OS-VI (Control).
Revised: We have the following result for the approximate OS-VI (Control).

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .
Revised: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .

Original: Let ∥·∥ ⋆ be either the supremum norm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over the state space ( ⋆ = 4 , ρ ).
Revised: Let (cid:107)·(cid:107) (cid:63) be either the supremum norm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over the state space ( (cid:63) = 4 , ρ ).

Original: For any k ≥ 1 , let Π k
Revised: For any k ≥ 1 , let Π k

Original: =
Revised: =

Original: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 1 ≤ i < k } .
Revised: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 1 ≤ i < k } .

Original: Assume that ∥ ϵ value k 
Revised: Assume that (cid:107) (cid:15) value k [Equation]

Original: ∥ ⋆ ≤ ϵ value for all k ≥ 1 .
Revised: ≤ (cid:15) value for all k ≥ 1 .

Original: Furthermore, define the effective discount factor as
Revised: Furthermore, deﬁne the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: We can compare this result with the convergence result of VI.
Revised: We can compare this result with the convergence result of VI.

Original: For VI with the supremum norm, following the proof of Equation (2.2) by Munos [2007], we can show that ∥ V ∗ − V π k ∥ ∞ ≤2 γ k
Revised: For VI with the supremum norm, following the proof of Equation (2.2) by Munos [2007], we can show that (cid:107) V ∗ − V π k [Equation] ∞ ≤

Original: [Equation]
Revised: [Equation]

Original: , with ∥ V i − T ∗ V i − 1 ∥ ∞ ≤ ε for all i < k (similar result for the L p -norm also holds, see Theorem 5.2 in Munos 2007).
Revised: , with (cid:107) V i − T ∗ V i − 1 (cid:107) ∞ ≤ (cid:15) value for all i < k (similar result for the L p -norm also holds, see Theorem 5.2 in Munos 2007).

Original: For the approximate VI, the initial error ∥ V ∗ − V [-
 ∥-] ∞ decays with the rate of O ( γ k ) .
Revised: For the approximate VI, the initial error (cid:107) V ∗ − V [-
 ∥-]0 (cid:107) ∞ decays with the rate of O ( γ k ) .

Original: This should be compared with O ( γ ′ k ) rate of OS-VI.
Revised: This should be compared with O ( γ (cid:48) k ) rate of OS-VI.

Original: The effect of error at each step ϵ value is also similar: approximate VI has (1 − γ ) − 2 dependence while approximate OS-VI has (1 − γ ′ )
Revised: The effect of error at each step (cid:15) value is also similar: approximate VI has (1 − γ ) − 2 dependence while approximate OS-VI has (1 − γ (cid:48) )

Original: − 2 .
Revised: − 2 .

Original: What is remarkable is that as opposed to γ , which is a fixed parameter of the problem and can be close to 1 , γ ′ can be made arbitrary close to zero when the  model is accurate.
Revised: What is remarkable is that as opposed to γ , which is a ﬁxed parameter of the problem and can be close to 1 , γ (cid:48) can be made arbitrary close to zero when the approximate model ˆ P becomes more accurate.

Original: The additional information given by ˆ P allows us to get much faster rate than VI.
Revised: The additional information given by ˆ P allows us to get much faster rate than VI.

Original: Of course, this requires the model to be accurate.
Revised: Of course, this requires the model to be accurate.

Original: An inaccurate model might be detrimental to the convergence rate, and may even lead to divergence.
Revised: An inaccurate model might be detrimental to the convergence rate, and may even lead to divergence.

Original: Similar conclusions can be made in comparing OS-VI with Policy Iteration and Modified Policy Iteration, as discussed in the supplementary material.
Revised: Similar conclusions can be made in comparing OS-VI with Policy Iteration and Modiﬁed Policy Iteration, as discussed in the supplementary material.

Original:  Operator splitting Dyna
Revised:  Operator splitting Dyna

Original: Inthis section, we generalize theoperator splitting approach to an RL problem in which only  samples from P are available.
Revised: In the RL setting, we only have access to samples from P .

Original: In this scenario, the approximate model ˆ Pmay not be given, as assumed in previous sections. However, one can learn such an approximate model using samples from  the environment as done in MBRL algorithms.
Revised: MBRL algorithms, such as variants of the Dyna architecture, learn ˆ P from those samples, and use it to ﬁnd the value function or policy.

Original: Unlike MBRL approaches that solely rely on the model, our approach takes advantage ofboth the true environment and the model in its updates. Therefore, one can think of  the operator splitting approach as a hybrid of MBRL and model-free RL. [Table] [Table]
Revised: The learned model ˆ P is generally different from P due to the ﬁniteness of the samples as well as the possibility of model approximation error: the true dynamics P may not be representable with the function approximator used to represent ˆ P .

Original: [Table]
Revised: This is another way to say that the world may be too big to be represented by our models.

Original: [Table] [Table] [Table] [Table]
Revised: A MBRL algorithm that uses ˆ P in lieu of P does not ﬁnd the true value of the true MDP.

Original: [Table] [Table] [Table]
Revised: Based on OS-VI, we propose OS-Dyna, as a hybrid model-based and model-free RL algorithm that takes advantage of both the true environment and the model in its updates, and can converge to the true value function despite using inaccurate ˆ P .

Original: [Table] [Table] [Table]
Revised: Learning ˆ P in OS-Dyna is similar to other MBRL algorithms: one can use various model learning approaches, either based on maximum likelihood estimate or a decision-aware approach, to learn it.

Original: Using a learned ˆ P , we can calculate V k from the auxiliary reward function ¯ r k ≜ ¯ r V k − 1 by solving the
Revised: Given a learned ˆ P , we can compute V k from the auxiliary reward function ¯ r k (cid:44) ¯ r V k − 1 by solving the

Original: PE or the control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.
Revised: PE or the Control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.

Original: A possible challenge of using a learned model is that in many cases, we can only take samples from it and do not have access to the full matrices in calculations.

Original: Fortunately, solving PE or control problems with such a model is a very common practice in RL and is also done in Dyna architecture.

Original: Therefore, we do not focus on this procedure. The ability of calculating V k from ¯ r k lets us see the value function as a function of ¯ r k . Based on this view, we shift our attention to ¯ r k . Note that since there is a fixed connection between ¯ r k and
Revised: As V k is a function of ¯ r k , we focus on how ¯ r k should be estimated.

Original: V k as the solution of PE or control in ( X , A , ¯ r k , ˆ P ) , the convergence of ¯ r k and V k are tied together.

Original: Therefore, focusing on updating ¯ r k will have the same convergence properties.

Original: Thefollowing update rules can be written for ¯ r k itself. For every ( x, a ) , 
Revised: The update rule of ¯ r k in OS-VI entails that for every ( x, a ) , we have

Original: [Equation]
Revised: [Equation]

Original: In OS-Dyna, we maintain a vector (or a function approximation) for ¯ r , and as we will show how, update it by samples.
Revised: We update ¯ r using samples, as shall be discussed soon, and then the value function is updated to

Original: The value function is updated to V π (¯ r, ˆ P ) (for PE) or V ∗ (¯ r, ˆ P ) (for control) with each update of ¯ r .
Revised: V π (¯ r, ˆ P ) (for PE) or V ∗ (¯ r, ˆ P ) (for Control) with most recent estimate of ¯ r .

Original: This way, we have the value functions in update rules (5.1) and (5.2).

Original: Theonly challenge is that the above update rules need access to distributions P ( ·| x, a ) and ˆ P ( ·| x, a ) for every ( x, a ) , while we only have samples from these distributions in some ( x, a ) pairs.
Revised: The challenge is that the above update rules need access to distribution P ( ·| x, a ) for every ( x, a ) , while we only have samples from P at some ( x, a ) pairs.

Original: Fortunately, this challenge has been tackled in developing sample-based algorithms based on the classic VI:
Revised: Fortunately, this challenge has been tackled in developing sample-based algorithms based on the classic VI:

Original: [Equation]
Revised: [Equation]

Original: where V k − 1 =
Revised: where V k − 1 =

Original: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a ′ Q k − 1 ( x, a ) in Control.
Revised: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a (cid:48) Q k − 1 ( x, a ) in Control.

Original: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Q-Iteration and Stochastic Approximation (SA).
Revised: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Value Iteration and Stochastic Approximation (SA)

Revised: [Borkar, 2008].

Original: In this paper we use SA to develop OS-Dyna, but we point out that other algorithms and techniques can also be applied to develop other versions of OSDyna.
Revised: In this paper we use SA to develop OS-Dyna, but we point out that other algorithms and techniques can also be applied to develop other versions of OS-Dyna.

Original: The key step in SA is to use samples to find an unbiased estimate of the intended update value.
Revised: The key step in SA is to use samples to form an unbiased estimate of the intended update value.

Original: For a step  ( X t , A t , R t , X ′ t ) in the true environment, we can have thefollowing estimateZ =
Revised: For a step in the true environment leading to ( X t , A t , R t , X (cid:48) t ) tuple, we can have the estimate

Original:  R t +γV
Revised: Z = R [Equation] +

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original:  E X ′ ∼
Revised: [Equation] E X [Equation] ∼

Original: [Equation]
Revised: [Equation]

Original: ( ·| X t ,A t ) [ V ( X ′ )] , where the expectation can also be estimated by samples from ˆ P ( ·| X t , A t ) .
Revised: | X [Equation] [Equation]

Original: Finally, we make the following update to ¯ r with learning rate α t
Revised: [ V ( X

Original: [Equation]
Revised: [Equation]

Original:  t 
Revised: )] , where the expectation can also be estimated by samples from ˆ P ( ·| X t , A t ) . This estimate Z of the update rule can then be used to update ¯ r . As an example, for a ﬁnite state-action problem, the update rule is

Original: [Equation]
Revised: [Equation]

Revised: where α t is the learning rate.

Original: The final procedure of OS-Dyna is presented in Algorithm 1.
Revised: The ﬁnal procedure of OS-Dyna is presented in Algorithm 1.

Original:  Experiments
Revised:  Experiments

Original: We evaluate both OS-VI and OS-Dyna in a finite MDP, comparing our algorithms with existing methods.
Revised: We evaluate both OS-VI and OS-Dyna in a ﬁnite MDP and compare them with existing methods.

Original: The MDP considered is a modified cliffwalk environment in a 6 × 6 grid with 4 actions (up, down, left and right).
Revised: Here we present the results for the Control problem on a modiﬁed cliffwalk environment in a 6 ×grid with 4 actions (UP, DOWN, LEFT, RIGHT).

Original: Full details of the environment is available in the supplementary material.
Revised: We postpone studying the PE problem, the results for other environments, and other relevant details to the supplementary material.

Original: Our convergence analysis shows that the convergence rates of our algorithms depend on the accuracy of ˆ P .
Revised: Our convergence analysis shows that the convergence rates of our algorithms depend on the accuracy of ˆ P .

Original: To test OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothed model ˆ P of transitions P with smoothing parameter λ as
Revised: To test OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothed model ˆ P of transitions P with smoothing parameter λ as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Figure]
Revised: [Figure]

Revised: N

Original: [Figure]
Revised: [Figure]

Original: m a li ze d k Vπ k− V ∗ k
Revised: m a li z e d || V k V * ||

Original: [Figure]
Revised: [Figure]

Original: where U ( A ) for some set A is the uniform distribution over A .
Revised: where U ( A ) for some set A is the uniform distribution over A .

Original: Here, λ allows making adjustments to the amount of error introduced in ˆ P w.r.t.
Revised: Here, λ allows making adjustments to the amount of error introduced in ˆ P w.r.t.

Original: P .
Revised: P .

Original: If λ = 0 , ˆ P
Revised: If λ = 0 , ˆ P

Original: = P will be the accurate model, and if λ = 1 , ˆ P will be uniform over possible next states in P .
Revised: = P will be the accurate model, and if λ = 1 , ˆ P will be uniform over possible next states in P .

Original: We compare OS-VI and OS-Dyna for Control with other existing methods.

Original: The results for the PE problem are qualitatively similar and are provided in the supplementary material along with more experiments.

Original: The left plot in Figure 2 shows the convergence of OS-VI compared to VI and the solutions the model itself would lead to.
Revised: The left plot in Figure 2 shows the convergence of OS-VI compared to VI and the solutions the model itself would lead to.

Original: The plot shows normalized error of V π k w.r.t V ∗ , i.e.
Revised: The plot shows normalized error of V π k w.r.t V ∗ , i.e., (cid:107) V

Original: ∥ V π k − V ∗ ∥ 1 / ∥ V ∗ ∥ 1 .
Revised: π k − V ∗ (cid:107) 1 / (cid:107) V ∗ (cid:107) 1 .

Original: It can be seen that OS-VI has faster convergence with more accurate models and introduces acceleration compared to VI across different model errors.
Revised: It can be seen that OS-VI has faster convergence with more accurate models and introduces acceleration compared to VI across different model errors.

Original: Note that the convergence of OS-VI has been achieved despite the error in the model.
Revised: Note that the convergence of OS-VI has been achieved despite the error in the model.

Original: The dashed lines show how a fully model-based algorithm would obtain a suboptimal solution by only relying on the model.
Revised: The dashed lines show how a fully model-based algorithm, which only uses ˆ P , would obtain a suboptimal solution.

Original: We also compare OS-Dyna withwith Dyna and Q-Learning in the RL setting.
Revised: We also compare OS-Dyna with Dyna and Q-Learning in the RL setting.

Original: At each iteration t , the algorithms are given a sample ( X t , A t , R t , X ′ t ) where X t , A t are selected uniformly at random.
Revised: At each iteration t , the algorithms are given a sample ( X t , A t , R t , X (cid:48) t ) where X t , A t are selected uniformly at random.

Original: For
Revised: For

Original: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.
Revised: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.

Original: If P MLE is the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ ) defined in (6.1) as their models.
Revised: If P MLE is the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ ) deﬁned in (6.1) as their models.

Original: The learning rates are constant α for iterations t ≤ N and then decay in the form of α t = α/ ( t − N ) afterwards.
Revised: The learning rates are constant α for iterations t ≤ N and then decay in the form of α t = α/ ( t − N ) afterwards.

Original: We have fine-tuned the learning rate schedule for each algorithm separately for the best results.
Revised: We have ﬁne-tuned the learning rate schedule for each algorithm separately for the best results.

Original: The right plot in Figure 2 shows the results for the RL setting.
Revised: The right plot in Figure 2 shows the results for the RL setting.

Original: We evaluate the expected return of the policy at iteration t in the initial state of the environment, i.e. V π t (0) .
Revised: We evaluate the expected return of the policy at iteration t in the initial state of the environment, i.e., V π t (0) .

Original: Again, OS-Dyna has converged to the optimal policy much faster than Q-Learning.
Revised: Again, OS-Dyna has converged to the optimal policy much faster than Q-Learning.

Original: Unlike OS-Dyna, Dyna has failed to find the optimal policy in presence of model error.
Revised: Unlike OS-Dyna, Dyna has failed to ﬁnd the optimal policy in presence of model error.

Original: The results show that OS-Dyna can effectively converge faster than Q-Learning without introducing bias to the final solution due to model error.
Revised: The results show that OS-Dyna can effectively converge faster than Q-Learning without introducing bias to the ﬁnal solution due to model error.

Original:  Conclusion
Revised:  Conclusion

Original: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can benefit from an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to the true value function in terms of the number of queries to the true model P .
Revised: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can beneﬁt from an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to the true value function in terms of the number of queries to the true model P .

Original: With a small model error, its convergence rate is exponentially faster compared to well-known dynamical programming algorithms such as Value Iteration and Policy Iteration.
Revised: With a small model error, its convergence rate is exponentially faster compared to well-known dynamical programming algorithms such as Value Iteration and Policy Iteration.

Original: We also proposed OS-Dyna as a hybrid model-based/model-free algorithm that can bring in the benefits of a model-based RL algorithm without converging to a biased solution, as Dyna or any other purely model-based RL algorithm does.
Revised: We also proposed OS-Dyna as a hybrid model-based/model-free algorithm that can bring in the beneﬁts of a model-based RL algorithm without converging to a biased solution, as Dyna or any other purely model-based RL algorithm does.

Original: This paper opens up several future directions.
Revised: This paper opens up several future directions.

Original: Empirically studying the algorithms on problems with large state spaces, for which a function approximator such as a DNN is required, is an obvious one.
Revised: Empirically studying the algorithms on problems with large state spaces, for which a function approximator such as a DNN is required, is an obvious one.

Original: This is postponed to a future work as our aim was to build the mathematical foundation and conducting experiments without worrying about challenges such as the optimization of a DNN.
Revised: This is postponed to a future work as our aim was to build the mathematical foundation and conducting experiments without worrying about challenges such as the optimization of a DNN.

Original: There are other algorithmic and theoretical directions to be pursued.
Revised: There are other algorithmic and theoretical directions to be pursued.

Original: One is exploring the space of splittings of I − γ P π .
Revised: One is exploring the space of splittings of I − γ P π .

Original: The other is whether we can design Operator Splitting variants of other DP algorithms such as Policy Iteration and Modified Policy Iteration, and study their convergence behaviour.
Revised: The other is whether we can design Operator Splitting variants of other DP algorithms such as Policy Iteration and Modiﬁed Policy Iteration, and study their convergence behaviour.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Revised: We would like to thank the anonymous reviewers for their comments that helped us to improve the clarity of the paper, as well as other members of our lab who provided feedback on a draft of this paper.

Original: AMF acknowledges the funding from the Canada CIFAR AI Chairs program, as well as the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program.
Revised: AMF acknowledges the funding from the Canada CIFAR AI Chairs program, as well as the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program.

Original: AR was partially supported by Borealis AI through the Borealis AIGlobal Fellowship
Revised: AR was partially supported by Borealis AI through the Borealis AI

Original:  Award.
Revised: Global Fellowship Award.

Original: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR and companies sponsoring the Vector Institute.
Revised: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR and companies sponsoring the Vector Institute.