Original: Operator Splitting Value Iteration
Revised: Operator Splitting Value Iteration

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We introduce new planning and reinforcement learning algorithms for discounted
Revised: We introduce new planning and reinforcement learning algorithms for discounted

Original: MDPs that utilize an approximate model of the environment to accelerate theconvergence of the value function.
Revised: MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function.

Original: Inspired by the splitting approach in numericallinear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both
Revised: Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both

Original: Policy Evaluation and Control problems.
Revised: Policy Evaluation and Control problems.

Original: OS-VI achieves a much faster convergence rate when the model is accurate enough.
Revised: OS-VI achieves a much faster convergence rate when the model is accurate enough.

Original: We also introduce a sample-basedversion of the algorithm called OS-Dyna.
Revised: We also introduce a sample-based version of the algorithm called OS-Dyna.

Original: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of modelapproximation error.
Revised: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of model approximation error.

Original:  Introduction
Revised:  Introduction

Original: Consider a planning problem for a discounted MDP with dynamics P .
Revised: Consider a planning problem for a discounted MDP with dynamics P .

Original: Suppose that we have accessto an approximate model ˆ P ≈ P as well.
Revised: Suppose that we have access to an approximate model ˆ P ≈ P as well.

Original: For example, P might be a high-fidelity, but slow, simulator,and ˆ P is a lower-fidelity, but fast, simulator.
Revised: For example, P might be a high-fidelity, but slow, simulator, and ˆ P is a lower-fidelity, but fast, simulator.

Original: Or in a different, but relevant, context of model-basedreinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which wecan only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaplyacquired.
Revised: Or in a different, but relevant, context of model-based reinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which we can only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaply acquired.

Original: Can we use this approximate model ˆ P to accelerate the computation of the value functionof a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?
Revised: Can we use this approximate model ˆ P to accelerate the computation of the value function of a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?

Original: The Value Iteration (VI) algorithm, and its approximate variant, is a fundamental algorithm in Dynamic Programming that can find the value of a policy or the optimal value function.
Revised: The Value Iteration (VI) algorithm, and its approximate variant, is a fundamental algorithm in Dynamic Programming that can find the value of a policy or the optimal value function.

Original: It is alsoa backbone of many reinforcement learning (RL) algorithms such as Temporal Difference Learning [Sutton, 1988], Fitted Value Iteration
Revised: It is also the backbone of many reinforcement learning (RL) algorithms such as Temporal Difference Learning [Sutton, 1988], Fitted Value Iteration

Original: [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári,2008], and Deep Q Network [Mnih et al., 2015].
Revised: [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári, 2008], and Deep Q Network [Mnih et al., 2015].

Original: This algorithm, however, can be slow when thediscount factor is close to 1 , as its convergence rate is O ( γ k ) .
Revised: This algorithm, however, can be slow when the discount factor is close to 1 , as its convergence rate is O ( γ k ) .

Original: Moreover, even though we could use
Revised: Moreover, even though we could use

Original: VI using ˆ P instead of P , effectively avoiding any need for expensive queries to P , the obtained valuefunction would converge to a solution different from the value function of the original MDP.
Revised: VI using ˆ P instead of P , effectively avoiding any need for expensive queries to P , the obtained value function would converge to a solution different from the value function of the original MDP.

Original: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that benefits froman approximate model ˆ P to potentially accelerate the convergence of the value function sequence tothe value function with respect to (w.r.t.)
Revised: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that benefits from an approximate model ˆ P to potentially accelerate the convergence of the value function sequence to the value function with respect to (w.r.t.)

Original: the true model P (Section 3).
Revised: the true model P (Section 3).

Original: This algorithm is for both
Revised: This algorithm is for both

Original: PE (Section 3.1) and Control (Section 3.2) problems.
Revised: PE (Section 3.1) and Control (Section 3.2) problems.

Original: The acceleration is not uniform though, anddepends on how close ˆ P is to P (Section 4).
Revised: The acceleration is not uniform though, and depends on how close ˆ P is to P (Section 4).

Original: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra,which is used for iterative solution of large linear systems of equations [Varga, 2000, Saad, 2003,
Revised: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra, which is used for iterative solution of large linear systems of equations [Varga, 2000, Saad, 2003,

Original: Golub and Van Loan, 2013].
Revised: Golub and Van Loan, 2013].

Original: With a proper choice of splitting, one may change the convergencerate of linear system of equation solvers.
Revised: With a proper choice of splitting, one may change the convergence 36th Conference on Neural Information Processing Systems (NeurIPS 2022). rate of linear system of equation solvers.

Original: We show that the conventional VI for PE can be seen as aparticular choice of splitting.
Revised: We show that the conventional VI for PE can be seen as a particular choice of splitting.

Original: This observation suggests that one may choose other forms of splittingin order to change the convergence rate.
Revised: This observation suggests that one may choose other forms of splitting in order to change the convergence rate.

Original: It turns out that we can choose a splitting that benefits from having access to ˆ P (Section 2).
Revised: It turns out that we can choose a splitting that benefits from having access to ˆ P (Section 2).

Original: The new splitting leads to OS-VI for PE.
Revised: The new splitting leads to OS-VI for PE.

Original: For the Control problem,the connection between solving linear systems of equations and VI is not as straightforward anymore,as the former is linear, while the latter is not, but we can still get inspired from the splitting approachto design OS-VI for Control.
Revised: For the Control problem, the connection between solving linear systems of equations and VI is not as straightforward anymore, as the former is linear, while the latter is not, but we can still get inspired from the splitting approach to design OS-VI for Control.

Original: The key step of such an algorithm is a new policy improvement step.
Revised: The key step of such an algorithm is a new policy improvement step.

Original: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate modelˆ P is learned using data.
Revised: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate model ˆ P is learned using data.

Original: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].
Revised: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].

Original: OS-Dyna is a hybrid of model-free and model-based algorithms.
Revised: OS-Dyna is a hybrid of model-free and model-based algorithms.

Original: It uses thelearned model in its inner planning loop, but uses samples from the true model P in order to correctthe effect of errors in the model.
Revised: It uses the learned model in its inner planning loop, but uses samples from the true model P in order to correct the effect of errors in the model.

Original: Existing MBRL algorithms would converge to an incorrect solutionif the approximate model ˆ P does not converge to the true model P .
Revised: Existing MBRL algorithms would converge to an incorrect solution if the approximate model ˆ P does not converge to the true model P .

Original: This would be the case whenevermodel approximation error exists.
Revised: This would be the case whenever model approximation error exists.

Original: On the other hand, OS-Dyna can still converge to the correct valuefunction even when ˆ P does not converge to P .
Revised: On the other hand, OS-Dyna can still converge to the correct value function even when ˆ P does not converge to P .

Original: As far as we know, this is the first model-based RLalgorithm with such property.
Revised: As far as we know, this is the first model-based RL algorithm with such property.

Original:  From value iteration to splitting-based linear system of equations solversand back
Revised:  From value iteration to splitting-based linear system of equations solvers and back

Original: We briefly describe the VI algorithm and the splitting methods for solving linear systems of equations, and explain their connections.
Revised: We briefly describe the VI algorithm and the splitting methods for solving linear systems of equations, and explain their connections.

Original: We consider a discounted Markov Decision Process (MDP)( X , A , R , P , γ )
Revised: We consider a discounted Markov Decision Process (MDP) ( X , A , R , P , γ )

Original: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].
Revised: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].

Original: Wedefer formal definitions to the supplementary material.
Revised: We defer formal definitions to the supplementary material.

Original: We only mention that for a policy π , wedenote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π =V π
Revised: We only mention that for a policy π , we denote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π =

Original:  ( R , P ) its state-value function.
Revised: V π ( R , P ) its state-value function.

Original: We also represent the optimal state-valuefunction by V ∗ =
Revised: We also represent the optimal state-value function by V ∗ =

Original: V ∗ ( R , P ) and the optimal policy by π ∗ =
Revised: V ∗ ( R , P ) and the optimal policy by π ∗ =

Original: π ∗ ( R , P ) .
Revised: π ∗ ( R , P ) .

Revised: The Bellman operator

Original: T π : B (  ) [-are 
 [Equation] ≜-]
Revised: T π : B ( X ) [-are 
 [Equation] ≜-]→ B ( X ) for policy π and the Bellman optimality operator T ∗ : B ( X ) → B ( X ) are( T

Original: [Equation]
Revised: [Equation]

Original: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ : V (cid:55)→max π { r π + γ P π V } .
Revised: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ : V (cid:55)→ max π { r π + γ P π V } .

Original: The greedy policy at state x ∈ X is
Revised: The greedy policy at state x ∈ X is

Original: [Equation]
Revised: [Equation]

Original: ← argmax

Original: [Equation]

Original: , or more compactly, π g ( V ) ← argmax π T π V .
Revised: or more compactly, π g ( V ) ← argmax π T π V .

Original: We have T ∗ V
Revised: We have T ∗ V

Original: = T π g ( V ) V , that is, the effect of the
Revised: = T π g ( V ) V , that is, the effect of the

Original: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellmanoperator of the greedy policy w.r.t.
Revised: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellman operator of the greedy policy w.r.t.

Original: V to V .
Revised: V to V .

Original: Finally, we denote the m -step transition kernel of policyπ by P π ( m ) and define the discounted future-state distribution as

Original: [Equation]

Original: 2.1 Value Iteration
Revised: 2.1 Value Iteration

Original: The value function V π and the optimal value function V ∗ are the fixed points of the operators T πand T ∗ , respectively, and satisfy the Bellman equation.
Revised: The value function V π and the optimal value function V ∗ are the fixed points of the operators T π and T ∗ , respectively, and satisfy the Bellman equation.

Original: For the PE problem, this means that
Revised: For the PE problem, this means that

Original: [Equation]
Revised: [Equation]

Original: There are several ways to compute the value function of a policy π or the optimal value function
Revised: There are several ways to compute the value function of a policy π or the optimal value function

Original: V ∗ , including the iterative methods such asthe Value Iteration (VI) andthe Policy Iteration (PI)algorithms, or solving a linear systems of equations (for PE) or linear programming (for Control).
Revised: V ∗ , including the iterative methods such as Value Iteration (VI) and Policy Iteration (PI) algorithms, or solving a linear systems of equations (for PE) or linear programming (for Control).

Original: Wefocus on the Value Iteration algorithm in this work.
Revised: We focus on the VI algorithm in this work.

Original: The VIalgorithm repeatedly applies the Bellmanoperator to the most recent approximation of the value function:
Revised: VI repeatedly applies the Bellman operator to the most recent approximation of the value function:

Original: Given an initial value function V 0 , itgenerates a sequence ( V k ) k ≥ 0 as follows:
Revised: Given an initial value function V 0 , it generates a sequence ( V k ) k ≥ 0 as follows:

Original: [Equation]
Revised: [Equation]

Original: VI for Control can be written in an equivalent form: At iteration k , we first compute the greedypolicy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .
Revised: VI for Control can be written in an equivalent form: At iteration k , we first compute the greedy policy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .

Original: Therefore, the policyimprovement step is obtained through finding a policy that is greedy w.r.t.
Revised: Therefore, the policy improvement step is obtained through finding a policy that is greedy w.r.t.

Original: the last value function
Revised: the last value function

Original: V k − 1 , that is, the best policy if we only look one step ahead.
Revised: V k − 1 , that is, the best policy if we only look one step ahead.

Original: This form will be conductive for our laterdevelopments.
Revised: This form will be conductive for our later developments.

Original: As the Bellman operator is a γ -contraction w.r.t.
Revised: As the Bellman operator is a γ -contraction w.r.t.

Original: the supremum norm, the convergencerate of V k to V π (or V ∗ ) would be O ( γ k ) .
Revised: the supremum norm, the convergence rate of V k to V π (or V ∗ ) would be O ( γ k ) .

Original: This rate can be slow when γ is very close to 1 .
Revised: This rate can be slow when γ is very close to 1 .

Original: 2.2 Matrix splitting for solving linear systems of equations
Revised: 2.2 Matrix splitting for solving linear systems of equations

Original: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear systemof equations (2.2).
Revised: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear system of equations (2.1).

Original: Consider the linear systems of equations Az = b , with A ∈ R d × d and z, b ∈ R d .
Revised: Consider the linear systems of equations Az = b , with A ∈ R d × d and z, b ∈ R d .

Original: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally,
Revised: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally,

Original: A , M , and N can be linear operators).
Revised: A , M , and N can be linear operators).

Original: Therefore, z satisfies Mz =
Revised: Therefore, z satisfies Mz =

Original: Nz + b .
Revised: Nz + b .

Original: The splitting-basediterative approach defines the new approximation z k given the current z k − 1 by solvingMz [Equation] = Nz k
Revised: The splitting-based iterative approach defines the new approximation z k given the current z k − 1 by solving

Original: [Equation]
Revised: [Equation]

Original: + b, or equivalently
Revised: or equivalently

Original: [Equation]
Revised: [Equation]

Original: To analyze the convergence of this iterative method, consider the error e k ≜ z k − z .
Revised: To analyze the convergence of this iterative method, consider the error e k ≜ z k − z .

Original: As Mz =
Revised: As Mz =

Original: Nz + b ,the dynamics of the error is
Revised: Nz + b , the dynamics of the error is

Original: [Equation]
Revised: [Equation]

Original: Let G ≜ M − 1 N .
Revised: Let G ≜ M − 1 N .

Original: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as
Revised: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as

Original: [Equation]
Revised: [Equation]

Original: The errors are (norm-) convergent if ∥ G ∥ =
Revised: The errors are (norm-) convergent if ∥ G ∥ =

Original: ∥ M − 1 N ∥ < 1 , for some choice of norm.
Revised: ∥ M − 1 N ∥ < 1 , for some choice of norm.

Original: More generally,the necessary and sufficient condition for convergence is that the spectral radius ρ ( G ) , the maximumof absolute values of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or
Revised: More generally, the necessary and sufficient condition for convergence is that the spectral radius ρ ( G ) , the maximum absolute value of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or

Original: Theorem 11.2.1 of Golub and Van Loan [2013].
Revised: Theorem 11.2.1 of Golub and Van Loan [2013].

Original: The convergence is faster if the spectral radius (ornorm) is closer to zero.
Revised: The convergence is faster if the spectral radius (or norm) is closer to zero.

Original: The success of this iterative procedure depends on how we choose M and N such that the norm (orspectral radius) is as small as possible.
Revised: The success of this iterative procedure depends on how we choose M and N such that the norm (or spectral radius) is as small as possible.

Original: Also we want to choose an M such that solving Mz k =
Revised: Also we want to choose an M such that solving Mz k =

Original: Nz k − 1 + b is not very expensive.
Revised: Nz k − 1 + b is not very expensive.

Original: For example, if M is an identity matrix I , we get that N
Revised: For example, if M is an identity matrix I , we get that N

Original: =
Revised: =

Original: I − A ,and the iteration becomes z k
Revised: I − A , and the iteration becomes z k

Original: = ( I − A )
Revised: = ( I − A )

Original: z k − 1 + b .
Revised: z k − 1 + b .

Original: This iteration is convergent if ρ ( I − A ) < 1 .
Revised: This iteration is convergent if ρ ( I − A ) < 1 .

Original: Othercommonly used choices lead to the Jacobi andthe Gauss-Seidel methods, which are described in thesupplementary material.
Revised: Other commonly used choices lead to the Jacobi and Gauss-Seidel methods that are described in the supplementary material.

Original: We are now ready to make the connection between splitting-based iterative methods to the VI for PE.
Revised: We are now ready to make the connection between splitting-based iterative methods and VI for PE.

Original: If we choose A = I − γ P π , we see thatthe equation
Revised: If we choose A = I − γ P π , we see that equation

Original: AV π = r π is indeed the Bellman equation forpolicy π (2.2).
Revised: AV π = r π is indeed the Bellman equation for policy π (2.1).

Original: The VI for PE, which is V k = γ P π V k − 1 + r π =
Revised: The VI for PE, which is V k = γ P π V k − 1 + r π =

Original: ( I − A )
Revised: ( I − A )

Original: V k − 1 + r π , corresponds tothe choice of M
Revised: V k − 1 + r π , corresponds to the choice of M

Original: =
Revised: =

Original: I and N = γ P π .
Revised: I and N = γ P π .

Original: This brings up the question of whether it is possible to split Ato other choices of M and N so that the resulting VI-like procedure has better improved convergenceproperties?
Revised: This brings up the question of whether it is possible to split A to other choices of M and N so that the resulting VI-like procedure has better convergence properties.

Original: We suggest a particular choice in the next section.
Revised: We suggest a particular choice in the next section.

Original:  Operator splitting value iteration algorithm
Revised:  Operator splitting value iteration algorithm

Original: This section introduces the Operator Splitting Value Iteration (OS-VI) algorithm.
Revised: This section introduces the Operator Splitting Value Iteration (OS-VI) algorithm.

Original: We start from the
Revised: We start from the

Original: PE problem, and introduce the Control version based on that.
Revised: PE problem and introduce the Control version based on that.

Original: 3.1 OS-VI for policy evaluation
Revised: 3.1 OS-VI for policy evaluation

Original: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as
Revised: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as

Original: [Equation]
Revised: [Equation]

Original: Following the recipe of (2.4), the OS-VI algorithm for PE would be
Revised: Following the recipe of (2.3), the OS-VI algorithm for PE would be

Original: [Equation]
Revised: [Equation]

Original: starting from an initial V 0 .
Revised: starting from an initial V 0 .

Original: 3 To gain more intuition and prepare for further developments, we definea few notations.
Revised: 3 To gain more intuition and prepare for further developments, we define a few notations.

Original: We define the Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga(1928 – 2022) who has made significant contributions to matrix analysis, as the mapping between thespace of all bounded functions over X to the same space as
Revised: We define the Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga (1928 – 2022) who has made significant contributions to matrix analysis, as the mapping between the space of all bounded functions over X to the same space as

Original: [Equation]
Revised: [Equation]

Original: Observe that (3.1) can be compactly written as
Revised: Observe that (3.1) can be compactly written as

Original: [Equation]
Revised: [Equation]

Original: It is not difficult to see that
Revised: It is not difficult to see that

Original: S
Revised: S

Original: π V π = V π , i.e., the value function of a policy π is a fixed-point of the
Revised: π V π = V π , i.e., the value function of a policy π is a fixed-point of the

Original: Varga operator S π .
Revised: Varga operator S π .

Original: This and other properties of the Varga operator are in the supplementary material.
Revised: This and other properties of the Varga operator are in the supplementary material.

Original: Given any value function V , define an auxiliary reward function ¯ r V : X ×
Revised: Given any value function V , define an auxiliary reward function ¯ r V : X ×

Original: A → R as
Revised: A → R as

Original: [Equation]
Revised: [Equation]

Original: Similar to the notation for r π , we define ¯ r πV :
Revised: Similar to the notation for r π , we define ¯ r πV :

Original: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministicpolicy π (and similarly for a stochastic policy).
Revised: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministic policy π (and similarly for a stochastic policy).

Original: With this notation,
Revised: With this notation,

Original: [Equation]
Revised: [Equation]

Original: This is the value function of following π in an MDP withthe dynamics ˆ P andthe reward ¯ r V .
Revised: This is the value function of following π in an MDP with dynamics ˆ P and reward ¯ r V .

Original: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximatedynamics, and a reward function that consists of the original reward r and the correction termγ ( P − ˆ P ) V k − 1 .
Revised: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximate dynamics, and a reward function that consists of the original reward r and the correction term γ ( P − ˆ P ) V k − 1 .

Original: The computation of this value function is a standard policy evaluation problemwith the approximate model.
Revised: The computation of this value function is a standard PE problem with the approximate model.

Original: For instance, we may use another VI (PE) with dynamics ˆ P to solveit: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P
Revised: For instance, we may use another VI (PE) with dynamics ˆ P to solve it: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P

Original: π
Revised: π

Original: U i − 1 .
Revised: U i − 1 .

Original: This convergesto S π V
Revised: This converges to S π V

Original: =
Revised: =

Original: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .
Revised: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .

Original: Note that aside  the computation of ¯ r πV ,which requires querying P in order to compute the P π V term, this iterative process only uses theapproximate model ˆ P , which is assumed to be cheap to access.
Revised: Note that aside from the computation of ¯ r πV , which requires querying P in order to compute the P π V term, this iterative process only uses the approximate model ˆ P , which is assumed to be cheap to access.

Original: What is the benefit of this OS-VI procedure?
Revised: What is the benefit of this OS-VI procedure?

Original: If the approximate model ˆ P is close to the true dynamics
Revised: If the approximate model ˆ P is close to the true dynamics

Original: P , this leads to a faster convergence of V k to V π , as shall be quantified soon.
Revised: P , this leads to a faster convergence of V k to V π , as shall be quantified soon.

Original: The faster convergenceis in terms of the number of queries to P , which is assumed to be expensive.
Revised: The faster convergence is in terms of the number of queries to P , which is assumed to be expensive.

Original: To see this, considerthe hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happensto match the reality perfectly.
Revised: To see this, consider the hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happens to perfectly match the reality.

Original: Then, S π V =
Revised: Then, S π V =

Original: ( I − γ P π ) − 1 ( r π + 0 V ) =
Revised: ( I − γ P π ) − 1 ( r π + 0 V ) =

Original: V π , and the value functionfor the original MDP is obtained in one iteration of OS-VI.
Revised: V π , and the value function for the original MDP is obtained in one iteration of OS-VI.

Original: Of course, we often can only hope forˆ P ≈ P .
Revised: Of course, we often can only hope for ˆ P ≈ P .

Original: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in moredetails, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ Pis only a close approximation of P .
Revised: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in more details, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ P is only a close approximation of P .

Original: 3.2 OS-VI for control
Revised: 3.2 OS-VI for control

Original: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ←π g ( V k − 1 ) =
Revised: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ← π g ( V k − 1 ) =

Original: argmax π T π V k − 1 in its policy improvement step, and uses one step of the Bellmanoperator w.r.t.
Revised: argmax π T π V k − 1 in its policy improvement step, and uses one step of the Bellman operator w.r.t.

Original: the obtained policy π k to compute the new estimate of the value function V k ← T π k V k − 1 , as described after (2.3).
Revised: the obtained policy π k to compute the new estimate of the value function V k ← T π k V k − 1 , as described after (2.2).

Original: The OS-VI algorithm for Control follows a similar structure withthe difference that (1) the improved policy is the optimizer of the Varga operator, and not the Bellmanoperator, and (2) the new value function is obtained by applying the Varga operator of the newlyobtained policy.
Revised: The OS-VI algorithm for Control follows a similar structure with the difference that 1) the improved policy is the optimizer of the Varga operator, and not the Bellman operator, and 2) the new value function is obtained by applying the Varga operator of the newly obtained policy.

Original: To be concrete, given a value function V , define the S -improved policy
Revised: To be concrete, given a value function V , define the S -improved policy

Original: [Equation]
Revised: [Equation]

Original: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .
Revised: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .

Original: We also define the Vargaoptimal operator S ∗ : B ( X ) → B ( X ) as
Revised: We also define the Varga optimal operator S ∗ : B ( X ) → B ( X ) as

Original: [Equation]
Revised: [Equation]

Original: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.
Revised: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.

Original: Vapplied to a value function V (compare it with T ∗ V 
Revised: V applied to a value function V (compare it with T ∗ V =

Original: = T π g ( V ) V ).
Revised: T π g ( V ) V ).

Original: The OS-VI (Control) is then simply
Revised: The OS-VI (Control) is then simply

Original: [Equation]
Revised: [Equation]

Original: which in its expanded form, consists of the following two steps:
Revised: which in its expanded form, consists of the following two steps:

Original: [Equation]
Revised: [Equation]

Original: Comparing the S -improved policy (3.4) used in the policy improvement step (3.6) of OS-VI withthe conventional greedy policy is insightful.
Revised: Comparing the S -improved policy [Equation] 4) used in the policy improvement step (3.6) of OS-VI with the conventional greedy policy is insightful.

Original: The greedy policy is argmax π T π V .
Revised: The greedy policy is argmax π T π V .

Original: Expanding T π V ,we see that the greedy policy is the maximizer of r π + γ P π V .
Revised: Expanding T π V , we see that the greedy policy is the maximizer of r π + γ P π V .

Original: The function r π + γ P π V is onlya single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy, is ingeneral different from the optimal policy, which maximizes V π .
Revised: The function r π + γ P π V is only a single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy, is in general different from the optimal policy, which maximizes V π .

Original: On the other hand, the S -improvedpolicy π V ( V ) solves a full MDP with an approximate model ˆ P and the reward function that hasboth the original reward r and the correction term γ
Revised: On the other hand, the S -improved policy π V ( V ) solves a full MDP with an approximate model ˆ P and the reward function that has both the original reward r and the correction term γ

Original: ( P − ˆ P ) V .
Revised: ( P − ˆ P ) V .

Original: In the special case that ˆ P = P , thecorrection term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.
Revised: In the special case that ˆ P = P , the correction term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.

Original: As oftenˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.
Revised: As often ˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.

Original: The partial policyevaluation step (3.7) updates the value function to a value that is closer to the optimal value function.
Revised: The partial policy evaluation step (3.7) updates the value function to a value that is closer to the optimal value function.

Original: Remark.
Revised: Remark.

Original: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context ofdynamic programming is not completely novel to this work.
Revised: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context of dynamic programming is not completely novel to this work.

Original: Kushner and Kleinman [1971] is oneof the earliest paper that mention the Jacobi and Gauss-Seidel procedures for computing the valuefunction.
Revised: Kushner and Kleinman [1971] is one of the earliest paper that mentions the Jacobi and Gauss-Seidel procedures for computing the value function.

Original: Porteus [1975] proposes several transformations to the reward andthe probability transitionmatrix with the goal of improving the computational cost of solving the transformed MDP.
Revised: Porteus [1975] proposes several transformations to the reward and probability transition matrix with the goal of improving the computational cost of solving the transformed MDP.

Original: One ofthe transformations, called pre-inverse transform , has some similarities with the operator splitting ofthis work.
Revised: One of the transformations, called pre-inverse transform , has some similarities with the operator splitting of this work.

Original: The end result, however, is different.
Revised: The end result, however, is different.

Original: Bacon and Precup [2016] provide a matrix splittingperspective on planning with options.
Revised: Bacon and Precup [2016] provide a matrix splitting perspective on planning with options.

Original: The connection between multi-step models and matrix splittingis developed in Chapter 4 of Bacon [2018].
Revised: The connection between multi-step models and matrix splitting is developed in Chapter 4 of Bacon [2018].

Original: Refer to the supplementary material for more discussion.
Revised: Refer to the supplementary material for more discussion.

Original: Convergence analysis of operator splitting value iteration
Revised: 3.3 Visualizing how OS-VI works

Revised: We visualize the value function trajectories of several algorithms, including OS-VI, on a 2-state MDP in order to provide some intuition on how OS-VI works. We consider the policy evaluation for the dynamics P π = [ 0 . 9 0 . 9 ] with the reward

Revised: r π = (cid:0) 1 − 0 .

Revised: 5 (cid:1) and γ = 0 .

Revised: We consider two approximate models: a relatively accurate one ˆ P π accurate =

Revised: 95 ] , and an inaccurate one ˆ P π inaccurate =

Revised: The results are shown in Figure 1.

Revised: The first algorithm is the conventional VI (PE), which repeatedly applies the Bellman operator according to the true model P π to the most recent approximation of the value function.

Revised: We use T π P to refer to its Bellman operator and to label the corresponding trajectory in the value space.

Revised: This algorithm converges to the true value function V π .

Revised: The second algorithm is VI (PE) that uses ˆ P π as the model.

Revised: This procedure is the basis of the Dyna architecture.

Revised: We use T π ˆ

Revised: P to refer to its Bellman operator and to label the corresponding trajectory in the value space.

Revised: Due to the error of T π ˆ P compared to T π P , the algorithm converges to a wrong value function ˆ V π , as both figures show.

Revised: We observe that even when the model is relatively accurate as in Figure 1a, the converged value function is quite wrong.

Revised: This illustrates one limitation of the conventional model-based RL algorithms where an inaccurate model may lead to significantly inaccurate estimate of the value function.

Revised: The OS-VI algorithm repeatedly applies the Varga operator S π to the most recent approximation of the value function.

Revised: As discussed earlier, each computation of S π V k − 1 corresponds to solving an

Revised: V (0) −−−

Revised: V ( 1 )

Revised: Vˆ V π

Revised: V π

Revised: T π P T π ˆ

Revised: P ¯ T πk

Revised: S π (a) More accurate model ˆ P

Revised: accurate

Revised: V (0) −−−

Revised: V ( 1 )

Revised: Vˆ V π

Revised: V π

Revised: T π P T π ˆ

Revised: P ¯ T πk

Revised: S π

Revised: [Figure]

Revised: auxiliary MDPs ( X , A , ¯ r V k − 1 , ˆ P , γ ) .

Revised: We denote the Bellman operator of this auxiliary MDP by ¯ T π k .

Revised: The figures show the trajectory generated by the iterative application of S π on the most recent value function as well as the trajectory for solving each auxiliary MDP, indicated by ¯ T πk .

Revised: We observe that the OS-VI algorithm converges to the correct value function despite using an incorrect model.

Revised: When the model is more accurate, very few iteration of OS-VI gets a value close to V π (two iterations in

Revised: Figure 1a)

Revised: ; when the model is less accurate, a few more iterations are needed.

Revised: Compared to VI, at least in these examples, the total number of iterations of OS-VI is significantly fewer.

Revised: When the initial value function is V 0 = 0 , the result of the first iteration of OS-VI is the same value function computed by the VI with the wrong model ˆ P π .

Original: {+This is because V 1 ← S π V 
+}
Revised: {+This is because V 1 ← S π V 
+}

Revised: = ( I − γ ˆ P π ) − 1 ¯ r πV 0 and ¯ r πV 0 =

Revised: r π , so V 1 =

Revised: ( I − γ ˆ P π ) − 1 r π .

Revised: This is the same solution as the value function obtained using the approximate model ˆ P .

Revised: In these figures, this shows itself by the overlapping of the red arrows followed by T π ˆ P and the first segment of the orange arrows, which are generated by the repeated application of ¯ T π 1 .

Revised: In further iterations of OS-VI, the auxiliary MDPs change and the path followed by ¯ T πk ( k ≥ 2 ) deviates from the solution of the VI with the wrong model.

Revised:  Convergence analysis of operator splitting value iteration

Original: In this section, we present the convergence analysis of OS-VI.
Revised: In this section, we present the convergence analysis of OS-VI.

Original: Our results show that OS-VI has an
Revised: Our results show that OS-VI has an

Original: O ( γ ′ k ) convergence rate for an effective discount factor γ ′ that depends on the error between ˆ P and
Revised: O ( γ ′ k ) convergence rate for an effective discount factor γ ′ that depends on the error between ˆ P and

Original: P .
Revised: P .

Original: For small enough error, γ ′ < γ and OS-VI has a faster convergence rate compared to the classic
Revised: For small enough error, γ ′ < γ and OS-VI has a faster convergence rate compared to the classic

Original: VI, Policy Iteration (PI), and Modified Policy Iteration (MPI), which all have O ( γ k ) behaviour.
Revised: VI, Policy Iteration (PI), and Modified Policy Iteration (MPI), which all have O ( γ k ) behaviour.

Original: Weprovide results for both the L ∞ and L p norms.
Revised: We provide results for both the L ∞ and L p norms.

Original: 4.1 Convergence of OS-VI for policy evaluation
Revised: 4.1 Convergence of OS-VI for policy evaluation

Original: We study  convergence behaviour of OS-VI (PE) in presence of error in value updates.
Revised: We study the convergence behaviour of OS-VI (PE) in presence of error in value updates.

Original: Morespecifically, we considerthe setting that in each iteration k , the update (3.2) has an error, i.e.,
Revised: Specifically, we consider that in each iteration k , the update (3.2) has an error, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The error ϵ value k encompasses various sources of errors that might occur in a practical application of
Revised: The error ϵ value k encompasses various sources of errors.

Original: OS-VI.

Original: This includes the function approximation error, caused by using a function approximatorto represent V k , and the estimation (i.e., statistical) error caused due to having a finite number ofsamples, instead of direct access to P , in the RL setting, see [Györfi et al., 2002] for a generaldiscussion of function approximation/estimation error in the supervised learning, and [Munos and 
Revised: One source is the function approximation error due to using a function approximator to represent V k , which is often required in large state-action spaces. Another is the estimation (i.e., statistical) error caused due to having a finite number of samples, instead of direct access to P , in the RL setting. Refer to Györfi et al. [2002], Steinwart and Christmann [2008] for the discussion of function approximation and estimation errors in the supervised learning context and to Munos and Szepesvári [2008], Antos et al.

Original: Szepesvári, 2008, Antos et al., 2008, Farahmand et al., 2016, Chen and Jiang, 2019].
Revised: [2008], Farahmand et al. [2016], Chen and Jiang [2019], Fan et al. [2019] for their analysis in the RL context.

Original: In this work,we do not analyze how the number of samples, the function approximator, etc.
Revised: In this work, we do not analyze how the number of samples, the function approximator, etc.

Original: affect the errors ϵ value k .
Revised: affect the errors ϵ value k .

Original: The result of this section provides an error propagation result similar to Munos [2007] (forapproximate VI), Munos[2003] (for approximate PI), and Scherrer et al. [2015] (for approximatemodified PI).
Revised: The result of this section provides an error propagation result similar to Munos [2007] (approximate VI), Munos

Original: To study convergence behaviour of OS-VI (PE), let G π
Revised: To study convergence behaviour of OS-VI (PE), let G π

Original: = ( I − γ ˆ P π ) − 1 γ
Revised: = ( I − γ ˆ P π ) − 1 γ

Original: ( P π − ˆ P π ) .
Revised: ( P π − ˆ P π ) .

Original: We use thefact that S
Revised: We use the fact that S

Original: π V π = V π
Revised: π V π = V π

Original: and write
Revised: and write

Original: [Equation]
Revised: [Equation]

Original: Now, we have that
Revised: Now, we have that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that for any square matrix F with a matrix norm ∥ F ∥ p < 1 , it holdsthat ∥ ( I − F ) − 1 ∥ p ≤ 1 1 −∥ F ∥ p (e.g., Lemma 2.3.3 of Golub and Van Loan 2013), and that thesupremum norm of a stochastic matrix ˆ P π is 1 .
Revised: where we used the fact that for any square matrix F with a matrix norm ∥ F ∥ p < 1 , it holds that ∥ ( I − F ) − 1 ∥ p ≤ [Equation] (e.g., Lemma 2.3.3 of Golub and Van Loan 2013), and that the supremum norm of a stochastic matrix ˆ P π is 1 .

Original: Assuming that ∥ ϵ value k ∥ ∞ ≤ ϵ value for all k ≥ 1 anddefining effective discount factor γ ′ = γ 1 − γ ∥P π − ˆ P π ∥ ∞ , the upper bounds (4.3) and (4.2) lead to
Revised: Assuming that ∥ ϵ value k ∥ ∞ ≤ ϵ value for all k ≥ 1 and defining effective discount factor γ ′ = γ [Equation] − ˆ P π ∥ ∞ , the upper bounds (4.3) and (4.2) lead to

Original: [Equation]
Revised: [Equation]

Original: A few remarks are in order.
Revised: A few remarks are in order.

Original: First, whenever γ ′ < γ , this is guaranteed to be faster than the convergencerate of the conventional VI, which is O ( γ k ) .
Revised: First, whenever γ ′ < γ , this is guaranteed to be faster than the convergence rate of the conventional VI, which is O ( γ k ) .

Original: This is the case if ∥P π − ˆ P π ∥ ∞ < 1 − γ .
Revised: This is the case if ∥P π − ˆ P π ∥ ∞ < 1 − γ .

Original: If the modelis very accurate, we obtain much faster rate than VI’s.
Revised: If the model is very accurate, we obtain much faster rate than VI’s.

Original: Since each iteration k corresponds to a queryto the true model P , a faster rate entails that the algorithm requires fewer total number of queries tothe expensive model to reach the same level of accuracy.
Revised: Since each iteration k corresponds to a query to the true model P , a faster rate entails that the algorithm requires less number of queries to the expensive model to reach the same level of accuracy.

Original: Second, although the model error ∥P π − ˆ P π ∥ ∞ is a reasonable choice to measure the distancesbetween distributions (it is the maximum of the Total Variation distance between P π ( ·| x ) and ˆ P π ( ·| x )over x , which itself can be upper bounded by their KL divergence; see the supplementary material), itis somewhat conservative as it takes the supremum over the state space.
Revised: Second, although the model error ∥P π − ˆ P π ∥ ∞ is a reasonable choice to measure the distances between distributions (it is the maximum of the Total Variation distance between P π ( ·| x ) and ˆ P π ( ·| x ) over x , which itself can be upper bounded by their KL divergence; see the supplementary material), it is somewhat conservative as it takes the supremum over the state space.

Original: Likewise, requiring ∥ ϵ value k ∥ ∞210 to be small is also conservative, as approximating S π V k − 1 using a function approximator givensamples (RL setting) often leads to an L p -norm type of guarantee.
Revised: Likewise, requiring ∥ ϵ value k ∥ ∞ to be small is also conservative, as approximating S π V k − 1 using a function approximator given samples (RL setting) often leads to an L p -norm type of guarantee.

Original: We now provide a differentanalysis to address these issues.
Revised: We now provide a different analysis to address these issues.

Original: To present the L p norm result, we need to define some notations.
Revised: To present the L p -norm result, we need to define some notations.

Original: First, we define the conditionaldiscounted future-state distribution of policy π under ˆ P as the following probability distribution:
Revised: First, we define the conditional discounted future-state distribution of policy π under ˆ P as the following probability distribution:

Original: Given a measurable set B , we have ˆ η π ( B | x ) =
Revised: Given a measurable set B , we have ˆ η π ( B | x ) =

Original: [-(1 − γ ) (cid:80) ∞ t =
-]
Revised: [-(1 − γ ) (cid:80) ∞ t =
-][Equation]

Original: γ t · P (cid:16) X t ∈ B | X 

Revised: [Equation] [Equation] ∈ B | X

Original: = x, π, ˆ P (cid:17) , wherethe chain ( X t ) t ≥ 0 starts from state x, and evolves by following policy π under  ˆ P transitions.
Revised: = x, π, ˆ P (cid:17) , where the chain ( X t ) t ≥ 0 starts from state x and evolves by following policy π under transitions ˆ P .

Original: This isthe same as (2.1) with the change of P to ˆ P .

Original: For an arbitrary distribution ρ over the state space, wedefine the  discounted future-state distribution concentration coefficient as
Revised: For an arbitrary distribution ρ over the state space, we define the approximate discounted future-state distribution concentration coefficient as

Original: [Equation]
Revised: [Equation]

Original:  Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.
Revised: [Equation] Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.

Original: the distribution ρ ,and it is assumed that for any x ∈ X , ˆ η π ( ·| x ) ≪ ρ , i.e., ˆ η π ( ·| x ) is absolutely continuous w.r.t.
Revised: the distribution ρ , and it is assumed that for any x ∈ X , ˆ η π ( ·| x ) ≪ ρ , i.e., ˆ η π ( ·| x ) is absolutely continuous w.r.t.

Original: ρ(otherwise, the coefficient would be set to infinity).
Revised: ρ (otherwise, the coefficient would be set to infinity).

Original: This coefficient measures how concentrated thedistribution ˆ η π ( ·| x ) is compared to ρ .
Revised: This coefficient measures how concentrated the distribution ˆ η π ( ·| x ) is compared to ρ .

Original: This is weighted according to the initial state distribution ρ .
Revised: This is weighted according to the initial state distribution ρ .

Original: Similar concentrability coefficients, but not exactly this one, have appeared in the error propagationresults [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al.
Revised: Similar concentrability coefficients, but not exactly this one, have appeared in the error propagation results [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al., 2015].

Original: Finally, we define the weighted χ 2 -divergence of ˆ P π and P π asthe following:
Revised: Finally, we define the weighted χ 2 -divergence of ˆ P π and P π as

Original: [Equation]
Revised: [Equation]

Original: This notion of model error is less strict in requiring accurate approximation P in all states.
Revised: This notion of model error is less strict in requiring accurate approximation P in all states.

Original: Usuallyonly a subset of state space is important or even reachable in a problem.
Revised: Usually only a subset of state space is important or even reachable in a problem.

Original: The above model error canonly focus on specific areas of  state space through the choice of distribution ρ .
Revised: The above model error can only focus on specific areas of the state space through the choice of distribution ρ .

Original: We are now ready to to provide the main theorem for the approximate OS [Table]
Revised: We are now ready to to provide the main theorem for the approximate OS-VI (PE).

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider the approximate OS-VI algorithm for PE (4.1) .
Revised: Consider the approximate OS-VI algorithm for PE (4.1) .

Original: Let ∥·∥ ⋆ be either the supremumnorm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over  state space ( ⋆ = 4 , ρ ).
Revised: Let ∥·∥ ⋆ be either the supremum norm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over the state space ( ⋆ = 4 , ρ ).

Original: Assume that ∥ ϵ value k
Revised: Assume that ∥ ϵ value k

Original: ∥ ⋆ ≤ ϵ value for all k ≥ 1 .
Revised: ∥ ⋆ ≤ ϵ value for all k ≥ 1 .

Original: Furthermore, define the effective discount factor as
Revised: Furthermore, define the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: For any k ≥ 0 , we have
Revised: For any k ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: 4.2 Convergence of OS-VI for control
Revised: 4.2 Convergence of OS-VI for control

Original: We turn to analyzing OS-VI for Control.
Revised: We turn to analyzing OS-VI for Control.

Original: Similar to the analysis of OS-VI for PE, we assume someerror in each step of OS-VI.

Original: Each iteration of OS-VI for PE was solving the PE problem in theauxiliary MDP

Original: ( X , A , ¯ r V , ˆ P , γ ) .

Original: The error of this solution was the error of the calculated valuefunction V k +1 w.r.t.

Original: the true value function in the MDP S π V k leading to (4.1).

Original: In Control, eachiteration is solving the control problem in the same auxiliary MDP. The difference isthat we have twotypes of error. First, the error of value function w.r.t.
Revised: We consider two types of errors: The first is the error between the computed value function and the true optimal value function of the auxiliary MDP, i.e.,

Original: true solution, i.e., V k − S ∗ V k − 1 .
Revised: V k − S ∗ V k − 1 .

Original: Second is thesuboptimality of  policy compared to the optimal policy of  auxiliary MDP, i.e., S π k V k − 1 − S ∗ V k − 1 .
Revised: The second is the suboptimality of obtained policy compared to the optimal policy of the auxiliary MDP, i.e., S π k V k − 1 − S ∗ V k − 1 .

Original: More specifically
Revised: Concretely, we have

Original: [Equation]
Revised: [Equation]

Revised: We have the following result for OS-VI (Control).

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .
Revised: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .

Original: Let ∥·∥ ⋆ be eitherthe supremum norm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over statespace ( ⋆ = 4 , ρ ).
Revised: Let ∥·∥ ⋆ be either the supremum norm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over the state space ( ⋆ = 4 , ρ ).

Original: For any k ≥ 1 , let Π k
Revised: For any k ≥ 1 , let Π k

Original: =
Revised: =

Original: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 0 ≤ i < k } .
Revised: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 1 ≤ i < k } .

Original: Assume that∥ ϵ value k
Revised: Assume that ∥ ϵ value k

Original: ∥ ⋆ ≤ ϵ value for all k ≥ 1 .
Revised: ∥ ⋆ ≤ ϵ value for all k ≥ 1 .

Original: Furthermore, define the effective discount factor as
Revised: Furthermore, define the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: We can compare this result with the convergence result of VI.
Revised: We can compare this result with the convergence result of VI.

Original: For VI with the supremum norm,following the proof of Equation (2.2) by Munos [2007], we can show that ∥ V ∗ − V π k ∥ ∞ ≤ 
Revised: For VI with the supremum norm, following the proof of Equation (2.2) by Munos [2007], we can show that ∥ V ∗ − V π k ∥ ∞ ≤ 2 γ k

Original: [Equation]
Revised: [Equation]

Original: , with ∥ V i − T ∗ V i − 1 ∥ ∞ ≤ ε for all
Revised: , with ∥ V i − T ∗ V i − 1 ∥ ∞ ≤ ε for all

Original: i < k (similar result for the
Revised: i < k (similar result for the

Original: L p -norm also holds, see Theorem 5.2 of Munos 2007).
Revised: L p -norm also holds, see Theorem 5.2 in Munos 2007).

Original: For the approximate VI, the initial error∥ V ∗ − V 

Revised: For the approximate VI, the initial error ∥ V ∗ − V

Original: ∥ ∞ decays with the rate of O ( γ k ) .
Revised: ∥ ∞ decays with the rate of O ( γ k ) .

Original: This should be compared with O ( γ ′ k ) rate of OS-VI.
Revised: This should be compared with O ( γ ′ k ) rate of OS-VI.

Original: The effect of error at each step ϵ value is also similar: approximate VI has (1 − γ ) − 2 dependence whileapproximate OS-VI has (1 − γ ′ ) − 2 .
Revised: The effect of error at each step ϵ value is also similar: approximate VI has (1 − γ ) − 2 dependence while approximate OS-VI has (1 − γ ′ ) − 2 .

Original: What is remarkable is that as opposed to γ , which is a fixedparameter of the problem and can be close to 1 , γ ′ can be made arbitrary close to zero when themodel is accurate.
Revised: What is remarkable is that as opposed to γ , which is a fixed parameter of the problem and can be close to 1 , γ ′ can be made arbitrary close to zero when the model is accurate.

Original: The additional information given by ˆ P allows us to get much faster rate than VI.
Revised: The additional information given by ˆ P allows us to get much faster rate than VI.

Original: Of course, this requires the model to be accurate.
Revised: Of course, this requires the model to be accurate.

Original: An inaccurate model might be detrimental to theconvergence rate, and may even lead to divergence.
Revised: An inaccurate model might be detrimental to the convergence rate, and may even lead to divergence.

Revised: Similar conclusions can be made in comparing OS-VI with Policy Iteration and Modified Policy Iteration, as discussed in the supplementary material.

Original:  Operator splitting Dyna
Revised:  Operator splitting Dyna

Original: In this section, we generalize the operator splitting approach to an RL problem in which only samplesfrom P are available.
Revised: In this section, we generalize the operator splitting approach to an RL problem in which only samples from P are available.

Original: In this scenario, the approximate model ˆ P may not be given, as assumedin previous sections.
Revised: In this scenario, the approximate model ˆ P may not be given, as assumed in previous sections.

Original: However, one can learn such an approximate model using samples from theenvironment as done in MBRL algorithms.
Revised: However, one can learn such an approximate model using samples from the environment as done in MBRL algorithms.

Original: Unlike MBRL approaches that solely rely on the model,our approach takes advantage of both the true environment and the model in its updates.
Revised: Unlike MBRL approaches that solely rely on the model, our approach takes advantage of both the true environment and the model in its updates.

Original: Therefore,one can think of the operator splitting approach as a hybrid of MBRL and model-free RL. 
Revised: Therefore, one can think of the operator splitting approach as a hybrid of MBRL and model-free RL. [Table] [Table] [Table]

Original: Using a learned ˆ P , we can calculate V k from the auxiliary reward function ¯ r k ≜ ¯ r V k − 1 by solving
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Using a learned ˆ P , we can calculate V k from the auxiliary reward function ¯ r k ≜ ¯ r V k − 1 by solving the

Original: PE or  control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.
Revised: PE or the control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.

Original: A possiblechallenge of using a learned model is that in many cases, we can only take samples from it and donot have access to the full matrices in calculations.
Revised: A possible challenge of using a learned model is that in many cases, we can only take samples from it and do not have access to the full matrices in calculations.

Original: Fortunately, solving PE or control problems withsuch a model is a very common practice in RL and is also done in Dyna architecture.
Revised: Fortunately, solving PE or control problems with such a model is a very common practice in RL and is also done in Dyna architecture.

Original: Therefore, wedo not focus on this procedure.
Revised: Therefore, we do not focus on this procedure.

Original: The ability of calculating V k from ¯ r k lets us see the value function as a function of ¯ r k .
Revised: The ability of calculating V k from ¯ r k lets us see the value function as a function of ¯ r k .

Original: Based onthis view, we shift our attention to ¯ r k .
Revised: Based on this view, we shift our attention to ¯ r k .

Original: Note that since there is a fixed connection between ¯ r k and
Revised: Note that since there is a fixed connection between ¯ r k and

Original: V k as the solution of PE or control in ( X , A , ¯ r k , ˆ P ) , the convergence of ¯ r k and V k are tied together.
Revised: V k as the solution of PE or control in ( X , A , ¯ r k , ˆ P ) , the convergence of ¯ r k and V k are tied together.

Original: Therefore, focusing on updating ¯ r k will have the same convergence properties.
Revised: Therefore, focusing on updating ¯ r k will have the same convergence properties.

Original: The following updaterules can be written for ¯ r k itself.
Revised: The following update rules can be written for ¯ r k itself.

Original: For every ( x, a ) ,
Revised: For every ( x, a ) ,

Original: [Equation]
Revised: [Equation]

Original: In OS-Dyna, we maintain a vector (or a function approximation) for ¯ r , and as we will show how,update it by samples.
Revised: In OS-Dyna, we maintain a vector (or a function approximation) for ¯ r , and as we will show how, update it by samples.

Original: The value function is updated to V π (¯ r, ˆ P ) (for PE) or V ∗ (¯ r, ˆ P ) (for control)with each update of ¯ r .
Revised: The value function is updated to V π (¯ r, ˆ P ) (for PE) or V ∗ (¯ r, ˆ P ) (for control) with each update of ¯ r .

Original: This way, we have the value functions in update rules (5.1) and (5.2).
Revised: This way, we have the value functions in update rules (5.1) and (5.2).

Original: The onlychallenge is that the above update rules need access to distributions P ( ·| x, a ) and ˆ P ( ·| x, a ) for every( x, a ) , while we only have samples from these distributions in some ( x, a ) pairs.
Revised: The only challenge is that the above update rules need access to distributions P ( ·| x, a ) and ˆ P ( ·| x, a ) for every ( x, a ) , while we only have samples from these distributions in some ( x, a ) pairs.

Original: Fortunately, thischallenge has been tackled in developing sample-based algorithms based on  classic value iteration:
Revised: Fortunately, this challenge has been tackled in developing sample-based algorithms based on the classic VI:

Original: [Equation]
Revised: [Equation]

Original: where V k − 1 =
Revised: where V k − 1 =

Original: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a ′ Q k − 1 ( x, a ) in Control.
Revised: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a ′ Q k − 1 ( x, a ) in Control.

Original: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Q-Iterationand Stochastic Approximation (SA).
Revised: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Q-Iteration and Stochastic Approximation (SA).

Original: In this paper we use SA to develop OS-Dyna, but we pointout that other algorithms and techniques can also be applied to develop other versions of OSDyna.
Revised: In this paper we use SA to develop OS-Dyna, but we point out that other algorithms and techniques can also be applied to develop other versions of OSDyna.

Original: The key step in SA is to use samples to find an unbiased estimate of the intended updatevalue.
Revised: The key step in SA is to use samples to find an unbiased estimate of the intended update value.

Original: For a step ( X t , A t , R t , X ′ t ) in the true environment, we can have the following estimate 
Revised: For a step ( X t , A t , R t , X ′ t ) in the true environment, we can have the following estimate Z =

Original: Z = R t + γV
Revised: R t + γV

Original: ( X′ t )
Revised: [Equation] X

Original: − γ E X ′ ∼ ˆ P ( ·| X t ,A t )
Revised: [Equation] E X ′ ∼ [Equation]

Original:  [ V ( X ′ )] , where the expectation can also be estimated bysamples from ˆ P ( ·| X t , A t ) .
Revised: ( ·| X t ,A t ) [ V ( X ′ )] , where the expectation can also be estimated by samples from ˆ P ( ·| X t , A t ) .

Original: Finally, we make the following update to ¯ r with learning rate α t¯ r
Revised: Finally, we make the following update to ¯ r with learning rate α t

Original: ( X t , A t ) ← ¯ r ( X t , A t ) + α t · ( Z − ¯ r
Revised: [Equation]

Original: ( X t , A t )) .
Revised: t [Equation]

Original: The final procedure of OS-Dyna is presented in Algorithm 1.
Revised: The final procedure of OS-Dyna is presented in Algorithm 1.

Original:  Experiments
Revised:  Experiments

Original: We evaluate both OS-VI and OS-Dyna in a finite MDP, comparing our algorithms with existingmethods.
Revised: We evaluate both OS-VI and OS-Dyna in a finite MDP, comparing our algorithms with existing methods.

Original: The MDP considered is a modified cliffwalk environment in a 6 × 6 grid with 4 actions(up, down, left and right).
Revised: The MDP considered is a modified cliffwalk environment in a 6 × 6 grid with 4 actions (up, down, left and right).

Original: Full details of the environment is available in the supplementary material.
Revised: Full details of the environment is available in the supplementary material.

Original: Our convergence analysis shows that the convergence rates of our algorithms depend on the accuracyof ˆ P .
Revised: Our convergence analysis shows that the convergence rates of our algorithms depend on the accuracy of ˆ P .

Original: To test OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothedmodel ˆ P of transitions P with smoothing parameter λ as
Revised: To test OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothed model ˆ P of transitions P with smoothing parameter λ as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Figure]

Revised: [Figure]

Revised: m a li ze d k V π k − V ∗ k

Original: [Figure]
Revised: [Figure]

Original: where U ( A ) for some set A is the uniform distribution over A .
Revised: where U ( A ) for some set A is the uniform distribution over A .

Original: Here, λ allows making adjustments tothe amount of error introduced in ˆ P w.r.t.
Revised: Here, λ allows making adjustments to the amount of error introduced in ˆ P w.r.t.

Original: P .
Revised: P .

Original: If λ = 0 , ˆ P
Revised: If λ = 0 , ˆ P

Original: = P will be the accurate model, and ifλ = 1 , ˆ P will be uniform over possible next states in P .
Revised: = P will be the accurate model, and if λ = 1 , ˆ P will be uniform over possible next states in P .

Original: We compare OS-VI and OS-Dyna for Control with other existing methods.
Revised: We compare OS-VI and OS-Dyna for Control with other existing methods.

Original: The results for the PEproblem are qualitatively similar and are provided in the supplementary material along with moreexperiments.
Revised: The results for the PE problem are qualitatively similar and are provided in the supplementary material along with more experiments.

Original: The left plot in Figure 1 shows the convergence of OS-VI compared to VI and thesolutions the model itself would lead to.
Revised: The left plot in Figure 2 shows the convergence of OS-VI compared to VI and the solutions the model itself would lead to.

Original: The plot shows normalized error of V π k w.r.t V ∗ , i.e.∥ V π k − V ∗ ∥ 1 / ∥ V ∗ ∥ 1 .
Revised: The plot shows normalized error of V π k w.r.t V ∗ , i.e. ∥ V π k − V ∗ ∥ 1 / ∥ V ∗ ∥ 1 .

Original: It can be seen that OS-VI has faster convergence with more accurate modelsand introduces acceleration compared to VI across different model errors.
Revised: It can be seen that OS-VI has faster convergence with more accurate models and introduces acceleration compared to VI across different model errors.

Original: Note that the convergenceof OS-VI has been achieved despite the error in the model.
Revised: Note that the convergence of OS-VI has been achieved despite the error in the model.

Original: The dashed lines show how a fullymodel-based algorithm would obtain a suboptimal solution by only relying on the model.
Revised: The dashed lines show how a fully model-based algorithm would obtain a suboptimal solution by only relying on the model.

Original: We also compare OS-Dyna with with Dyna and Q-Learning in the RL setting.
Revised: We also compare OS-Dyna with with Dyna and Q-Learning in the RL setting.

Original: At each iteration t , thealgorithms are given a sample ( X t , A t , R t , X ′ t ) where X t , A t are selected uniformly at random.
Revised: At each iteration t , the algorithms are given a sample ( X t , A t , R t , X ′ t ) where X t , A t are selected uniformly at random.

Original: For
Revised: For

Original: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.
Revised: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.

Original: If P MLEis the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ )defined in (6.1) as their models.
Revised: If P MLE is the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ ) defined in (6.1) as their models.

Original: The learning rates are constant α for iterations t ≤ N and then decayin the form of α t = α/ ( t − N ) afterwards.
Revised: The learning rates are constant α for iterations t ≤ N and then decay in the form of α t = α/ ( t − N ) afterwards.

Original: We have fine-tuned the learning rate schedule for eachalgorithm separately for the best results.
Revised: We have fine-tuned the learning rate schedule for each algorithm separately for the best results.

Original: The right plot in Figure 1 shows the results for the RL setting.
Revised: The right plot in Figure 2 shows the results for the RL setting.

Original: We evaluate the expected return of thepolicy at iteration t in the initial state of the environment, i.e. V π k (0) .
Revised: We evaluate the expected return of the policy at iteration t in the initial state of the environment, i.e. V π t (0) .

Original: Again, OS-Dyna has convergedto the optimal policy much faster than Q-Learning for all model errors.
Revised: Again, OS-Dyna has converged to the optimal policy much faster than Q-Learning.

Original: Unlike OS-Dyna,classic Dyna has failed to find the optimal policy in presence of model error.
Revised: Unlike OS-Dyna, Dyna has failed to find the optimal policy in presence of model error.

Original: The results show that OS-Dynacan effectively converge faster than Q-Learning without introducing bias to the final solution due tomodel error.
Revised: The results show that OS-Dyna can effectively converge faster than Q-Learning without introducing bias to the final solution due to model error.

Original:  Conclusion
Revised:  Conclusion

Original: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can benefitfrom an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to thetrue value function in terms of the number of queries to the true model P .
Revised: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can benefit from an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to the true value function in terms of the number of queries to the true model P .

Original: With a small modelerror, its convergence rate is exponentially faster compared to well-known dynamical programmingalgorithms such as Value Iteration and Policy Iteration.
Revised: With a small model error, its convergence rate is exponentially faster compared to well-known dynamical programming algorithms such as Value Iteration and Policy Iteration.

Original: We also proposed OS-Dyna as a hybridmodel-based/model-free algorithm that can bring in the benefits of a model-based RL algorithmwithout converging to a biased solution, as Dyna or any other purely model-based RL algorithmdoes.
Revised: We also proposed OS-Dyna as a hybrid model-based/model-free algorithm that can bring in the benefits of a model-based RL algorithm without converging to a biased solution, as Dyna or any other purely model-based RL algorithm does.

Original: This paper opens up several future directions.
Revised: This paper opens up several future directions.

Original: Empirically studying the algorithms on problemswith large state spaces, for which a function approximator such as a DNN is required, is an obviousone.
Revised: Empirically studying the algorithms on problems with large state spaces, for which a function approximator such as a DNN is required, is an obvious one.

Original: This is postponed to a future work as our aim was to build the mathematical foundation andconducting experiments without worrying about challenges such as the optimization of a DNN.
Revised: This is postponed to a future work as our aim was to build the mathematical foundation and conducting experiments without worrying about challenges such as the optimization of a DNN.

Original: Thereare other algorithmic and theoretical directions to be pursued.
Revised: There are other algorithmic and theoretical directions to be pursued.

Original: One is exploring the space of splittingsof I − γ P π .
Revised: One is exploring the space of splittings of I − γ P π .

Original: The other is whether we can design Operator Splitting variants of other DP algorithmssuch as Policy Iteration and Modified Policy Iteration, and study their convergence behaviour. 
Revised: The other is whether we can design Operator Splitting variants of other DP algorithms such as Policy Iteration and Modified Policy Iteration, and study their convergence behaviour. Acknowledgments and Disclosure of Funding

Revised: AMF acknowledges the funding from the Canada CIFAR AI Chairs program, as well as the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery

Revised: Grant program.

Revised: AR was partially supported by Borealis AI through the Borealis AI Global Fellowship

Revised: Award.

Revised: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR and companies sponsoring the Vector Institute.