Original: Operator Splitting Value Iteration
Revised: Operator Splitting Value Iteration

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We introduce new planning and reinforcement learning algorithms for discounted
Revised: We introduce new planning and reinforcement learning algorithms for discounted

Original: MDPs that utilize an approximate model of the environment to accelerate theconvergence of the value function.
Revised: MDPs that utilize an approximate model of the environment to accelerate theconvergence of the value function.

Original: Inspired by the splitting approach in numericallinear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both
Revised: Inspired by the splitting approach in numericallinear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both

Original: Policy Evaluation and Control problems.
Revised: Policy Evaluation and Control problems.

Original: OS-VI achieves a much faster convergence rate when the model is accurate enough.
Revised: OS-VI achieves a much faster convergence rate when the model is accurate enough.

Original: We also introduce a sample-basedversion of the algorithm called OS-Dyna.
Revised: We also introduce a sample-basedversion of the algorithm called OS-Dyna.

Original: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of modelapproximation error.
Revised: Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of modelapproximation error.

Original:  Introduction
Revised:  Introduction

Original: Consider a planning problem for a discounted MDP with dynamics P .
Revised: Consider a planning problem for a discounted MDP with dynamics P .

Original: Suppose that we have accessto an approximate model ˆ P ≈ P as well.
Revised: Suppose that we have accessto an approximate model ˆ P ≈ P as well.

Original: For example, P might be a high-ﬁdelity, but slow, simulator,and ˆ P is a lower-ﬁdelity, but fast, simulator.
Revised: For example, P might be a high-fidelity, but slow, simulator,and ˆ P is a lower-fidelity, but fast, simulator.

Original: Or in a different, but relevant, context of model-basedreinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which wecan only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaplyacquired.
Revised: Or in a different, but relevant, context of model-basedreinforcement learning (MBRL), P is the unknown dynamics of a real-world system, from which wecan only acquire expensive samples, and ˆ P is a learned model, from which samples can be cheaplyacquired.

Original: Can we use this approximate model ˆ P to accelerate the computation of the value functionof a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?
Revised: Can we use this approximate model ˆ P to accelerate the computation of the value functionof a policy π (Policy Evaluation (PE) problem) or the optimal value function (Control problem)?

Original: The Value Iteration (VI) algorithm, and its approximate variant, is a fundamental algorithm in Dynamic Programming that can ﬁnd the value of a policy or the optimal value function.
Revised: The Value Iteration (VI) algorithm, and its approximate variant, is a fundamental algorithm in Dynamic Programming that can find the value of a policy or the optimal value function.

Original: It is alsoa backbone of many reinforcement learning (RL) algorithms such as Temporal Difference Learning [Sutton, 1988], Fitted Value Iteration
Revised: It is alsoa backbone of many reinforcement learning (RL) algorithms such as Temporal Difference Learning [Sutton, 1988], Fitted Value Iteration

Original: [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári,2008], and Deep Q Network [Mnih et al., 2015].
Revised: [Gordon, 1995, Ernst et al., 2005, Munos and Szepesvári,2008], and Deep Q Network [Mnih et al., 2015].

Original: This algorithm, however, can be slow when thediscount factor is close to 1 , as its convergence rate is O ( γ k ) .
Revised: This algorithm, however, can be slow when thediscount factor is close to 1 , as its convergence rate is O ( γ k ) .

Original: Moreover, even though we could use
Revised: Moreover, even though we could use

Original: VI using ˆ P instead of P , effectively avoiding any need for expensive queries to P , the obtained valuefunction would converge to a solution different from the value function of the original MDP.
Revised: VI using ˆ P instead of P , effectively avoiding any need for expensive queries to P , the obtained valuefunction would converge to a solution different from the value function of the original MDP.

Original: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that beneﬁts froman approximate model
Revised: This paper proposes an algorithm called Operator Splitting Value Iteration (OS-VI) that benefits froman approximate model

Original: ˆ P to potentially accelerate the convergence of the value function sequence tothe value function with respect to (w.r.t.)
Revised: ˆ P to potentially accelerate the convergence of the value function sequence tothe value function with respect to (w.r.t.)

Original: the true model P (Section 3).
Revised: the true model P (Section 3).

Original: This algorithm is for both
Revised: This algorithm is for both

Original: PE (Section 3.1) and Control (Section 3.2) problems.
Revised: PE (Section 3.1) and Control (Section 3.2) problems.

Original: The acceleration is not uniform though, anddepends on how close ˆ P  to Pis (Section 4).
Revised: The acceleration is not uniform though, anddepends on how close ˆ P is to P (Section 4).

Original: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra,which is used for iterative solution of large linear systems of equations [Varga, 2000, Saad, 2003,
Revised: A key inspiration behind OS-VI is the (matrix) splitting approach in the numerical linear algebra,which is used for iterative solution of large linear systems of equations [Varga, 2000, Saad, 2003,

Original: Golub and Van Loan, 2013].
Revised: Golub and Van Loan, 2013].

Original: With a proper choice of splitting, one may change the convergencerate of linear system of equation solvers.
Revised: With a proper choice of splitting, one may change the convergencerate of linear system of equation solvers.

Original: We show that the conventional VI for PE can be seen asa particular choice of splitting.
Revised: We show that the conventional VI for PE can be seen as aparticular choice of splitting.

Original: This  suggests that one may choose other forms of splitting in orderto change the convergence rate.
Revised: This observation suggests that one may choose other forms of splittingin order to change the convergence rate.

Original: It turns out that we can choose a splitting that beneﬁts from having access to ˆ P (Section 2).
Revised: It turns out that we can choose a splitting that benefits from having access to ˆ P (Section 2).

Original: This leads to OS-VI for PE.
Revised: The new splitting leads to OS-VI for PE.

Original: For the Control problem, the connection betweensolving linear systems of equations and VI is not as straightforward anymore, as the former is linear,while the latter is not, but we can still get inspired from the splitting approach to design OS-VI for Control.
Revised: For the Control problem,the connection between solving linear systems of equations and VI is not as straightforward anymore,as the former is linear, while the latter is not, but we can still get inspired from the splitting approachto design OS-VI for Control.

Original: The key step of such an algorithm is a new policy improvement step.
Revised: The key step of such an algorithm is a new policy improvement step.

Original: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate modelˆ P is learned using data.
Revised: The form of the OS-VI algorithm opens up a connection to MBRL where the approximate modelˆ P is learned using data.

Original: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].
Revised: This leads to the OS-Dyna algorithm, inspired by a generic Dyna architecture [Sutton, 1990].

Original: OS-Dyna is a hybrid of model-free and model-based algorithms.
Revised: OS-Dyna is a hybrid of model-free and model-based algorithms.

Original: It uses thelearned model in its inner planning loop, but uses samples from the true model P in order to correctthe effect of errors in the model.
Revised: It uses thelearned model in its inner planning loop, but uses samples from the true model P in order to correctthe effect of errors in the model.

Original: Existing MBRL algorithms would converge to an incorrect solutionif the approximate model
Revised: Existing MBRL algorithms would converge to an incorrect solutionif the approximate model

Original: ˆ P does not converge to the true model P .
Revised: ˆ P does not converge to the true model P .

Original: This would be the case whenevermodel approximation error exists.
Revised: This would be the case whenevermodel approximation error exists.

Original: On the other hand, OS-Dyna can still converge to the correct valuefunction even when ˆ P does not converge to P .
Revised: On the other hand, OS-Dyna can still converge to the correct valuefunction even when ˆ P does not converge to P .

Original: As far as we know, this is the ﬁrst model-based withsuch property.
Revised: As far as we know, this is the first model-based RLalgorithm with such property.

Original:  From value iteration to splitting-based linear system of equation solversand back
Revised:  From value iteration to splitting-based linear system of equations solversand back

Original: We brieﬂy describe the VI algorithm and the splitting methods for solving linear systems of equations, and explain their connections.
Revised: We briefly describe the VI algorithm and the splitting methods for solving linear systems of equations, and explain their connections.

Original: We consider a discounted Markov Decision Process (MDP)( X , A , R , P , γ )
Revised: We consider a discounted Markov Decision Process (MDP)( X , A , R , P , γ )

Original: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].
Revised: [Bertsekas and Tsitsiklis, 1996, Szepesvári, 2010, Sutton and Barto, 2019].

Original: Wedefer formal deﬁnitions to the supplementary material.
Revised: Wedefer formal definitions to the supplementary material.

Original: We only mention that for a policy π , wedenote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π = V π
Revised: We only mention that for a policy π , wedenote by P π its transition kernel, by r π : X → R the expected value of its reward distribution, and by V π = V π

Original: ( R , P ) its state-value function.
Revised: ( R , P ) its state-value function.

Original: We also represent the optimal state-valuefunction by V ∗ =
Revised: We also represent the optimal state-valuefunction by V ∗ =

Original: V ∗ ( R , P ) and the optimal policy by π ∗ =
Revised: V ∗ ( R , P ) and the optimal policy by π ∗ =

Original: π ∗ ( R , P ) .
Revised: π ∗ ( R , P ) .

Original: The Bellman operator T π : B (X ) → B ( X ) for policy π and the Bellman optimality operator T ∗ : B ( X ) → B ( X ) are {+
+}
Revised: T π : B ( ) are {+
+}

Original: [Equation]
Revised: [Equation]

Revised: ≜

Original: [Equation]
Revised: [Equation]

Original: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ : V (cid:55)→max π { r π + γ P π V } .
Revised: These operators can be written more compactly as T π : V (cid:55)→ r π + γ P π V and T ∗ : V (cid:55)→max π { r π + γ P π V } .

Original: The greedy policy at state x ∈ X is
Revised: The greedy policy at state x ∈ X is

Original: [Equation]
Revised: [Equation]

Revised: ← argmax

Revised: [Equation]

Original:  or more compactly, π g ( V ) ← argmax π T π V .
Revised: , or more compactly, π g ( V ) ← argmax π T π V .

Original: We have T ∗ V
Revised: We have T ∗ V

Original: = T π g ( V ) V , that is, the effect of the
Revised: = T π g ( V ) V , that is, the effect of the

Original: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellmanoperator of the greedy policy w.r.t.
Revised: Bellman optimality operator T ∗ applied to a value function V is the same as applying the Bellmanoperator of the greedy policy w.r.t.

Original: V to V .
Revised: V to V .

Original: Finally, we denote the m -step transition kernel of policyπ by P π ( m ) and deﬁne the discounted future-state distribution as
Revised: Finally, we denote the m -step transition kernel of policyπ by P π ( m ) and define the discounted future-state distribution as

Original: [Equation]
Revised: [Equation]

Original: 2.1 Value Iteration
Revised: 2.1 Value Iteration

Original: The value function V π and the optimal value function V ∗ are the ﬁxed points of the operators T πand T ∗ , respectively, and satisfy the Bellman equation.
Revised: The value function V π and the optimal value function V ∗ are the fixed points of the operators T πand T ∗ , respectively, and satisfy the Bellman equation.

Original: For the PE problem, this means that
Revised: For the PE problem, this means that

Original: [Equation]
Revised: [Equation]

Original: There are several ways to compute the value function of a policy π or the optimal value function
Revised: There are several ways to compute the value function of a policy π or the optimal value function

Original: V ∗ , including the iterative methods such as the Value Iteration (VI) and the Policy Iteration (PI)algorithms, or solving a linear systems of equations (for PE) or linear programming (for Control).
Revised: V ∗ , including the iterative methods such as the Value Iteration (VI) and the Policy Iteration (PI)algorithms, or solving a linear systems of equations (for PE) or linear programming (for Control).

Original: Wefocus on the Value Iteration algorithm in this work.
Revised: Wefocus on the Value Iteration algorithm in this work.

Original: The VI algorithm repeatedly applies the Bellmanoperator to the most recent approximation of the value function:
Revised: The VI algorithm repeatedly applies the Bellmanoperator to the most recent approximation of the value function:

Original: Given an initial value function V 0 , itgenerates a sequence ( V k ) k ≥ 0 as follows:
Revised: Given an initial value function V 0 , itgenerates a sequence ( V k ) k ≥ 0 as follows:

Original: [Equation]
Revised: [Equation]

Original: VI for Control can be written in an equivalent form: At iteration k , we ﬁrst compute the greedypolicy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .
Revised: VI for Control can be written in an equivalent form: At iteration k , we first compute the greedypolicy π k ← π g ( V k − 1 ) (policy improvement step), and then V k ← T π k V k − 1 .

Original: Therefore, the policyimprovement step is obtained through ﬁnding a policy that is greedy w.r.t.
Revised: Therefore, the policyimprovement step is obtained through finding a policy that is greedy w.r.t.

Original: the last value function
Revised: the last value function

Original: V k − 1 , that is, the best policy if we only look one step ahead.
Revised: V k − 1 , that is, the best policy if we only look one step ahead.

Original: This form will be conductive for our laterdevelopments.
Revised: This form will be conductive for our laterdevelopments.

Original: As the Bellman operator is a γ -contraction w.r.t.
Revised: As the Bellman operator is a γ -contraction w.r.t.

Original: the supremum norm, the convergencerate of V k to V π (or V ∗ ) would be O ( γ k ) .
Revised: the supremum norm, the convergencerate of V k to V π (or V ∗ ) would be O ( γ k ) .

Original: This  can be slow when γ is very close to 1 .
Revised: This rate can be slow when γ is very close to 1 .

Original: 2.2 Matrix splitting for solving linear systems of equations
Revised: 2.2 Matrix splitting for solving linear systems of equations

Original: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear systemof equations (2.2).
Revised: The VI for PE can be seen as a (matrix) splitting-based iterative method to solve the linear systemof equations (2.2).

Original: Consider the linear systems of equations Az = b , with A ∈ R d × d and z, b ∈ R d .
Revised: Consider the linear systems of equations Az = b , with A ∈ R d × d and z, b ∈ R d .

Original: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally,
Revised: Suppose that A is decomposed to A = M − N for some choices of M, N ∈ R d × d (more generally,

Original: A , M , and N can be linear operators).
Revised: A , M , and N can be linear operators).

Original: Therefore, z satisﬁes Mz =
Revised: Therefore, z satisfies Mz =

Original: Nz + b .
Revised: Nz + b .

Original: The splitting-basediterative approach deﬁnes the new approximation z k given the current z k − 1 by solving 
Revised: The splitting-basediterative approach defines the new approximation z k given the current z k − 1 by solving Mz [Equation] = Nz k

Original: [Equation]
Revised: [Equation]

Original:  or equivalently
Revised: + b, or equivalently

Original: [Equation]
Revised: [Equation]

Original: To analyze the convergence of this iterative method, consider the error e k (cid:44) z k − z .
Revised: To analyze the convergence of this iterative method, consider the error e k ≜ z k − z .

Original: As Mz =
Revised: As Mz =

Original: Nz + b ,the dynamics of the error is
Revised: Nz + b ,the dynamics of the error is

Original: [Equation]
Revised: [Equation]

Original: Let G (cid:44) M − 1 N .
Revised: Let G ≜ M − 1 N .

Original: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as
Revised: The norm of the sequence ( e k ) k ≥ 1 can be upper bounded as

Original: [Equation]
Revised: [Equation]

Original: The errors are (norm-) convergent if (cid:107) G (cid:107) =
Revised: The errors are (norm-) convergent if ∥ G ∥ =

Original: (cid:107) M − 1 N (cid:107) < 1 , for some choice of norm.
Revised: ∥ M − 1 N ∥ < 1 , for some choice of norm.

Original: More generally,the necessary and sufﬁcient condition for convergence is that the spectral radius ρ ( G ) , the maximumof absolute values of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or
Revised: More generally,the necessary and sufficient condition for convergence is that the spectral radius ρ ( G ) , the maximumof absolute values of eigenvalues of G , is smaller than one, see e.g., Theorem 4.1 of Saad [2003] or

Original: Theorem 11.2.1 of Golub and Van Loan [2013].
Revised: Theorem 11.2.1 of Golub and Van Loan [2013].

Original: The convergence is faster if the spectral radius (ornorm) is closer to zero.
Revised: The convergence is faster if the spectral radius (ornorm) is closer to zero.

Original: The success of this iterative procedure depends on how we choose M and N such that the norm (orspectral radius) is as small as possible.
Revised: The success of this iterative procedure depends on how we choose M and N such that the norm (orspectral radius) is as small as possible.

Original: Also we want to choose an M such that solving Mz k =
Revised: Also we want to choose an M such that solving Mz k =

Original: Nz k − 1 + b is not very expensive.
Revised: Nz k − 1 + b is not very expensive.

Original: For example, if M is an identity matrix I , we get that N
Revised: For example, if M is an identity matrix I , we get that N

Original: =
Revised: =

Original: I − A ,and the iteration becomes z k
Revised: I − A ,and the iteration becomes z k

Original: = ( I − A )
Revised: = ( I − A )

Original: z k − 1 + b .
Revised: z k − 1 + b .

Original: This iteration is convergent if ρ ( I − A ) < 1,which is the case if ρ ( A ) < 1 .
Revised: This iteration is convergent if ρ ( I − A ) < 1 .

Original: Other commonly used choices lead to the Jacobi and the Gauss-Seidelmethods, which are described in the supplementary material.
Revised: Othercommonly used choices lead to the Jacobi and the Gauss-Seidel methods, which are described in thesupplementary material.

Original: We are now ready to make the connection between splitting-based iterative methods to the VI for PE.
Revised: We are now ready to make the connection between splitting-based iterative methods to the VI for PE.

Original: If we choose A = I − γ P π , we see that the equation
Revised: If we choose A = I − γ P π , we see that the equation

Original: AV π = r π is indeed the Bellman equationfor policy π (2.2).
Revised: AV π = r π is indeed the Bellman equation forpolicy π (2.2).

Original: The VI for PE, which is V k = γ P π V k − 1 + r π =
Revised: The VI for PE, which is V k = γ P π V k − 1 + r π =

Original: ( I − A )
Revised: ( I − A )

Original: V k − 1 + r π , correspondsto the choice of M=
Revised: V k − 1 + r π , corresponds tothe choice of M

Original:  I and N =I − γ P π .
Revised: = I and N = γ P π .

Original: This brings up the question of whether it is possibleto split A to  other choices of M and N so that the resulting VI-like procedure has better improvedconvergence properties?
Revised: This brings up the question of whether it is possible to split Ato other choices of M and N so that the resulting VI-like procedure has better improved convergenceproperties?

Original: We suggest a particular choice in the next section.
Revised: We suggest a particular choice in the next section.

Original:  Operator splitting value iteration algorithm
Revised:  Operator splitting value iteration algorithm

Original: This sections introduces the Operator Splitting Value Iteration (OS-VI) algorithm.
Revised: This section introduces the Operator Splitting Value Iteration (OS-VI) algorithm.

Original: We start from the
Revised: We start from the

Original: PE problem, and introduce the Control version based on that.
Revised: PE problem, and introduce the Control version based on that.

Original: 3.1 OS-VI for policy evaluation
Revised: 3.1 OS-VI for policy evaluation

Original: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as
Revised: Given a policy π , true model P , and approximate model ˆ P , we split I − γ P π to M π and N π as

Original: [Equation]
Revised: [Equation]

Original: Following the recipe of (2.4), the OS-VI algorithm for PE would be
Revised: Following the recipe of (2.4), the OS-VI algorithm for PE would be

Original: [Equation]
Revised: [Equation]

Original: starting from an initial V 1 .
Revised: starting from an initial V 0 .

Original:  To gain more intuition and prepare for further developments, we deﬁne a few notations.
Revised: 3 To gain more intuition and prepare for further developments, we definea few notations.

Original: We deﬁnethe Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga (1928 – 2022) who hasmade signiﬁcant contributions to matrix analysis, as the mapping between the space of all boundedfunctions over X to the same space as
Revised: We define the Varga operator S π : B ( X ) → B ( X ) , named after Richard S Varga(1928 – 2022) who has made significant contributions to matrix analysis, as the mapping between thespace of all bounded functions over X to the same space as

Original: [Equation]
Revised: [Equation]

Original: Observe that (3.1) can be compactly written as
Revised: Observe that (3.1) can be compactly written as

Original: [Equation]
Revised: [Equation]

Original: It is not difﬁcult to see that S
Revised: It is not difficult to see that S

Original: π V π = V π , i.e., the value function of a policy π is a ﬁxed-point of the
Revised: π V π = V π , i.e., the value function of a policy π is a fixed-point of the

Original: Varga operator S π .
Revised: Varga operator S π .

Original: This and other properties of the Varga operator are in the supplementary material.
Revised: This and other properties of the Varga operator are in the supplementary material.

Original: Given any value function V , deﬁne an auxiliary reward function ¯ r V : X ×
Revised: Given any value function V , define an auxiliary reward function ¯ r V : X ×

Original: A → R as
Revised: A → R as

Original: [Equation]
Revised: [Equation]

Original: Similar to the notation for r π , we deﬁne ¯ r πV :
Revised: Similar to the notation for r π , we define ¯ r πV :

Original: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministicpolicy π (and similarly for a stochastic policy).
Revised: X → R as ¯ r πV ( x ) = ¯ r V ( x, π ( x )) for a deterministicpolicy π (and similarly for a stochastic policy).

Original: With this notation,
Revised: With this notation,

Original: [Equation]
Revised: [Equation]

Original: This is the value function of following π in an MDP with the dynamics ˆ P and the reward ¯ r V .
Revised: This is the value function of following π in an MDP with the dynamics ˆ P and the reward ¯ r V .

Original: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximatedynamics, and a reward function that consists of the original reward r and the correction termγ ( P − ˆ P ) V k  .
Revised: Therefore, at each iteration of OS-VI (PE), we evaluate the policy π according to the approximatedynamics, and a reward function that consists of the original reward r and the correction termγ ( P − ˆ P ) V k − 1 .

Original: The computation of this value function is a standard policy evaluation problem withthe approximate model.
Revised: The computation of this value function is a standard policy evaluation problemwith the approximate model.

Original: For instance, we may use another VI (PE) with dynamics ˆ P to solveit: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P
Revised: For instance, we may use another VI (PE) with dynamics ˆ P to solveit: We initialize U 0 ← V , and then for i ≥ 1 , we set U i ← ¯ r πV + γ ˆ P

Original: π
Revised: π

Original: U i − 1 .
Revised: U i − 1 .

Original: This convergesto S π V
Revised: This convergesto S π V

Original: =
Revised: =

Original: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .
Revised: ( I − γ ˆ P π ) − 1 ¯ r πV with the usual rate of O ( γ i ) .

Original: Note that aside the computation of ¯ r πV ,which requires querying P in order to compute the P π V term, this iterative process only uses theapproximate model ˆ P , which is assumed to be cheap to access.
Revised: Note that aside the computation of ¯ r πV ,which requires querying P in order to compute the P π V term, this iterative process only uses theapproximate model ˆ P , which is assumed to be cheap to access.

Original: What is the beneﬁt of this OS-VI procedure?
Revised: What is the benefit of this OS-VI procedure?

Original: If the approximate model ˆ P is close to the true dynamics
Revised: If the approximate model ˆ P is close to the true dynamics

Original: P , this leads to a faster convergence of V k to V π , as shall be quantiﬁed soon.
Revised: P , this leads to a faster convergence of V k to V π , as shall be quantified soon.

Original: The faster convergenceis in terms of the number of queries to P , which is assumed to bethe expensive.
Revised: The faster convergenceis in terms of the number of queries to P , which is assumed to be expensive.

Original: To see this, considerthe hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happensto match the reality perfectly.
Revised: To see this, considerthe hypothetical case that ˆ P is exactly the same as P , for example, if the cheap simulator happensto match the reality perfectly.

Original: Then, S π V
Revised: Then, S π V

Original: =
Revised: =

Original: ( I − γ P π ) − 1 ( r π + 0 V ) =
Revised: ( I − γ P π ) − 1 ( r π + 0 V ) =

Original: V π , and the value functionfor the original MDP is obtained in one iteration of OS-VI.
Revised: V π , and the value functionfor the original MDP is obtained in one iteration of OS-VI.

Original: Of course, we often can only hope forˆ P ≈ P .
Revised: Of course, we often can only hope forˆ P ≈ P .

Original: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in moredetails, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ Pis only a close approximation of P .
Revised: In Section 4, we study the impact of error in ˆ P on the convergence rate of OS-VI in moredetails, and show that the convergence of OS-VI can be much faster than classic algorithms even if ˆ Pis only a close approximation of P .

Original: 3.2 OS-VI for control
Revised: 3.2 OS-VI for control

Original: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ←π g ( V k − 1 )
Revised: The VI for Control can be seen as an iterative procedure that computes the greedy policy π k ←π g ( V k − 1 )

Original: =
Revised: =

Original: argmax π T π V k − 1 in its policy improvement step, and uses one step of the Bellmanoperator w.r.t.
Revised: argmax π T π V k − 1 in its policy improvement step, and uses one step of the Bellmanoperator w.r.t.

Original: the obtained policy π k to compute the new estimate of the value function V k ←
Revised: the obtained policy π k to compute the new estimate of the value function V k ←

Original: T π k V k − 1 , as described after (2.3).
Revised: T π k V k − 1 , as described after (2.3).

Original: The OS-VI algorithm for Control follows a similar structure withthe difference that (1) the improved policy is the optimizer of the Varga operator, and not the Bellmanoperator, and (2) the new value function is obtained by applying the Varga operator of the newlyobtained policy.
Revised: The OS-VI algorithm for Control follows a similar structure withthe difference that (1) the improved policy is the optimizer of the Varga operator, and not the Bellmanoperator, and (2) the new value function is obtained by applying the Varga operator of the newlyobtained policy.

Original: To be concrete, given a value function V , deﬁne the S -improved policy
Revised: To be concrete, given a value function V , define the S -improved policy

Original: [Equation]
Revised: [Equation]

Original: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .
Revised: This policy is the optimal policy for the auxiliary MDP ( X , A , ¯ r V , ˆ P , γ ) .

Original: We also deﬁne the Vargaoptimal operator S ∗ : B ( X ) → B ( X ) as
Revised: We also define the Vargaoptimal operator S ∗ : B ( X ) → B ( X ) as

Original: [Equation]
Revised: [Equation]

Original: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.
Revised: The function S ∗ V is equal to S π V ( V ) V , i.e., the Varga operator of the S -improved policy w.r.t.

Original: Vapplied to a value function V (compare it with T ∗ V
Revised: Vapplied to a value function V (compare it with T ∗ V

Original: = T π g ( V ) V ).
Revised: = T π g ( V ) V ).

Original: The OS-VI (Control) is then simply
Revised: The OS-VI (Control) is then simply

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which in its expanded form, consists of the following two steps:
Revised: which in its expanded form, consists of the following two steps:

Original: [Equation]
Revised: [Equation]

Original: Comparing thethe S -improved policy (3.4) used in the policy improvement step (3.6) of OS-VIwith the conventional greedy policy is insightful.
Revised: Comparing the S -improved policy (3.4) used in the policy improvement step (3.6) of OS-VI withthe conventional greedy policy is insightful.

Original: The greedy policy is argmax π T π V .
Revised: The greedy policy is argmax π T π V .

Original: Expanding T π V , we see that the greedy policy is the maximizer of r π + γ P π V .
Revised: Expanding T π V ,we see that the greedy policy is the maximizer of r π + γ P π V .

Original: The function r π + γ P π V isonly a single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy,is in general different from the optimal policy, which maximizes the return.
Revised: The function r π + γ P π V is onlya single-step bootstrapped estimate of the value of V π , and its maximizer, the greedy policy, is ingeneral different from the optimal policy, which maximizes V π .

Original: On the other hand, the S -improved policy π V ( V ) solves a full MDP with an approximate model ˆ P and the reward functionthat has both the original reward r and the correction term NV .
Revised: On the other hand, the S -improvedpolicy π V ( V ) solves a full MDP with an approximate model ˆ P and the reward function that hasboth the original reward r and the correction term γ ( P − ˆ P ) V .

Original: In the special case that ˆ P = P ,the correction term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.
Revised: In the special case that ˆ P = P , thecorrection term is zero, and π V ( V ) would be the optimal policy π ∗ for the original MDP.

Original: Asoften ˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.
Revised: As oftenˆ P ≈ P , the value function of policy π V ( V ) is not exactly the optimal value.

Original: The partial policyevaluation step (3.7) updates the value function to a value that is closer to the optimal value function.
Revised: The partial policyevaluation step (3.7) updates the value function to a value that is closer to the optimal value function.

Original: Remark.
Revised: Remark.

Original: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context ofdynamic programming is not completely novel to this work.
Revised: The use of matrix splitting-based ideas, either explicitly or implicitly, in the context ofdynamic programming is not completely novel to this work.

Original: Kushner and Kleinman [1971] is oneof the earliest paper that mention the Jacobi and Gauss-Seidel procedures for computing the valuefunction.
Revised: Kushner and Kleinman [1971] is oneof the earliest paper that mention the Jacobi and Gauss-Seidel procedures for computing the valuefunction.

Original: Porteus [1975] proposes several transformations to the reward and the probability transitionmatrix with the goal of improving the computational cost of solving the transformed MDP.
Revised: Porteus [1975] proposes several transformations to the reward and the probability transitionmatrix with the goal of improving the computational cost of solving the transformed MDP.

Original: One ofthe transformations, called pre-inverse transform , has some similarities with the operator splitting ofthis work.
Revised: One ofthe transformations, called pre-inverse transform , has some similarities with the operator splitting ofthis work.

Original: The end result, however, is different.
Revised: The end result, however, is different.

Original: Bacon and Precup [2016] provide a matrix splittingperspective on planning with options.
Revised: Bacon and Precup [2016] provide a matrix splittingperspective on planning with options.

Original: The connection between multi-step models and matrix splittingis developed in Chapter 4 of Bacon [2018].
Revised: The connection between multi-step models and matrix splittingis developed in Chapter 4 of Bacon [2018].

Original: Refer to the supplementary material for more discussion.
Revised: Refer to the supplementary material for more discussion.

Original:  Convergence analysis of operator splitting value iteration
Revised:  Convergence analysis of operator splitting value iteration

Original: In this section, we present the convergence analysis of OS-VI.
Revised: In this section, we present the convergence analysis of OS-VI.

Original: Our results show that OS-VI has an
Revised: Our results show that OS-VI has an

Original: O ( γ (cid:48) k ) convergence rate for an effective discount factor γ (cid:48) that depends on the error between ˆ P and
Revised: O ( γ ′ k ) convergence rate for an effective discount factor γ ′ that depends on the error between ˆ P and

Original: P .
Revised: P .

Original: For small enough error, γ (cid:48) < γ and OS-VI has a faster convergence rate compared to the classic
Revised: For small enough error, γ ′ < γ and OS-VI has a faster convergence rate compared to the classic

Original: VI, Policy Iteration (PI), and Modiﬁed Policy Iteration (MPI), which all have O ( γ k ) behaviour.
Revised: VI, Policy Iteration (PI), and Modified Policy Iteration (MPI), which all have O ( γ k ) behaviour.

Original: Weprovide results for both the L ∞ and L p norms.
Revised: Weprovide results for both the L ∞ and L p norms.

Original: 4.1 Convergence of OS-VI for policy evaluation
Revised: 4.1 Convergence of OS-VI for policy evaluation

Original: We study convergence behaviour of OS-VI (PE) in presence of error in value updates.
Revised: We study convergence behaviour of OS-VI (PE) in presence of error in value updates.

Original: Morespeciﬁcally, we consider the setting that in each iteration k , the update (3.2) has an error, i.e.,
Revised: Morespecifically, we consider the setting that in each iteration k , the update (3.2) has an error, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The error (cid:15) value k encompasses various sources of errors that might occur in a practical application of
Revised: The error ϵ value k encompasses various sources of errors that might occur in a practical application of

Original: OS-VI.
Revised: OS-VI.

Original: This includes the function approximation error, caused by using a function approximatorto represent V k , and the estimation (i.e., statistical) error caused due to having a ﬁnite number ofsamples, instead of direct access to P , in the RL setting, see [Györﬁ et al., 2002] for a generaldiscussion of function approximation/estimation error in the supervised learning, and [Munos and
Revised: This includes the function approximation error, caused by using a function approximatorto represent V k , and the estimation (i.e., statistical) error caused due to having a finite number ofsamples, instead of direct access to P , in the RL setting, see [Györfi et al., 2002] for a generaldiscussion of function approximation/estimation error in the supervised learning, and [Munos and

Original: Szepesvári, 2008, Antos et al., 2008, Farahmand et al., 2016, Chen and Jiang, 2019].
Revised: Szepesvári, 2008, Antos et al., 2008, Farahmand et al., 2016, Chen and Jiang, 2019].

Original: In this work, we do not analyze how the number of samples, the function approximator, etc. affect the errors(cid:15) value k .
Revised: In this work,we do not analyze how the number of samples, the function approximator, etc. affect the errors ϵ value k .

Original: The result of this section provides an error propagation result similar to Munos [2007] (forapproximate VI), Munos [2003] (for approximate PI), and Scherrer et al.
Revised: The result of this section provides an error propagation result similar to Munos [2007] (forapproximate VI), Munos [2003] (for approximate PI), and Scherrer et al.

Original: [2015] (for approximatemodiﬁed PI).
Revised: [2015] (for approximatemodified PI).

Original: To study convergence behaviour of OS-VI (PE), let G π
Revised: To study convergence behaviour of OS-VI (PE), let G π

Original: = ( I − γ ˆ P π ) − 1 γ
Revised: = ( I − γ ˆ P π ) − 1 γ

Original: ( P π − ˆ P π ) .
Revised: ( P π − ˆ P π ) .

Original: We use thefact that S
Revised: We use thefact that S

Original: π V π = V π
Revised: π V π = V π

Original: and write
Revised: and write

Original: [Equation]
Revised: [Equation]

Original: Now, we have that
Revised: Now, we have that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that for any square matrix F with a matrix norm (cid:107) F (cid:107) p < 1 , it holdsthat (cid:13)(cid:13)

Original: ( I

Original: [Equation]

Original: F )

Original: [Equation]

Original:  1 (cid:13)(cid:13) p ≤ [Equation] F (cid:107) p (e.g., Lemma 2.3.3 of Golub and Van Loan 2013), and that thesupremum norm of a stochastic matrix ˆ P π is 1 .Assuming that (cid:107) (cid:15) value k (cid:107) ∞ ≤ (cid:15) value for all k ≥ 1 anddeﬁning effective discount factor γ (cid:48)
Revised: where we used the fact that for any square matrix F with a matrix norm ∥ F ∥ p < 1 , it holdsthat ∥ ( I − F ) − 1 ∥ p ≤ 1 1 −∥ F ∥ p (e.g., Lemma 2.3.3 of Golub and Van Loan 2013), and that thesupremum norm of a stochastic matrix ˆ P π is 1 .

Original: =

Original:  γ 1 − γ (cid:107)P π − ˆ P π (cid:107) ∞ , the upper bounds (4.3) and (4.2) lead [-to(cid:107) V π − V k (cid:107) ∞ ≤ γ (cid:48) k (cid:107) V π − V 
 (cid:107) ∞ + 1 −-]
Revised: Assuming that ∥ ϵ value k ∥ ∞ ≤ ϵ value for all k ≥ 1 anddefining effective discount factor γ ′ = γ 1 − γ ∥P π − ˆ P π ∥ ∞ , the upper bounds (4.3) and (4.2) lead [-to(cid:107) V π − V k (cid:107) ∞ ≤ γ (cid:48) k (cid:107) V π − V 
 (cid:107) ∞ + 1 −-]to

Original: [Equation]
Revised: [Equation]

Original: (cid:15) value .

Original: A few remarks are in order.
Revised: A few remarks are in order.

Original: First, whenever γ (cid:48) < γ , this is guaranteed to be faster than the convergencerate of the conventional VI, which is O ( γ k ) .
Revised: First, whenever γ ′ < γ , this is guaranteed to be faster than the convergencerate of the conventional VI, which is O ( γ k ) .

Original: This is the case if (cid:107)P π − ˆ P π (cid:107) ∞ < 1 − γ .
Revised: This is the case if ∥P π − ˆ P π ∥ ∞ < 1 − γ .

Original: If the modelis very accurate, we obtain much faster rate than VI’s.
Revised: If the modelis very accurate, we obtain much faster rate than VI’s.

Original: Since each iteration k corresponds to a queryto the true model P , a faster rate entails that the algorithm requires fewer total number of queries tothe expensive model to reach the same level of accuracy.
Revised: Since each iteration k corresponds to a queryto the true model P , a faster rate entails that the algorithm requires fewer total number of queries tothe expensive model to reach the same level of accuracy.

Original: Second, although the model error (cid:107)P π − ˆ P π (cid:107) ∞ is a reasonable choice to measure the distancesbetween distributions (it is the maximumover states of the Total Variation distance between P π ( ·| x )and ˆ P π ( ·| x ) , which itself can be upper bounded by their KL divergence; see the supplementarymaterial), it is somewhat conservative as its takes the supremum over the state space.
Revised: Second, although the model error ∥P π − ˆ P π ∥ ∞ is a reasonable choice to measure the distancesbetween distributions (it is the maximum of the Total Variation distance between P π ( ·| x ) and ˆ P π ( ·| x )over x , which itself can be upper bounded by their KL divergence; see the supplementary material), itis somewhat conservative as it takes the supremum over the state space.

Original: Likewise,requiring (cid:107) (cid:15) value k (cid:107) ∞  to be small is also conservative, as approximating S π V k − 1 using a functionapproximator given samples (RL setting) often lead to an L p -norm type of guarantee.
Revised: Likewise, requiring ∥ ϵ value k ∥ ∞ 210 to be small is also conservative, as approximating S π V k − 1 using a function approximator givensamples (RL setting) often leads to an L p -norm type of guarantee.

Original: We nowprovide a different analysis to address these issues.
Revised: We now provide a differentanalysis to address these issues.

Original: To present the L p norm result, we need to deﬁne some notations.
Revised: To present the L p norm result, we need to define some notations.

Original: First, we deﬁne the conditionaldiscounted future-state distribution of policy π under ˆ P as the following probability distribution:
Revised: First, we define the conditionaldiscounted future-state distribution of policy π under ˆ P as the following probability distribution:

Original: Given a measurable set B , we have ˆ η π ( B | x ) =
Revised: Given a measurable set B , we have ˆ η π ( B | x ) =

Original: (1 − γ ) (cid:80) ∞
Revised: (1 − γ ) (cid:80) ∞

Original: t =

Revised: t =

Original: γ t · P (cid:16) X t ∈ B | X 

Revised: γ t · P (cid:16) X t ∈ B | X

Original: = x, π, ˆ P (cid:17) , wherethe chain ( X t ) t ≥ 0 starts from state x , and evolves by following policy π under ˆ P transitions.
Revised: = x, π, ˆ P (cid:17) , wherethe chain ( X t ) t ≥ 0 starts from state x , and evolves by following policy π under ˆ P transitions.

Original: This isthe same as (2.1) with the chance of P to ˆ P .
Revised: This isthe same as (2.1) with the change of P to ˆ P .

Original: For an arbitrary distribution ρ over the state space, wedeﬁne the discounted future-state distribution concentration coefﬁcient as
Revised: For an arbitrary distribution ρ over the state space, wedefine the discounted future-state distribution concentration coefficient as

Original: [Equation]
Revised: [Equation]

Original: Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.
Revised: Here dˆ η π ( ·| x ) d ρ is the Radon–Nikodym derivative of the distribution ˆ η π ( ·| x ) w.r.t.

Original: the distributionρ .
Revised: the distribution ρ ,and it is assumed that for any x ∈ X , ˆ η π ( ·| x ) ≪ ρ , i.e., ˆ η π ( ·| x ) is absolutely continuous w.r.t. ρ(otherwise, the coefficient would be set to infinity).

Original: This coefﬁcient measures how concentrated the distribution ˆ η π ( ·| x ) is compared to ρ .
Revised: This coefficient measures how concentrated thedistribution ˆ η π ( ·| x ) is compared to ρ .

Original: Thisis weighted according to the initial state distribution ρ .
Revised: This is weighted according to the initial state distribution ρ .

Original: Similar concentrability coefﬁcients, butnot exactly this one, have appeared in the error propagation results [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al., 2015].
Revised: Similar concentrability coefficients, but not exactly this one, have appeared in the error propagationresults [Kakade and Langford, 2002, Munos, 2003, 2007, Farahmand et al., 2010, Scherrer et al.

Original: Finally, we deﬁne the weightedχ 2 -divergence of ˆ P π and P π as the following:
Revised: Finally, we define the weighted χ 2 -divergence of ˆ P π and P π as the following:

Original: [Equation]
Revised: [Equation]

Original: This notion of model error is less strict in requiring accurate approximation P in all states.
Revised: This notion of model error is less strict in requiring accurate approximation P in all states.

Original: Usuallyonly a subset of state space is important or even reachable in a problem.
Revised: Usuallyonly a subset of state space is important or even reachable in a problem.

Original: The above model error canonly focus on speciﬁc areas of state space through the choice of distribution ρ .
Revised: The above model error canonly focus on specific areas of state space through the choice of distribution ρ .

Original: We are now ready to to provide the main theorem for the approximate OS-VI (PE).
Revised: We are now ready to to provide the main theorem for the approximate OS [Table]

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider the approximate OS-VI algorithm for PE (4.1) .
Revised: Consider the approximate OS-VI algorithm for PE (4.1) .

Original: Let (cid:107)·(cid:107) (cid:63) be either the supremumnorm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over state space ( (cid:63) = 4 , ρ ).
Revised: Let ∥·∥ ⋆ be either the supremumnorm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over state space ( ⋆ = 4 , ρ ).

Original: Assume that (cid:107) (cid:15) value k (cid:107) (cid:63) ≤ (cid:15) value for all k ≥ 1 .
Revised: Assume that ∥ ϵ value k ∥ ⋆ ≤ ϵ value for all k ≥ 1 .

Original: Furthermore, deﬁne the effective discount factor as
Revised: Furthermore, define the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: For any k ≥ 0 , we have
Revised: For any k ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: 4.2 Convergence of OS-VI for control
Revised: 4.2 Convergence of OS-VI for control

Original: Now we turn to analyzing OS-VI for Control.
Revised: We turn to analyzing OS-VI for Control.

Original: Similar to the analysis of OS-VI for PE, we assumesome error in each step of OS-VI.
Revised: Similar to the analysis of OS-VI for PE, we assume someerror in each step of OS-VI.

Original: Each iteration of OS-VI for PE was solving the PE problem inthe auxiliary MDP
Revised: Each iteration of OS-VI for PE was solving the PE problem in theauxiliary MDP

Original: ( X , A , ¯ r V , ˆ P , γ ) .
Revised: ( X , A , ¯ r V , ˆ P , γ ) .

Original: The error of this solution was the error of the calculated valuefunction V k +1 w.r.t.
Revised: The error of this solution was the error of the calculated valuefunction V k +1 w.r.t.

Original: the true value function in the MDP S π V k leading to (4.1).
Revised: the true value function in the MDP S π V k leading to (4.1).

Original: In control, eachiteration is solving the control problem in the same auxiliary MDP.
Revised: In Control, eachiteration is solving the control problem in the same auxiliary MDP.

Original: The difference is that we have twotypes of error.
Revised: The difference is that we have twotypes of error.

Original: First, the error of value function w.r.t.
Revised: First, the error of value function w.r.t.

Original: true solution, i.e., V k − S ∗ V k − 1 .
Revised: true solution, i.e., V k − S ∗ V k − 1 .

Original: Second is thesuboptimality of policy compared to the optimal policy of auxiliary MDP, i.e., S π k [Equation]
Revised: Second is thesuboptimality of policy compared to the optimal policy of auxiliary MDP, i.e., S π k V k − 1 − S ∗ V k − 1 .

Original: More speciﬁcally
Revised: More specifically

Original: [Equation]
Revised: [Equation]

Original: We can now present the results for Control.

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .
Revised: Consider the approximate OS-VI algorithm for control (4.5) - (4.6) .

Original: Let (cid:107)·(cid:107) (cid:63) be eitherthe supremum norm (cid:107)·(cid:107) ∞ ( (cid:63) = ∞ ) or (cid:107)·(cid:107) 4 ,ρ for ρ being an arbitrary distribution over state space( (cid:63) = 4 , ρ ).
Revised: Let ∥·∥ ⋆ be eitherthe supremum norm ∥·∥ ∞ ( ⋆ = ∞ ) or ∥·∥ 4 ,ρ for ρ being an arbitrary distribution over statespace ( ⋆ = 4 , ρ ).

Original: For any k ≥ 1 , let Π k
Revised: For any k ≥ 1 , let Π k

Original: =
Revised: =

Original: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 0 ≤ i < k } .
Revised: { π ∗ , π k } ∪ { π V ( V i − 1 ) : 0 ≤ i < k } .

Original: Assume that (cid:107) (cid:15) value k (cid:107) (cid:63) ≤ (cid:15) value for all k ≥ 1 .
Revised: Assume that∥ ϵ value k ∥ ⋆ ≤ ϵ value for all k ≥ 1 .

Original: Furthermore, deﬁne the effective discount factor asγ (cid:48) = γ
Revised: Furthermore, define the effective discount factor as

Original: [Equation]
Revised: [Equation]

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: (cid:63) + 2 γ (cid:48) (1 − γ (cid:48)

Original: k − 1 ) (1 − γ (cid:48) )· (cid:15) value + 1 − γ (cid:48) · (cid:13)(cid:13)(cid:13) (cid:15) policy k (cid:13)(cid:13)(cid:13) (cid:63) .

Original: We can compare this result with the convergence result of VI.
Revised: We can compare this result with the convergence result of VI.

Original: For VI with the supremum norm,following the proof of Equation (2.2) by Munos [2007], we can show that (cid:107) V ∗ − V π k (cid:107) ∞ 
Revised: For VI with the supremum norm,following the proof of Equation (2.2) by Munos [2007], we can show that ∥ V ∗ − V π k ∥ ∞ ≤

Original: [-≤2 γ k 1 − γ (cid:107) V ∗ − V 
 (cid:107) ∞ + 2 γ (1 − γ k − 1 ) ε (1 − γ ), with (cid:107) V-]
Revised: [-≤2 γ k 1 − γ (cid:107) V ∗ − V 
 (cid:107) ∞ + 2 γ (1 − γ k − 1 ) ε (1 − γ ), with (cid:107) V-][Equation]

Original:  i − T ∗ V i − 1 (cid:107) ∞ ≤ ε for all i < k (similar result for the
Revised: , with ∥ V i − T ∗ V i − 1 ∥ ∞ ≤ ε for all i < k (similar result for the

Original: L p -norm also holds, see Theorem 5.2 of Munos 2007).
Revised: L p -norm also holds, see Theorem 5.2 of Munos 2007).

Original: For the approximate VI, the initial error(cid:107) V ∗ − V 0 (cid:107){+
 ∥+} ∞ decays with the rate of O ( γ k ) .
Revised: For the approximate VI, the initial error∥ V ∗ − V {+
 ∥+} ∞ decays with the rate of O ( γ k ) .

Original: This should be compared with O ( γ (cid:48) k ) rate of OS-VI.
Revised: This should be compared with O ( γ ′ k ) rate of OS-VI.

Original: The effect of error at each step (cid:15) value is also similar: approximate VI has (1 − γ ) − 2 dependence whileapproximate OS-VI has (1 − γ (cid:48) ) − 2 .
Revised: The effect of error at each step ϵ value is also similar: approximate VI has (1 − γ ) − 2 dependence whileapproximate OS-VI has (1 − γ ′ ) − 2 .

Original: What is remarkable is that as opposed to γ , which is a ﬁxedparameter of the problem and can be close to 1 , γ (cid:48) can be made arbitrary close to zero when themodel is accurate.
Revised: What is remarkable is that as opposed to γ , which is a fixedparameter of the problem and can be close to 1 , γ ′ can be made arbitrary close to zero when themodel is accurate.

Original: The additional information given by ˆ P allows us to get much faster rate than VI.
Revised: The additional information given by ˆ P allows us to get much faster rate than VI.

Original: Of course, this requires the model to be accurate.
Revised: Of course, this requires the model to be accurate.

Original: An inaccurate model might be detrimental to theconvergence rate, and may even lead to divergence.
Revised: An inaccurate model might be detrimental to theconvergence rate, and may even lead to divergence.

Original:  Operator splitting Dyna
Revised:  Operator splitting Dyna

Original: In this section, we generalize the operator splitting approach to an RL problem in which only samplesfrom P are available.
Revised: In this section, we generalize the operator splitting approach to an RL problem in which only samplesfrom P are available.

Original: In this scenario, the approximate model ˆ P may not be given, as assumed in previous sections.
Revised: In this scenario, the approximate model ˆ P may not be given, as assumedin previous sections.

Original: However, one can learn such an approximate model using samples from theenvironment as done in MBRL algorithms.
Revised: However, one can learn such an approximate model using samples from theenvironment as done in MBRL algorithms.

Original: Unlike MBRL approaches that solely rely on the model,our approach takes advantage of both the true environment and the model in its updates.
Revised: Unlike MBRL approaches that solely rely on the model,our approach takes advantage of both the true environment and the model in its updates.

Original: Therefore,one can think of the operator splitting approach as a hybrid of MBRL and model-free RL.
Revised: Therefore,one can think of the operator splitting approach as a hybrid of MBRL and model-free RL.

Original: Using a learned ˆ P , we can calculate V k from the auxiliary reward function ¯ r k (cid:44) ¯ r V k − 1 by solving
Revised: Using a learned ˆ P , we can calculate V k from the auxiliary reward function ¯ r k ≜ ¯ r V k − 1 by solving

Original: PE or control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.
Revised: PE or control problem in the auxiliary MDP ( X , A , ¯ r k , ˆ P ) , as discussed in Section 3.

Original: A possiblechallenge of using a learned model is that in many cases, we can only take samples from it and donot have access to the full matrices in calculations.
Revised: A possiblechallenge of using a learned model is that in many cases, we can only take samples from it and donot have access to the full matrices in calculations.

Original: Fortunately, solving PE or control problems withsuch a model is a very common practice in RL and is also done in Dyna architecture.
Revised: Fortunately, solving PE or control problems withsuch a model is a very common practice in RL and is also done in Dyna architecture.

Original: Therefore, wedo not focus on this procedure.
Revised: Therefore, wedo not focus on this procedure.

Original: The ability of calculating V k from ¯ r k lets us see the value function as a function of ¯ r k .
Revised: The ability of calculating V k from ¯ r k lets us see the value function as a function of ¯ r k .

Original: Based onthis view, we shift our attention to ¯ r k .
Revised: Based onthis view, we shift our attention to ¯ r k .

Original: Note that since there is a ﬁxed connection between ¯ r k and
Revised: Note that since there is a fixed connection between ¯ r k and

Original: V k as the solution of PE or control in ( X , A , ¯ r k , ˆ P ) , the convergence of ¯ r k and V k are tied together.
Revised: V k as the solution of PE or control in ( X , A , ¯ r k , ˆ P ) , the convergence of ¯ r k and V k are tied together.

Original: Therefore, focusing on updating ¯ r k will have the same convergence properties.
Revised: Therefore, focusing on updating ¯ r k will have the same convergence properties.

Original: The following updaterules can be written for ¯ r k itself.
Revised: The following updaterules can be written for ¯ r k itself.

Original: For every  x, a [Equation]
Revised: For every ( x, a ) ,

Original: Policy Evaluation

Original: [Equation]
Revised: [Equation]

Original: In OS-Dyna, we maintain a vector (or a function approximation) for ¯ r , and as we will show how,update it by samples.
Revised: In OS-Dyna, we maintain a vector (or a function approximation) for ¯ r , and as we will show how,update it by samples.

Original: The value function is updated to V π (¯ r, ˆ P ) (for PE) or V ∗ (¯ r, ˆ P ) (for control)with each update of ¯ r .
Revised: The value function is updated to V π (¯ r, ˆ P ) (for PE) or V ∗ (¯ r, ˆ P ) (for control)with each update of ¯ r .

Original: This way, we have the value functions in update rules (5.1) and (5.2).
Revised: This way, we have the value functions in update rules (5.1) and (5.2).

Original: Theonly challenge is that the above update rules need access to distributions P ( ·| x, a ) and ˆ P ( ·| x, a ) forevery x, a  , while we only have samples from these distributions in some  x, a  pairs.
Revised: The onlychallenge is that the above update rules need access to distributions P ( ·| x, a ) and ˆ P ( ·| x, a ) for every( x, a ) , while we only have samples from these distributions in some ( x, a ) pairs.

Original: Fortunately, thischallenge has been tackled in developing sample-based algorithms based on classic value iteration:
Revised: Fortunately, thischallenge has been tackled in developing sample-based algorithms based on classic value iteration:

Original: [Equation]
Revised: [Equation]

Original: where V k − 1 =
Revised: where V k − 1 =

Original: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a (cid:48) Q k − 1 ( x, a ) in control.
Revised: Q k − 1 ( x, π ( x )) in PE and V k − 1 = max a ′ Q k − 1 ( x, a ) in Control.

Original: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Q-Iterationand Stochastic Approximation (SA).
Revised: There are multiple approaches to develop sample-based algorithms based on (5.3) such as Fitted Q-Iterationand Stochastic Approximation (SA).

Original: In this paper we use SA to develop OS-Dyna, but we pointout that other algorithms and techniques can also be applied to develop other versions of OSDyna.
Revised: In this paper we use SA to develop OS-Dyna, but we pointout that other algorithms and techniques can also be applied to develop other versions of OSDyna.

Original: The key step in SA is to use samples to ﬁnd an unbiased estimate of the intended updatevalue.
Revised: The key step in SA is to use samples to find an unbiased estimate of the intended updatevalue.

Original: For a step ( X t , A t , R t , X (cid:48) t ) in the true environment, we can have the following estimate
Revised: For a step ( X t , A t , R t , X ′ t ) in the true environment, we can have the following estimate

Original: Z = R t + γV
Revised: Z = R t + γV

Original: ( X (cid:48) t )
Revised: ( X ′ t )

Original: − γ E X (cid:48) ∼ ˆ P ( ·| X t ,A t )
Revised: − γ E X ′ ∼ ˆ P ( ·| X t ,A t )

Original: [ V ( X (cid:48) )] , where the expectation can also be estimated bysamples from ˆ P ( ·| X t , A t ) .
Revised: [ V ( X ′ )] , where the expectation can also be estimated bysamples from ˆ P ( ·| X t , A t ) .

Original: Finally, we make the following update to ¯ r with learning rate α t¯ 
Revised: Finally, we make the following update to ¯ r with learning rate α t¯ r

Original: [Equation]
Revised: ( X t , A t ) ← ¯ r ( X t , A t ) + α t · ( Z − ¯ r ( X t , A t )) .

Original: The ﬁnal procedure of OS-Dyna is presented in Algorithm 1.
Revised: The final procedure of OS-Dyna is presented in Algorithm 1.

Original:  Experiments
Revised:  Experiments

Original: We evaluate both OS-VI and OS-Dyna in a ﬁnite MDP, comparing our algorithms with existingmethods.
Revised: We evaluate both OS-VI and OS-Dyna in a finite MDP, comparing our algorithms with existingmethods.

Original: The MDP considered is a modiﬁed cliffwalk environment in a 6 × 6 grid with 4 actions (up,down, left and right).
Revised: The MDP considered is a modified cliffwalk environment in a 6 × 6 grid with 4 actions(up, down, left and right).

Original: Full details of the environment is available in the appendix.
Revised: Full details of the environment is available in the supplementary material.

Original: Our convergence[Figure] analysis shows that the convergence rates of our algorithms depend on the accuracy of ˆ P .
Revised: Our convergence analysis shows that the convergence rates of our algorithms depend on the accuracyof ˆ P .

Original: To test OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothed model ˆ P oftransitions P with smoothing parameter λ as
Revised: To test OS-VI and OS-Dyna with models of different accuracies, we introduce the smoothedmodel ˆ P of transitions P with smoothing parameter λ as

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: where U ( A ) for some set A is the uniform distribution over A .
Revised: where U ( A ) for some set A is the uniform distribution over A .

Original: Here, λ allows making adjustments tothe amount of error introduced in ˆ P w.r.t.
Revised: Here, λ allows making adjustments tothe amount of error introduced in ˆ P w.r.t.

Original: P .
Revised: P .

Original: If λ = 0 , ˆ P
Revised: If λ = 0 , ˆ P

Original: = P will be the accurate model, and ifλ = 1 , ˆ P will be uniform over possible next states in P .
Revised: = P will be the accurate model, and ifλ = 1 , ˆ P will be uniform over possible next states in P .

Original: We compare OS-VI and OS-Dyna for control with other existing methods.
Revised: We compare OS-VI and OS-Dyna for Control with other existing methods.

Original: The results for the PEproblem are qualitatively similar and are provided in the supplementary material along with moreexperiments.
Revised: The results for the PEproblem are qualitatively similar and are provided in the supplementary material along with moreexperiments.

Original: The left plot in Figure 1 shows the convergence of OS-VI compared to VI and thesolutions the model itself would lead to.
Revised: The left plot in Figure 1 shows the convergence of OS-VI compared to VI and thesolutions the model itself would lead to.

Original: The plot shows normalized error of V π k w.r.t V ∗ , i.e.(cid:107) V π k − V ∗ (cid:107) 1 / (cid:107) V ∗ (cid:107) 1 .
Revised: The plot shows normalized error of V π k w.r.t V ∗ , i.e.∥ V π k − V ∗ ∥ 1 / ∥ V ∗ ∥ 1 .

Original: It can be seen that OS-VI hasa faster convergence with more accurate modelsand introduces speedup compared to VI across different model errors.
Revised: It can be seen that OS-VI has faster convergence with more accurate modelsand introduces acceleration compared to VI across different model errors.

Original: Note that this convergencehas been achieved despite the error in the model.
Revised: Note that the convergenceof OS-VI has been achieved despite the error in the model.

Original: The dashed lines show how a fully model-basedalgorithm would obtain a suboptimal solution by only relying on the model.
Revised: The dashed lines show how a fullymodel-based algorithm would obtain a suboptimal solution by only relying on the model.

Original: We also compare OS-Dyna with with Dyna and QLearning in the RL setting.
Revised: We also compare OS-Dyna with with Dyna and Q-Learning in the RL setting.

Original: At each iteration t , thealgorithms are given a sample ( X t , A t , R t , X (cid:48) t ) where X t , A t are selected uniformly at random.
Revised: At each iteration t , thealgorithms are given a sample ( X t , A t , R t , X ′ t ) where X t , A t are selected uniformly at random.

Original: For
Revised: For

Original: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.
Revised: OS-Dyna and Dyna we use the smoothed Maximum-likelihood Estimation (MLE) model.

Original: If P MLEis the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ )deﬁned in (6.1) as their models.
Revised: If P MLEis the current MLE estimation of the environment transitions, OS-Dyna and Dyna use ˆ P ( P MLE , λ )defined in (6.1) as their models.

Original: The learning rates are constant α for iterations t ≤ N and then decayin the form of α t = α/ ( t − N ) afterwards.
Revised: The learning rates are constant α for iterations t ≤ N and then decayin the form of α t = α/ ( t − N ) afterwards.

Original: We have ﬁne-tuned the learning rate schedule for eachalgorithm separately for the best results.
Revised: We have fine-tuned the learning rate schedule for eachalgorithm separately for the best results.

Original: The right plot in Figure 1 shows the results for the RL setting.
Revised: The right plot in Figure 1 shows the results for the RL setting.

Original: We evaluate the expected return ofthe policy at iteration t in the initial state of the environment, i.e. V π k (0) .
Revised: We evaluate the expected return of thepolicy at iteration t in the initial state of the environment, i.e. V π k (0) .

Original: Again, OS-Dyna hasconverged to the optimal policy much faster than QLearning for all model errors.
Revised: Again, OS-Dyna has convergedto the optimal policy much faster than Q-Learning for all model errors.

Original: Unlike OS-Dyna,the classic Dyna has failed to ﬁnd the optimal policy in presence of model error.
Revised: Unlike OS-Dyna, classic Dyna has failed to find the optimal policy in presence of model error.

Original: The results showthat OS-Dyna can effectively converge faster than QLearning without introducing bias to the ﬁnalsolution due to model error.
Revised: The results show that OS-Dynacan effectively converge faster than Q-Learning without introducing bias to the final solution due tomodel error.

Original:  Conclusion
Revised:  Conclusion

Original: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can beneﬁtfrom an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to thetrue value function in terms of the number of queries to the true model P .
Revised: This paper introduced the Operator Splitting Value Iteration (OS-VI) algorithm, which can benefitfrom an approximate model ˆ P ≈ P to accelerate the convergence of the approximate value to thetrue value function in terms of the number of queries to the true model P .

Original: With a small modelerror, its convergence rate is exponentially faster compared to well-known dynamical programmingalgorithms such as Value Iteration and Policy Iteration.
Revised: With a small modelerror, its convergence rate is exponentially faster compared to well-known dynamical programmingalgorithms such as Value Iteration and Policy Iteration.

Original: We also proposed OS-Dyna as a hybridmodel-based/model-free algorithm that can bring in the beneﬁts of a model-based RL algorithmwithout converging to a biased solution, as Dyna or any other purely model-based RL algorithm does.
Revised: We also proposed OS-Dyna as a hybridmodel-based/model-free algorithm that can bring in the benefits of a model-based RL algorithmwithout converging to a biased solution, as Dyna or any other purely model-based RL algorithmdoes.

Revised: This paper opens up several future directions.

Revised: Empirically studying the algorithms on problemswith large state spaces, for which a function approximator such as a DNN is required, is an obviousone.

Revised: This is postponed to a future work as our aim was to build the mathematical foundation andconducting experiments without worrying about challenges such as the optimization of a DNN.

Revised: Thereare other algorithmic and theoretical directions to be pursued.

Revised: One is exploring the space of splittingsof I − γ P π .

Revised: The other is whether we can design Operator Splitting variants of other DP algorithmssuch as Policy Iteration and Modified Policy Iteration, and study their convergence behaviour.