Original: P ARAMETERIZED PROJECTED B ELLMAN OPERATOR
Revised: P ARAMETERIZED PROJECTED B ELLMAN OPERATOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The Bellman operator is a cornerstone of reinforcement learning, widely used in a plethora of works, from value-based methods to modern actor-critic approaches.
Revised: The Bellman operator is a cornerstone of reinforcement learning, widely used in a plethora of works, from value-based methods to modern actor-critic approaches.

Original: In problems with unknown models, the Bellman operator requires transition samples that strongly determine its behavior, as uninformative samples can result in negligible updates or long detours before reaching the fixed point.
Revised: In problems with unknown models, the Bellman operator requires transition samples that strongly determine its behavior, as uninformative samples can result in negligible updates or long detours before reaching the fixed point.

Original: In this work, we introduce the novel idea of obtaining an approximation of the Bellman operator, which we call projected Bellman operator (PBO).
Revised: In this work, we introduce the novel idea of obtaining an approximation of the Bellman operator, which we call projected Bellman operator (PBO).

Original: Our PBO is a parametric operator on the parameter space of a given value function.
Revised: Our PBO is an operator on the parameter space of a given value function.

Original: Given the parameters of a value function, PBO outputs the parameters of a new value function and converges to a fixed point in the limit, as a standard Bellman operator.
Revised: Given the parameters of a value function, PBO outputs the parameters of a new value function and converges to a fixed point in the limit, as a standard Bellman operator.

Original: Notably, our PBO can approximate repeated applications of the true Bellman operator at once, as opposed to the sequential nature of the standard Bellman operator.
Revised: Notably, our PBO can obtain approximated repeated applications of the true Bellman operator, as opposed to the sequential nature of the standard Bellman operator.

Original: We prove the important consequences of this finding for different classes of problems by analyzing PBO in terms of stability, convergence, and approximation error.
Revised: We show how to obtain PBOs for representative classes of RL problems, and how to approximate it resorting to neural network regression.

Original: Eventually, we propose an approximate value-iteration algorithm to show how PBO can overcome the limitations of classical methods, opening up multiple research directions as a novel paradigm in reinforcement learning.
Revised: Eventually, we propose an approximate value-iteration algorithm to learn PBOs and empirically evince how it can overcome the limitations of classical methods, opening up multiple research directions as a novel paradigm in reinforcement learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Value-based reinforcement learning (RL) is a popular class of algorithms for solving sequential decision-making problems with unknown dynamics (Sutton & Barto, 2018).
Revised: Value-based reinforcement learning (RL) is a popular class of algorithms for solving sequential decision-making problems with unknown dynamics (Sutton & Barto, 2018).

Original: For a given problem, value-based algorithms aim at obtaining the most accurate estimate of the expected return from each state, i.e., a value function.
Revised: For a given problem, value-based algorithms aim at obtaining the most accurate estimate of the expected return from each state, i.e., a value function.

Original: For instance, the well-known value-iteration algorithm computes value functions by iterated applications of the Bellman operator (Bellman, 1966), of which the true value function is the fixed point.
Revised: For instance, the well-known value-iteration algorithm computes value functions by iterated applications of the Bellman operator (Bellman, 1966), of which the true value function is the fixed point.

Original: Although the Bellman operator can be applied exactly in dynamic programming, it needs to be estimated from samples at each application when dealing with unknown models of RL problems, i.e., empirical Bellman operator (Watkins, 1989; Bertsekas, 2019).
Revised: Although the Bellman operator can be applied exactly in dynamic programming, it needs to be estimated from samples at each application when dealing with unknown models of RL problems, i.e., empirical Bellman operator (Watkins, 1989; Bertsekas, 2019).

Original: Intuitively, the dependence of value iteration on the samples has an impact on the efficiency of the algorithms and on the quality of the obtained estimate of the value function, which becomes accentuated when solving continuous problems that require value-based methods with function approximation, e.g., approximate value iteration (AVI) (Munos, 2005; Munos & Szepesv´ari, 2008).In this paper, we tackle the original research problem of obtaining a function resembling the Bellman operator.
Revised: Intuitively, the dependence of value iteration on the samples has an impact on the efficiency of the algorithms and on the quality of the obtained estimated value function, which becomes accentuated when solving continuous problems that require value-based methods with function approximation, e.g., approximate value iteration (AVI) (Munos, 2005; Munos & Szepesv´ari, 2008).

Original: In an ideal scenario, such a function should behave exactly like thetrue Bellman operator, being a contraction-mapping ending up to the same fixed point.
Revised: Moreover, in AVI approaches, costly function approximation steps are needed to project the output of the Bellman operator back to the considered action-value functional space. [Equation] [Figure]

Original: Reasonably, the lack of knowledge of the perfect model of the environment limits us to using the empirical Bellman operator instead of the true one.
Revised: [Equation] [Figure] [Equation] [Figure]

Original: Thus, we introduce the novel approach of using samplesfrom the empirical Bellman operator to obtain a new operator, which we call projected Bellman operator (PBO).
Revised: [Equation] In this paper, we tackle these limitations by introducing the novel approach of using samples to obtain a new operator, which we call projected Bellman operator (PBO).

Original: Our PBO is aparametric function Γ ϕ : Ω → Ω defined on parameters ω ∈ Ω of the value function.
Revised: Our PBO is a function Λ : Ω → Ω defined on parameters ω ∈ Ω of the value function.

Original: Similarly to the  Bellman operator that computes an updated value function Q ω k +1 given one Q ω k  , our PBO computes the updated value function  parameters
Revised: Contrarily to the standard (empirical) Bellman operator, which uses action-value functions Q ω k to compute targets that are then projected to obtain Q ω k +1 , our PBO uses the parameters of the action-value function to compute updated parameters

Original: ω k
Revised: ω k

Original: +

Revised: +

Original: =
Revised: =

Original: Γ ϕ ( ω k ) .
Revised: Λ( ω k ) directly (Figure 1).

Original: The parametric nature of our  PBO enables to apply it for an arbitrary number of iterations without using  samples, asopposed to the standard Bellman operator. The crucial advantage of our approach is visualized in Figure 1.
Revised: The crucial advantages of our approach are twofold: (i) the output of PBO always belongs to the considered action-value functional space, avoiding, therefore, the costly projection step typical when using the Bellman operator, and (ii) once learned, PBO is applicable for an arbitrary number of iterations without using further samples, as visualized in Figure 2.

Original: Starting from initial parameters ω 0 , AVI approaches obtain consecutive approximations of the value function 
Revised: Starting from initial parameters ω 0 , AVI approaches obtain consecutive approximations of the value function Q ω k by

Original:  Q ω k by applying the Bellman operator iteratively over samples, e.g., for n iterations (Figure 1b).
Revised: [Figure] Q ∗ and Q ω ∗ are respectively the optimal value function and its projection on the parametric space applying the Bellman operator iteratively over samples (Figure 2b).

Original: Notably, starting from initial parameters ω 0 ,our PBO can produce a chain of updated parameters of arbitrary length (blue lines in Figure 1a). This means that an accurate PBO can compute the optimal value function parameters starting from any initial parameters in the chosen space. In the following,
Revised: Instead, our PBO makes use of the samples to learn the operator only. Then, starting from initial parameters ω 0 , PBO can produce a chain of updated parameters of arbitrary length (blue lines in Figure 2a) without requiring further samples.

Original: [Figure] and ω ∗ after formally introducing PBO and a novel algorithm for value estimation based on it, we analyze its advantageous propertiesunder a theoretical and empirical lens for different classes of problems.
Revised: In the following, after formally introducing PBO and a novel algorithm for value estimation based on it, we analyze its advantageous properties for different classes of problems.

Original: Thus, our contribution is threefold: (i) we introduce the notion of projected Bellman operator for a given policy and for the greedy one; (ii) we show how to derive different PBOs according to the class of problems at hand; (iii) we develop a novel algorithm for value estimation based on PBO and show its advantages over related baselines on several RL problems.
Revised: Thus, our contribution is threefold: (i) we introduce the notion of projected Bellman operator (PBO); (ii) we show how to derive different PBOs according to the class of problems at hand; (iii) we develop a novel algorithm for value estimation based on PBO and show its advantages over related baselines on several RL problems.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Our work is, to the best of our knowledge, the first attempt to obtain an approximated parametric version of the Bellman operator.
Revised: Our work is, to the best of our knowledge, the first attempt to obtain a variant of the Bellman operator that acts on the parameters of action-value functions.

Original: Nevertheless, several works in the literature proposed variants of the standard Bellman operator to induce some desired behavior.Variants of the Bellman Operator.
Revised: Nevertheless, several works in the literature proposed variants of the standard Bellman operator to induce some desired behavior.

Original: For instance, variants of the Bellman operator are widely studied for entropy-regularized MDPs (Neu et al., 2017; Geist et al., 2019; Belousov & Peters, 2019).
Revised: Variants of the Bellman operator are widely studied for entropy-regularized MDPs (Neu et al., 2017; Geist et al., 2019; Belousov & Peters, 2019).

Original: The softmax (Haarnoja et al., 2017; Song et al., 2019), mellowmax (Asadi & Littman, 2017), and optimistic (Tosatto et al., 2019) operators are all examples of variants of the Bellman operator to obtain maximum-entropy exploratory policies.
Revised: The softmax (Haarnoja et al., 2017; Song et al., 2019), mellowmax (Asadi & Littman, 2017), and optimistic (Tosatto et al., 2019) operators are all examples of variants of the Bellman operator to obtain maximum-entropy exploratory policies.

Original: Besides favoring exploration, other approaches address the limitations of the standard Bellman operator.
Revised: Besides favoring exploration, other approaches address the limitations of the standard Bellman operator.

Original: For instance, the consistent Bellman operator (Bellemare et al., 2016) is a modified operator that addresses the problem of inconsistency of the optimal action-value functions for suboptimal actions.
Revised: For instance, the consistent Bellman operator (Bellemare et al., 2016) is a modified operator that addresses the problem of inconsistency of the optimal action-value functions for suboptimal actions.

Original: The distributional Bellman operator (Bellemare et al., 2017) enables to operate on the whole return distribution, instead of its expectation, i.e., the value function (Bellemare et al., 2023).
Revised: The distributional Bellman operator (Bellemare et al., 2017) enables to operate on the whole return distribution, instead of its expectation, i.e., the value function (Bellemare et al., 2023).

Original: Furthermore, the logistic Bellman operator uses a logistic loss to solve a convex linear programming problem to find optimal value functions (Bas-Serrano et al., 2021).
Revised: Furthermore, the logistic Bellman operator uses a logistic loss to solve a convex linear programming problem to find optimal value functions (Bas-Serrano et al., 2021).

Original: Finally, the Bayesian Bellman operator is a method in Bayesian RL to infer a posterior over 
Revised: Finally, the Bayesian Bellman operator is a method in Bayesian RL to infer a posterior over Bellman operators centered on the true

Original: Bellmanoperators centered on the true Bellman operator (Fellows et al., 2021).
Revised: Bellman operator (Fellows et al., 2021).

Original: We point out that our PBO can be seamlessly applied on an arbitrary variant of the standard Bellman operator with just minor adaptations.
Revised: We point out that our PBO can be seamlessly applied on an arbitrary variant of the standard Bellman operator with just minor adaptations.

Original: Operator Learning.
Revised: Operator learning.

Original: Literature in operator learning is mostly focused on supervised learning, with methods for learning operators over vector spaces (Micchelli & Pontil, 2005) and parametric approaches for learning non-linear operators (Chen & Chen, 1995), with a resurgence of recent contributions in deep learning.
Revised: Literature in operator learning is mostly focused on supervised learning, with methods for learning operators over vector spaces (Micchelli & Pontil, 2005) and parametric approaches for learning non-linear operators (Chen & Chen, 1995), with a resurgence of recent contributions in deep learning.

Original: For example, Kovachki et al.
Revised: For example, Kovachki et al.

Original: (2021) learn mappings between infinite function spaces with deep neural networks, or Kissas et al.
Revised: (2021) learn mappings between infinite function spaces with deep neural networks, or Kissas et al.

Original: (2022) apply an attention mechanism to learn correlations in the target function for efficient operator learning.
Revised: (2022) apply an attention mechanism to learn correlations in the target function for efficient operator learning.

Original: The literature about operator learning is much larger, but we consider it out of the scope of this work that, to the best of our knowledge, is the first to deal with the original problem of operator learning in RL.
Revised: The literature about operator learning is much larger, but we consider it out of the scope of this work that, to the best of our knowledge, is the first to deal with the original problem of operator learning in RL.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We consider discounted Markov decision processes (MDPs) M = ⟨S , A , P , R , γ ⟩ , where S is a measurable state space, A is a measurable action space, P : S
Revised: We consider discounted Markov decision processes (MDPs) M = ⟨S , A , P , R , γ ⟩ , where S is a measurable state space, A is a measurable action space, P : S

Original: ×
Revised: ×

Original: A → ∆( S ) 1 is the transition kernel of the dynamics of the system, R : S ×
Revised: A → ∆( S ) 1 is the transition kernel of the dynamics of the system, R : S ×

Original: A → R is a reward function and γ ∈ [0 , 1) is a discount factor (Puterman, 1990).
Revised: A → R is a reward function and γ ∈ [0 , 1) is a discount factor (Puterman, 1990).

Original: A deterministic policy π : S → A is a function mapping a state to an action, inducing a value function
Revised: A deterministic policy π : S → A is a function mapping a state to an action, inducing a value function

Original: V π ( s ) ≜ E [ (cid:80) ∞ t =

Revised: V π ( s ) ≜ E [ (cid:80) ∞ t =

Original: γ t R ( S t , π ( S t )) | S 

Revised: γ t R ( S t , π ( S t )) | S

Original: = s ] , i.e., expected cumulative discounted reward following policy π .
Revised: = s ] , i.e., expected cumulative discounted reward following policy π .

Original: Similarly, the action-value function Q π ( s, a ) ≜
Revised: Similarly, the action-value function Q π ( s, a ) ≜

Original: E [ (cid:80) ∞ t =

Revised: E [ (cid:80) ∞ t =

Original: γ t R ( S t , A t ) | S 

Revised: γ t R ( S t , A t ) | S

Original: = s,
Revised: = s,

Original: A 0 = a, A t = π
Revised: A 0 = a, A t = π

Original: ( S t )] is the expected discounted cumulative reward executing action a in state s , following policy π thereafter.
Revised: ( S t )] is the expected discounted cumulative reward executing action a in state s , following policy π thereafter.

Original: RL aims to find an optimal policy π ∗ yielding the optimal value function V ∗ ( s ) ≜ max π : S→A V π ( s ) , ∀ s ∈ S (Puterman, 1990).
Revised: RL aims to find an optimal policy π ∗ yielding the optimal value function V ∗ ( s ) ≜ max π : S→A V π ( s ) , ∀ s ∈ S (Puterman, 1990).

Original: The  Bellman operator  is a fundamental tool in RL for obtaining optimal policies. Given an arbitrary action-value function Q and policy π , the Bellman operator Γ π is defined as:
Revised: The (optimal) Bellman operator Γ ∗ is a fundamental tool in RL for obtaining optimal policies, and it is defined as

Original: [Equation]
Revised: [Equation]

Original: Similarly, the optimal Bellman operator Γ ∗ is defined as: [Equation] The value function induced by a policy π is the fixed point of the Bellman operator Γ π Q π = Q π . Furthermore, the fixed pointfor the optimal value function is Γ ∗ Q ∗ =
Revised: It is well-known that Bellman operators are contraction mappings in L ∞ -norm, such that iterating over an action-value function leads to the fixed point Γ ∗ Q ∗ =

Original: Q ∗. It is well-known that Bellman operators are contraction mappings in L ∞ -norm, such that iterating over a value function leads to the fixed point in the limit (Bertsekas, 2015).
Revised: Q ∗ in the limit (Bertsekas, 2015).

Original: We consider the use of function approximation to represent value functions and denote Ω the space of their parameters.
Revised: We consider the use of function approximation to represent value functions and denote Ω the space of their parameters.

Original: Additionally, we define 
Revised: Additionally, we define Q Ω =

Original: Q Ω = { Q ω : S ×
Revised: { Q ω : S ×

Original: A → R | ω ∈ Ω } as the set of value functions representable by parameters of Ω .
Revised: A → R | ω ∈ Ω } as the set of value functions representable by parameters of Ω .

Original:  A PPROXIMATING THE B ELLMAN OPERATOR
Revised:  P ROJECTED B ELLMAN OPERATOR

Original: The application of the Bellman operator in RL requires transition samples (Equation 1 and 2) (Munos, 2003; 2005; Munos & Szepesv´ari, 2008).
Revised: The application of the Bellman operator in RL requires transition samples (Equation 1) (Munos, 2003; 2005; Munos & Szepesv ´ ari, 2008).

Original: The impact of this requirement is twofold: (i) the behavior of the Bellman operator strictly depends on the quality of the samples, and, thus, a poor sample selection can result in negligible updates and long detours before convergence; (ii) the direction of the update is determined exclusively by the current samples, meaning that the empirical Bellman operator has no memory of previously observed samples (Bellemare et al., 2016; Agarwal et al., 2019; Fellows et al., 2021).
Revised: The impact of this requirement is twofold: (i) the behavior of the Bellman operator strictly depends on the quality of the samples, thus a poor sample selection can result in negligible updates and long detours before convergence; (ii) the direction of the update is determined exclusively by the current samples, meaning that the empirical Bellman operator has no memory of previously observed samples (Bellemare et al., 2016; Agarwal et al., 2019; Fellows et al., 2021).

Original: Ideally, we want to obtain an operator that mirrors the behavior of the true Bellman operator while being independent of the transition samples and extrapolating from previous experience.
Revised: Ideally, we want to obtain an operator that mirrors the behavior of the true Bellman operator while being independent of the transition samples and extrapolating from previous experience.

Original: Thus, we introduce the idea of a parametric approximation of the true Bellman operator which we call projected Bellman operator (PBO).
Revised: Thus, we introduce the idea of an alternative operator, which we call projected Bellman operator (PBO), defined as follows.

Original: Definition 1.
Revised: Definition 1.

Original: Let Q Ω =
Revised: Let Q Ω =

Original: { Q ω : S ×
Revised: { Q ω : S ×

Original: A → R | ω ∈ Ω } be a function approximation space for the action-value function, induced by the parameter space Ω .A projected Bellman operator (PBO) is a function Γ ϕ : Ω → Ω with parameters ϕ ∈ Φ acting on the parameters ω ∈ Ω .
Revised: A → R | ω ∈ Ω } be a function approximation space for the action-value function, induced by the parameter space Ω .

Original: We call π -PBO the PBO Γ π ϕ w.r.t. a given policy π and optimal PBO the PBO Γ ∗ ϕ w.r.t. to the greedy policy. Note thatwe use the subscript ϕ to distinguish PBO from the standard Bellman operator Γ .
Revised: A projected Bellman operator (PBO) is a function Λ ∗ : Ω → Ω , such that

Original: 4. L EARNING PROJECTED B ELLMAN OPERATORS

Original: We formalize the problem of learning the closest approximation of the π -PBO and the optimal PBO with parametric functions Γ π ϕ and Γ ∗ ϕ , respectively, for a generic RL setting.

Original: Definition 2.

Original: Let π be a policy, we define the π -PBO as the operator

Original: [Equation]
Revised: [Equation]

Original: for some state-action distribution ρ and parameter distribution ν .

Original: Note that the symbol Γ π is the standard Bellman operatorw.r.t. policy π on Q Ω .
Revised: Note that the symbol Γ ∗ is the regular optimal Bellman operator on Q Ω .

Original: Conversely, πPBO is a parametric operator Γ π ϕ acting on the space of parameters Ω .
Revised: Conversely, PBO is an operator Λ ∗ acting on the space of parameters Ω .

Original: The existence of π -PBO is not guaranteed for every setting and has to be studied for each parameterization of the value function. An interesting case is when Q Ω is stable by the Bellman operator, i.e., Γ π Q Ω ⊂ Q Ω . In this case, π -PBO exists for any ω ∈ Ω , and there exist parameters ω ∗ such that Γ π Q ω =
Revised: All proofs of following results can be found in

Original: Q ω ∗ .
Revised: Appendix A.

Original: The operator defined as ω (cid:55)→ ω ∗ is optimal since it has a value of 0 when evaluated on the function.
Revised: 4. F UNCTION APPROXIMATION FOR PROJECTED BELLMAN OPERATORS

Original: In addition, π -PBO can be computed in closed form if it exists for Γ π Q ω . Definition 3. We define the optimal PBO as the operator
Revised: In the following, we show that PBO can be expressed in closed-form for representative classes of

Original: [Equation] (4) for some state-action distribution ρ and parameter distribution ν . Again, the symbol Γ ∗ appears twice to represent two different objects. It is important to note that the optimal Bellman operator is not defined for every function Q ∈ Q Ω because, for some parameters ω , a state s ′ , and an action a ′ , max a ′ ∈A Q ω ( s ′ , a ′ ) in Equation (2) might not be defined in the case of continuous actions.
Revised: RL problems: (i) finite MDPs, (ii) low-rank MDPs (Agarwal et al., 2020; Sekhari et al., 2021), and (iii) linear quadratic regulation problems (Bradtke, 1992; Pang & Jiang, 2021).

Original: This means that the existence of the optimal PBO is not guaranteed and that Ω has to be carefully chosen.

Original: Similarly to π -PBO, the case where Q Ω is stable by the optimal Bellman operator, i.e., Γ ∗ Q Ω ⊂ Q Ω , guarantees the existence of the optimal PBO and can be computed in closed form if Γ ∗ Q ω has one.

Original: The parametric nature of our PBO gives the flexibility of designing different PBOs according to the problem at hand.

Original: In the following, we define different PBOs, characterizing their properties for increasingly complex classes of RL problems (all proofs in Appendix A), and we further empirically analyze them in Section 6.

Original: 4. PBO S IN FINITE M ARKOV DECISION PROCESSES
Revised: 4.1. F INITE M ARKOV DECISION PROCESSES

Original: Let us consider finite state and action spaces of cardinality N and M respectively, and a tabular setting with Ω = R N · M .
Revised: Let us consider finite state and action spaces of cardinality N and M respectively, and a tabular setting with Ω = R N · M .

Original: Intuitively, there is a trivial bijection between Q Ω and Ω , which allows us to write the parameters of the value function as Q instead of ω .

Original: We notice that the space of Q functions is stable by the Bellman operators since any Bellman operator brings a table in R N · M to a table in R N · M ; thus, π -PBO and the optimal PBO exist.

Original: For Ω ∈ R N , where N is the number of parameters, we can derive a class of linear PBOs defined as

Original: [Equation]

Original: where ϕ is the concatenated vector of A and b .

Original: For any linear PBO parameterized by A and b with ∥ A ∥ 2 < 1 , infinite iterations of this operator lead to the fixed point ( I − A ) − 1 b by the Banach fixed-point theorem (Khamsi & Kirk, 2011) 2 .

Original: This fixed point can be computed in closed form, and it is differentiable w.r.t.

Original: the parameters ϕ .

Original: The π -PBO exists and it is equal to the Bellman operator

Original: [Equation]

Original: This clearly shows that π -PBO is a linear function Γ ϕ parameterized by A = γP Π ∈ R N · M × N · M and b = R ∈ R N · M . Notably, π -PBO is a γ -contraction mapping for the sup -norm since the Bellman operator is a γ -contraction mapping.
Revised: Intuitively, there is a trivial bijection between Q Ω and Ω , which allows us to write the parameters of the action-value function as Q instead of ω .

Original: Proposition 1.
Revised: Proposition 1.

Original: Theoptimal PBO exists and it is equal to the optimal Bellman operator
Revised: The PBO exists and it is equal to the optimal Bellman operator

Original: [Equation]
Revised: [Equation]

Original: The non-linearity caused by the maximum operator requires the introduction of another class of PBOs that can be optimal in this setting. We call this class max-lin PBOs, defined as Γ ϕ ( Q ) =
Revised: Note that, since the optimal Bellman operator is a γ -contraction mapping for the sup -norm, PBO for finite MDPs (Equation 3) is also a γ -contraction mapping.

Original: b + A max a ∈A Q , with parameters b ∈ R N · M and A ∈ R N · M × N . Then, the optimal PBO is max-lin with parameters A = γP and b = R . Finally, since the optimal Bellman operator is a γ -contraction mapping for the sup -norm, the optimal PBO is also a γ -contraction mapping.
Revised: 4.1. L OW - RANK M ARKOV DECISION PROCESSES

Original: Low-rank MDPs is a class of problems with two feature maps σ : S ×
Revised: Low-rank MDPs is a class of problems with two feature maps σ : S ×

Original: A → R d and µ : S → R d , such that P ( s ′ | s, a ) = ⟨ σ ( s, a ) , µ ( s ′ ) ⟩ and R ( s, a ) = ⟨ σ ( s, a ) , θ ⟩ , for all ( s, a, s ′ )
Revised: A → R d and µ : S → R d , such that P ( s ′ | s, a ) = ⟨ σ ( s, a ) , µ ( s ′ ) ⟩ and R ( s, a ) = ⟨ σ ( s, a ) , θ ⟩ , for all ( s, a, s ′ )

Original: ∈ S ×
Revised: ∈ S ×

Original: A × S and for θ ∈ R d .
Revised: A × S and for θ ∈ R d .

Original: We assume, without loss of generality, that for all ( s, a ) ∈ S × A , ∥ σ ( s, a ) ∥ 1 ≤and max {∥ µ ( s ) ∥ 1 , ∥ θ ∥ 1 } ≤ √ d .Provably, the space of value functions is the space of linear functions in the parameters, i.e., Q Ω = {⟨ σ ( · , · ) , ω ⟩| ω ∈ R d } . This space is stable by the Bellman Γ since operators and the closed form can be obtained, as explained in Section 4.1.
Revised: We assume, without loss of generality, that for all ( s, a ) ∈ S × A , ∥ σ ( s, a ) ∥ 1 ≤and max {∥ µ ( s ) ∥ 1 , ∥ θ ∥ 1 } ≤ √ d .

Original: In case of continuous and unbounded state and action spaces, for a policy π and for parameters ω ∈ R d , the π -PBO is [Equation] where A π = γ (cid:82) S µ ( s ′ ) E a ′ ∼ π ( ·| s ′ ) 
Revised: We set the space of action-value functions to be the space of linear functions in the parameters, i.e., Q Ω = {⟨ σ ( · , · ) , ω ⟩| ω ∈ R d } .

Original: [ σ ( s ′ , a ′ )] T d s ′ .

Original: This result shows that π -PBO is a linear PBO; thus, under the latent variable representation assumption, π -PBO is a γ -contraction.

Original: Proposition 2.
Revised: Proposition 2.

Original: In case of continuous state and action spaces and for ω ∈ R d , theoptimal PBO is
Revised: In case of continuous state and action spaces and for ω ∈ R d , the PBO is

Original: [Equation]
Revised: [Equation]

Original: The optimal PBO is not a linear PBO. However, its class can be understood from its closed form. To make the study easier, we first come back to finite state and action spaces of size N and M respectively.
Revised: The closed-form is again a γ -contraction mapping under the assumption that the MDP has a latent variable representation.

Original: We define a new class of operator that we call lin-max-lin , parameterized by A 1 ∈ R N · M × d , A 2 ∈ R N × d , and b ∈ R d , with the following form: Γ ϕ ( w ) = b + A T 2 max a ′ ∈A A 1 w , where ϕ is the concatenation of all the parameters. The optimal PBO is a lin-max-lin PBO (proof in Appendix A) with parameters:
Revised: 4.1. L INEAR QUADRATIC REGULATION

Original: [Equation]

Original: For infinite state and action spaces, the optimal PBO is a lin-max-lin operator with parameters having an infinite number of dimensions and, in both finite and infinite cases, the optimal PBO is a γ -contraction mapping under the assumption that the MDP has a latent variable representation.

Original: Eventually, we consider the continuous MDPs class of linear quadratic regulator (LQR) with
Revised: Eventually, we consider the continuous MDPs class of linear quadratic regulator (LQR) with

Original: S =
Revised: S =

Original: A = R .
Revised: A = R .

Original: The transition model P ( s, a ) =
Revised: The transition model P ( s, a ) =

Original: As + Ba is deterministic and the reward function R ( s, a ) =
Revised: As + Ba is deterministic and the reward function R ( s, a ) =

Original: Qs 2 + 2 Ssa + Ra 2 is quadratic, where A , B Q , S and R , are constants inherent to the MDP.
Revised: Qs 2 + 2 Ssa + Ra 2 is quadratic, where A , B Q , S and R , are constants inherent to the MDP.

Original: As commonly done for LQR, we consider an undiscounted setting, thus we do not analyze the contraction property of the PBOs.
Revised: As commonly done for LQR, we consider an undiscounted setting, thus we do not analyze the contraction property of PBO.

Original: We choose to parameterize the action-value functions with a 3 -dimensions parameter vectorQ Ω = { ( s, a ) (cid:55)→
Revised: We choose to parameterize the action-value functions with a 3 -dimensional parameter vector

Original: Gs 2 + 2 Isa + Ma 2 | ( G, I,

Original: M ) ∈ R 3 } , for visualization purposes.

Original: Thus, the optimal action-value function is Q ∗ ( s, a ) =

Original: [-( Q + A 2 P ) s 
-]
Revised: [-( Q + A 2 P ) s 
-]

Original: + 2( S +

Original: ABP )

Original: sa

Original: + ( R + B 2 P ) a 2 , where P is the solution of the Riccati equation (proof in Appendix A).

Original: The space of value functions Q Ω is not stable for all the Bellman operators, and it depends on the policy

Original: π

Original: (see Appendix A).

Original: Nevertheless, the stability of Q Ω can be shown for the policies that are greedy with respect to a value function belonging to Q Ω .

Original: Note that the greedy policy of a value function only exists if the parameters w parameterizing the value function belong to R 2 × R −∗ ∪ R × { 0 } × { 0 } .

Original: Indeed, for w = ( G, I, M ) ∈ R 3 and s ∈ S , arg max a ∈A Gs 2 + 2 Isa + Maexists if and only if M < 0 or I =

Original: M = 0 .

Original: In the case where the greedy policy exists, πPBO exists, is linear and can be derived in closed form.

Original: For a value function parameterized by w = ( G, I,

Original: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } , π -PBO is

Original: [Equation]

Original: where π Q w is the greedy policy with respect to Q w , b

Original: [Equation]
Revised: [Equation]

Original: AB B 2 (cid:3) T · (cid:2) [Equation] C 2 (cid:3) and C = − IM .
Revised: | ( G, I, M ) ∈ R 3 } , for visualization purposes.

Original: A geometrical interpretation of the behavior of this PBO is also available in the proof (Appendix A).

Original: The space of value functions Q Ω is stable by the optimal Bellman operator when the optimal Bellman operator is defined, i.e., for parameters in R

Original: 2 × R −∗ ∪ R × { 0 } × { 0 } , as for π -PBO.

Original: Proposition 3.
Revised: Proposition 3.

Original: Theoptimal PBO exists and for any w ∈ R 2 × R − ∗ 3 its closed form is
Revised: The PBO exists and for any ω ∈ R 2 × R −∗ 2 its closed form is

Original: [Equation][Table]
Revised: [Equation]

Original: The optimal PBO can be understood as a composition of the function: Λ : w (cid:55)→ G − I 2 M and a linear operator with the following parameters: A = (cid:2) A AB
Revised: A geometrical interpretation of the behavior of this PBO is available in Appendix A.

Original: B 2 (cid:3) T and b =

Original: [ Q S R ] T . We then have Γ ∗ ϕ ( w ) =
Revised: 4. A PPROXIMATING PROJECTED B ELLMAN OPERATORS

Original: Λ( w ) · A + b . We call this class of operator Λ -lin . A geometrical interpretation of the behavior of this PBO is available in Appendix A. Table 1 summarizes the theoretical results stated in this section.
Revised: For generic RL problems that do not fall into the previously described categories, the true PBO is unknown and has to be estimated from samples.

Original: For every setting, we report the stability of Q Ω by the Bellman operator along with the class of PBOand its modulus of contraction for the sup-norm. The purple color specifies that the results only hold with the latent variable representation assumption. Moreover, we highlight in orange the result that only holds for greedy policies, and in brown the result that only holds when the optimal Bellman operator is defined.
Revised: We propose to approximate the unknown PBO with a parameterized PBO Λ ϕ differentiable w.r.t.

Original: We devise an algorithm to learn optimal PBOs for a chosen class of PBOs. This choice has to be made according to the parameterization ofthe value function, as discussed in Section 4. Since we define our PBOs and value functions as differentiable with respect to their parameters, we can optimize with gradient descent on a loss function.
Revised: its parameters ϕ ∈ Φ , enabling the use of gradient descent on a loss function.

Original: A first idea would be to learn an operator that minimizes the TD-error in the definition ofthe optimal PBO (Equation 4).
Revised: A first idea would be to learn an operator that minimizes the loss in the definition of PBO (Equation 2).

Original: While this approach has the benefit of allowing to sample as many parameters as needed from Ω , it has the major drawback of not leveraging the crucial fact that learning an operator allowsus to iterate over it as many times as desired.
Revised: While this approach has the benefit of allowing to sample as many parameters as needed from Ω , it does not leverage the crucial fact that approximating an operator allows to iterate over it as many times as desired.

Original: Thus, our key insight is to include multiple iterations of the operator to the loss.
Revised: Thus, our key insight is to include multiple iterations of the operator to the loss.

Original: The function to minimize becomes:
Revised: The function to minimize becomes

Original: [Equation]
Revised: [Equation]

Original: (13) for all ω ∈ Ω , s ∈ S , and a ∈ A , and where K is an arbitrary number of optimal Bellman operator iterations.
Revised: 6) for all ω ∈ Ω , s ∈ S , and a ∈ A , and where K is an arbitrary number of optimal Bellman operator iterations.

Original: An additional idea is to add a term that corresponds to an infinite number of iterations, i.e., the fixed point, when possible.
Revised: An additional idea is to add a term that corresponds to an infinite number of iterations, i.e., the fixed point, when possible.

Original: Now, the loss becomes:
Revised: Now, the loss becomes

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Fixed
Revised: [Equation] [Equation] for all ω ∈ Ω , s ∈ S , and a ∈ A , where Λ ωϕ is the fixed point of the parameterized PBO. Note that the addition of the fixed point term is only possible for classes of parameterized PBOs where the fixed point can be computed and the computation is differentiable in the parameters ϕ .

Original: [Equation]
Revised: [Figure]

Original:  1: Inputs: samples D = {⟨ s j , a j , r j , s ′ j ⟩} Jj =1 , parameters W = { ω l } Ll =1 , #Bellman iterations K , #training steps I , #fitting steps H , operator Γ ϕ to learn with initial parameters ϕ
Revised: Algorithm 1: Inputs: samples D = {⟨ s j , a j , r j , s ′ j ⟩} J j =1 , parameters W = { ω l } Ll =1 , #Bellman iterations K , #training steps I , #fitting steps H , operator Λ ϕ to learn with initial parameters ϕ

Original: = ϕ 0 , learning rate λ 2: for i ∈ 1 . . .
Revised: = ϕ 0 , learning rate λ 2: for i ∈ 1 . . .

Original: I do 3: # Compute the target T 4: for k ∈ 1 . . .
Revised: I do 3: # Compute the target T 4: for k ∈ 1 . . .

Original: K do 5: for ( w l , ( s j , a j , rj , s ′ j ))
Revised: K do 5: for ( ω l , ( s j , a [Equation] , r

Original: ∈ W × D do 6:
Revised: [Equation]

Original: T kl ( s
Revised: , s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: W × D do [Equation] [Equation] [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Figure]

Original: 7: end for 8: end for 9: # Minimize the loss starting from the previous parameters 10: ϕ i = ϕ i −11: for ∈ 1 . . .
Revised: 7: end for 8: end for 9: # Minimize the loss starting from the previous parameters 10: ϕ i = ϕ i −11: for ∈ 1 . . .

Original: H do 12: for j ∈ 1 . . .
Revised: H do 12: for j ∈ 1 . . .

Original: J do 13: ϕ i ← ϕ
Revised: J do 13: ϕ i ← ϕ

Original: i − λ
Revised: i − λ

Original: L (cid:88) l =
Revised: L (cid:88) l =

Original: J (cid:88) j
Revised: J (cid:88) j

Original: =
Revised: =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = ∇ ϕ (cid:16) T kl ( s j , a j ) − Q Γ k ϕ
Revised: = ∇ [Equation] (cid:16) [Equation]

Revised: kl ( s

Original: i
Revised: [Equation] [Equation] [Equation] ( ω

Revised: [Equation]

Original: w l ) ( s j , a j ) (cid:17)14: end for 15: end for 16: end for 17: return Parameters
Revised: ( s [Equation] [Equation] (cid:17)14: end for 15: end for 16: end for 17: return Parameters

Original: ϕ
Revised: ϕ

Original: I of operator Γ ϕ
Revised: I of operator Λ

Original: I for all ω ∈ Ω , s ∈ S , and a ∈ A , where Γ w ϕ is the fixed point of the PBO. Note that the addition of the fixed point term is only possible for classes of PBOs where the fixed point can be computed and the computation is differentiable in the parameters of the PBO, such as for the class of linear PBOs described in Section 4.2.
Revised: We devise an approximate value iteration algorithm to learn parameterized PBOs by solving the optimization problems (6) and (7).

Original: Thepseudo-code of our algorithm is presented in Algorithm 1. The general structure of the algorithm closely resembles the one of Fitted Q -Iteration (Ernst et al., 2005), alternating one phase in which we compute the targets, and another one where we apply gradient descent to fit the parameters w.r.t.
Revised: The general structure of the algorithm closely resembles the one of Fitted Q -Iteration (Ernst et al., 2005), alternating one phase in which we compute the targets, and another one where we apply gradient descent to fit the parameters w.r.t.

Original: the targets.
Revised: the targets (Algorithm 1).

Original: For this reason, we call our algorithm projected Q -iteration (PQI).
Revised: For this reason, we call our algorithm Projected Q -Iteration (PQI).

Original: When possible, the fixed point of the  PBO is added to the loss.
Revised: When possible, the fixed point of the parameterized PBO is added to the loss.

Original: An interesting property of our method is that it tries to learn all the optimal Bellman iterations at once, as opposed to FQI, that learns each Bellman iteration sequentially: Q i ∈ arg inf Q i ∈Q ∥ Q i − Γ ∗ Q i− 1 ∥ . Thus, we can conclude that PBO enables to build a richer loss compared to other AVI methods.
Revised: An interesting property of our method is that it enables to build a rich loss by including multiple optimal Bellman iterations at the same time, as opposed to FQI that learns each Bellman iteration sequentially: [Equation] i

Original:  Figureillustrates this advantage by comparing the way PQI and FQI behave in the space of all possible action-value functions.
Revised: [Equation] Figureillustrates this advantage by comparing the way PQI and FQI behave in the space of all possible action-value functions.

Original: The ability of iterating for an arbitrary number of times enables us to derive theoretical advantages of PQI by leveraging established results in approximation error for AVI.
Revised: The ability of iterating for an arbitrary number of times enables us to derive theoretical advantages of PQI by leveraging established results in approximation error for AVI.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Theorem 3.4 of Farahmand (2011)) Let K ∈ N ∗ , ρ, ν two distribution probabilities over S×A .
Revised: (Theorem 3.4 of Farahmand (2011)) Let K ∈ N ∗ , ρ, ν two distribution probabilities over S×A .

Original: For any sequence ( Q k ) Kk =0{+K k =
+} ⊂ B ( S × A , R γ ) where R γ depends on the reward function and the discount factor, we have
Revised: For any sequence ( Q k ) {+K k =
+} ⊂ B ( S × A , R γ ) where R γ depends on the reward function and the discount factor, we have

Original: [Equation]
Revised: [Equation]

Original: where α k and C K,γ,R γ do not depend on the sequence ( Q k )
Revised: where α k and C K,γ,R γ do not depend on the sequence ( Q k )

Original: Kk =0 .
Revised: Kk =0 .

Original: F ( r ; K, ρ, γ ) relies on the concentrability coefficients of the greedy policies w.r.t.
Revised: F ( r ; K, ρ, γ ) relies on the concentrability coefficients of the greedy policies w.r.t.

Original: the value functions.
Revised: the value functions.

Original: This theorem shows that the distance between the value function and the optimal value function depends on the approximation errors for all the iterations.
Revised: This theorem shows that the distance between the value function and the optimal value function depends on the approximation errors for all the iterations.

Original: The loss of FQI contains only one term of the sum, while the loss of PQI contains the entire sum.
Revised: The loss of FQI contains only one term of the sum, while the loss of PQI contains the entire sum.

Original: In PQI, the terms are summed up with equal weights, which is not the case for the error propagation.
Revised: In PQI, the terms are summed up with equal weights, which is not the case for the error propagation.

Original: Future works could investigate how to weight those terms for increased performance.[Figure] | P P i | || V * V i ||
Revised: Future works could investigate how to weight those terms for increased performance.

Original: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We empirically analyze our PQI algorithm with different classes of  PBOs w.r.t.
Revised: We empirically analyze our PQI algorithm with different classes of parameterized PBOs w.r.t.

Original: FQI (Ernst et al., 2005) and, when possible, the additional baseline least-squares policy iteration (LSPI) (Lagoudakis & Parr, 2003).
Revised: FQI (Ernst et al., 2005) and, when possible, the additional baseline least-squares policy iteration (LSPI) (Lagoudakis & Parr, 2003).

Original: Note that the fixed point is always added to the loss for the linear PBO, unless stated otherwise.
Revised: The fixed point is always added to the loss for the linear PBO, unless stated otherwise.

Original: The pseudocodes of FQI and LSPI are provided in Appendix B.
Revised: The pseudocodes of FQI and LSPI are provided in Appendix B.

Original: Experimental details.
Revised: Experimental details.

Original: Each algorithm receives a fixed dataset of samples D and has access to the same number of optimal Bellman iterations K .
Revised: Each algorithm receives a fixed dataset of samples D and has access to the same number of optimal Bellman iterations K .

Original: We define the initialparameters, which we call validation parameters, for both FQI and LSPI such that they result in a zero-function.
Revised: We define the initial parameters for both FQI and LSPI such that they result in a zero-function.

Original: We store the parameters after each iteration of the algorithms; then, we compute the action-value function for each iteration and the value function associated with the greedy policy computed from the actionvalue function at each iteration.
Revised: We store the parameters after each iteration of the algorithms; then, we compute the action-value function for each iteration and the value function associated with the greedy policy computed from the action-value function at each iteration.

Original: PQI can access the same functions by iterating over the validation parameters for more than K iterations, thanks to our  PBO.
Revised: PQI can access the same functions by iterating over the initial parameters for more than K iterations, thanks to our parameterized PBO.

Original: When possible, we include the parameters corresponding to an infinite number of iterations, i.e., the fixed point of the  PBO.
Revised: When possible, we include the parameters corresponding to an infinite number of iterations, i.e., the fixed point of the parameterized PBO.

Original: In the figures, the PBOs that use more complex function approximators than linearPBOs are denoted PBO deep .
Revised: We learn parameterized PBOs both with linear function approximators ( linear PBO ) and neural networks ( neural PBO ). We also report the performance of the fixed point of the linear PBO (linear PBO fixed point). For problems when PBO is known in closed-form, we include it in the analysis as a reference ( PBO in the figures).

Original: Additional details on the experiments are in Appendix C.
Revised: Additional details on the experiments are in Appendix C.

Original: 6. C HAIN - WALK
Revised: 5. C HAIN - WALK

Original:  N0.
Revised: [Figure] N

Original: [Figure]
Revised: [Figure]

Original: We initially consider the chain-walk environment in Figure 3, with a chain of length N = 20 .
Revised: We initially consider the chain-walk environment in Figure 4, with a chain of length N = 20 .

Original: We parameterize value functions as tables to leverage our theoretical results on finite
Revised: We parameterize value functions as tables to leverage our theoretical results on finite

Original: MDPs (Section 4.2), training a linear PBO and a max-lin PBO.
Revised: MDPs (Section 4.1.1).

Original: In this simple setting, we also have access to the optimal

Original: PBO and we include it in our analysis. We plot the ℓ 2 -norm between the iterated value functions and the optimal ones in Figure 4a.
Revised: We plot the ℓ 2 -norm between the iterated value functions and the optimal ones in Figure 5a.

Original: For all methods, the Bellman operator is iterated for K = 5 times during training.
Revised: For all methods, the Bellman operator is iterated for K = 5 times during training.

Revised: The vertical black dotted line denotes the split between the iterations that are learnt and the iteration that can be only reached by parameterized PBO.

Original: Notably, we can iterate the learned PBO for an arbitrary number of times.
Revised: Notably, we can iterate the learnt PBO for an arbitrary number of times.

Original: This property is crucial to nullify the distance between the value function computed with the  PBOs and the optimal value function.
Revised: This property is crucial to nullify the distance between the value function computed with the parameterized PBOs and the optimal value function.

Original: The superior performance of PQI comes from the fact that both classes of operators are close to the class of theoptimal PBO.
Revised: The superior performance of PQI comes from the fact that both classes of operators are close to the class of the PBO.

Original: Since in this settingwe have access to the optimal Bellman operator through the optimal PBO 4 , we can foresee that any algorithm minimizing the approximation error by learning parameters of value functions would perform in the best case as good as the optimal Bellman operator, meaning that their performance would coincide with the one ofthe optimal PBO at iteration 5 .
Revised: Since in this setting the PBO perfectly replicates the behavior of the optimal Bellman operator, we can foresee that any algorithm minimizing the approximation error by learning parameters of value functions would perform in the best case as good as the PBO, meaning that their performance would coincide with the one of PBO at iteration 5 .

Original: For this reason, any algorithm designed as such would perform worse than or equal to FQI in this setting.
Revised: For this reason, any algorithm designed as such would perform worse than or equal to FQI in this setting.

Original: Note that LSPI is not aiming at reducing the approximation error, this is why the reasoning does not apply to this baseline.
Revised: Note that LSPI is not aiming at reducing the approximation error, this is why the reasoning does not apply to this baseline.

Revised: Analysis of the size of the dataset of parameters.

Revised: Ideally, the dataset of parameters should contain parameters close to the initial ones (the parameters with only zeros), others close to optimal

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: parameters, and others in between.

Revised: Since we only assume to know the initial parameters, we take parameters close to them

Revised: [Figure]

Revised: in the sense that the truncated normal distribution used for data collection is centered.

Revised: Figure 5b shows the distance to the optimal value function for different number of parameters.

Revised: Reasonably, the more parameters are given to the algorithm, the better the performance.

Revised: Analysis on the number of Bellman iterations K .

Revised: We additionally analyze the behavior of PBOs for a different number of Bellman iterations than K = 5 .

Revised: We consider K = 2 and K = 15 , observing that the PBOs still behave well as shown in Figure 6.

Original: 6. L INEAR QUADRATIC REGULATOR
Revised: 5. L INEAR QUADRATIC REGULATOR

Original: We consider the LQR setting in Section 4.4, leveraging our theoretical analysis by parameterizing value functions accordingly.
Revised: We consider the LQR setting in Section 4.1.3, leveraging our theoretical analysis by parameterizing value functions accordingly.

Original: The theoretical analysis suggests using a Λ -lin PBO, andwe also use a linear PBO.
Revised: Again, we consider a neural PBO and a linear PBO.

Original: Figure 4b shows a metric proportional to the distance between the optimal value function and the value functions obtained at each iteration.
Revised: Figure 7a shows a metric proportional to the distance between the optimal value function and the value functions obtained at each iteration.

Original: For all methods, we set K = 2 .
Revised: For all methods, we set K = 2 .

Original: As in the previous experiment, the chosen class of operators is close to the class of the optimal PBO.
Revised: As in the previous experiment, the chosen class of operators is close to the class of the optimal PBO.

Original: We see that, at iteration 2 , the performance of FQI and the PQI are on par, while afterward, PQI keeps improving until the fixed point is reached.
Revised: We see that, at iteration 2 , the performance of FQI and the PQI are on par, while afterward, PQI keeps improving until the fixed point is reached.

Original: Again, we can state that any algorithm that learns parameters of value functions trying to minimize the approximation error, would perform worse than FQI.
Revised: Again, we can state that any algorithm that learns parameters of value functions trying to minimize the approximation error, would perform worse than FQI.

Original: FQI follows the curve of theoptimal PBO, meaning that the training cannot be improved.
Revised: FQI follows the curve of the PBO, meaning that the training cannot be improved.

Original: LSPI performs badly because the first policy for the evaluation step is a uniform policy, and the value function associated with the uniform policy is not a quadratic function.
Revised: LSPI performs badly because the first policy for the evaluation step is a uniform policy, and the value function associated with the uniform policy is not a quadratic function.

Revised: Analysis on the number of Bellman iterations K .

Revised: Since the convergence of all the algorithms is fast in term of iterations, we additionally show the performance with 4 Bellman iterations instead of 2 .

Original: 6. N ON - LINEAR PROJECTED B ELLMAN OPERATORS
Revised: Figures 7b confirms that PBOs perform well for a larger number of Bellman iterations. Analysis on the size of the dataset of samples. We focus on neural PBO. Figure 8a shows the distance to the optimal value function for trainings with a varying number of samples.

Revised: When the grid becomes thinner, the distances are decreasing.

Revised: However this decrease is not strong.

Revised: This is why

Revised: [Figure]

Revised: V i |

Revised: [Figure]

Revised: [Figure]

Revised: the size of the dataset is not a crucial parameter for learning PBO.

Revised: Visualization.

Revised: This setting allow us to plot the parameters of the action-value functions in a 3D space.

Revised: The parameters corresponding to the iterations of PQI and FQI are shown in Figure 8b.

Revised: The training is done with K = 2 .

Revised: The first parameters for the three algorithms are on (0 , 0 , 0) and then, the iterations goes towards the optimal parameters on the bottom right of the figure.

Revised: In this case, being able to iterate after the training clearly makes the difference, as it is clear that PQI is able to get closer to the optimal parameters after the second iteration

Revised: [Figure]

Revised: [Figure]

Revised: || V

Revised: * V i ||

Original:  We consider a more advanced setting where the value function is approximated by a neural network, thus the space of value functions is not stable by the optimal Bellman operator.
Revised: [Figure] [Figure] 5. N ON - LINEAR PROJECTED B ELLMAN OPERATORS We consider a more advanced setting where the value function is approximated by a neural network, thus the space of value functions is not stable by the optimal Bellman operator.

Original: We focus on the caron-hill MDP (Ernst et al., 2005), as implemented in MushroomRL (D’Eramo et al., 2021), and the bicycle MDP in Randlov & Alstrøm (1998).
Revised: We focus on the caron-hill MDP (Ernst et al., 2005), as implemented in MushroomRL (D’Eramo et al., 2021) [Figure] and the bicycle MDP in Randlov & Alstrøm (1998).

Original: We parameterize value functions with one hidden layer of 30 neurons, and we compare PQI only to FQI, since LSPI is not suitable for this setting.
Revised: We parameterize value functions with one hidden layer of 30 neurons [Figure] and we compare PQI only to FQI, since LSPI is not suitable for this setting.

Original: We train a linear PBO and, inspired by the theoretical analysis in Section 4.3, we train an operator close to being lin-max-lin – its architecture is lin-ReLU-lin-max-lin-ReLU-lin .
Revised: We train a linear PBO and a neural PBO.

Original: We set K =for car-on-hill and K = 8 for bicycle.
Revised: We set K = 9 for car-on-hill and K = 8 for bicycle.

Original: Figure 5 shows  that PBOs outperform FQI.
Revised: Figure 9 shows the policies on car-on-hill for FQI and PQI trained with the neural PBO [Figure] where PQI is able to obtain a significantly more accurate policy than FQI. Figures 10 and 11a show that PBOs outperform FQI in terms of performance too.

Original: Interestingly, iterating further produces a drop in performance forthe linear PBO, indicating that the class of linear operator is too simple.
Revised: Interestingly, iterating further produces a drop in performance for linear PBO, indicating that the class of linear operator is too simple.

Original: Remarkably, deep PBO either shows no deterioration (Figure 5a), or a slight deterioration due to opposite performance in different seeds (Figure 5b and Appendix C.4).
Revised: Remarkably, neural PBO either shows no deterioration (Figure 10a), or a slight deterioration due to opposite performance in different seeds (Figure 11a and Appendix C.4).

Original: Further tuning should substantially reduce the number of failing seeds.
Revised: Further tuning should substantially reduce the number of failing seeds.

Original: This experimentclearly evinces the advantage ofPQI. FQI is a greedy algorithm that follows a path to the optimal parameters stepby-step, thus, if an error is made at an iteration, it propagates until the last iteration (Theorem 1).
Revised: This experiment evinces the limits of FQI where errors made at any iteration propagates until the last iteration (Theorem 1).

Original: Conversely, PQI can change the path to the optimal parameters at each optimization step.
Revised: Conversely, PQI can change the path to the optimal parameters at each optimization step.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced the original idea of obtaining an approximation of the true Bellman operator, which we call projected Bellman operator (PBO).
Revised: We introduced the original idea of obtaining an approximation of the true Bellman operator, which we call projected Bellman operator (PBO).

Original: We defined PBO for different classes of problems, studied their theoretical properties of stability and convergence, and devised a novel approximate value iteration algorithm that provably reduces the error propagation over Bellman iterations.
Revised: We defined PBO for different classes of problems, studied their theoretical properties of stability and convergence, and devised a novel approximate value iteration algorithm that provably reduces the error propagation over Bellman iterations.

Original: Finally, we have empirically shown the benefit of PBO over related baselines on a variety of problems.
Revised: Finally, we have empirically shown the benefit of PBO over related baselines on a variety of problems.

Original: We consider this work a seminal study of PBO as a novel paradigm of RL, that can lead to multiple research directions, among which investigating its use in online settings is a first foreseeable avenue.
Revised: We consider this work a seminal study of PBO as a novel paradigm of RL, that can lead to multiple research directions, among which investigating its use in online settings is a first foreseeable avenue.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: We aim to guarantee the reproducibility of our work by providing the proofs of our theoretical results and additional details of the conducted experiments in the appendix.
Revised: We aim to guarantee the reproducibility of our work by providing the proofs of our theoretical results and additional details of the conducted experiments in the appendix.

Original: Further, we provide the code for reproducing our results in the supplementary material, and we will make it public upon acceptance
Revised: Further, we provide the code for reproducing our results in the supplementary material, and we will make it public upon acceptance

Original: A P ROOFS
Revised: A P ROOFS

Original: Prooffor Section 4.2: existence of π -PBO for a finite MDP.
Revised: Proof of the closed-form of the PBO for a finite MDP.

Original: As suggested in Section 4.1, we compute the  Bellman iterationfor a policy π over a table Q ∈ R N · M
Revised: We compute the optimal Bellman iteration over a table Q ∈ R N · M

Original: [Equation]
Revised: [Equation]

Original: where P ∈ R N · M × N is the transition probability matrix of the environment and Π ∈ R N × N · M is the probability matrix if the policy π is followed.
Revised: where P ∈ R N · M × N is the transition probability matrix of the environment.

Original: From these equations, we get that π -PBO exists and Γ π ϕ ( Q ) = R + γP Π Q . Proof for Section 4.2: existence of the optimal PBO for a finite MDP. We proceed the same way as for π -PBO.
Revised: From these equations, the operator Q (cid:55)→ R + γP max a ′ ∈A Q ( · , a ′ ) , evaluated on the objective function from the definition of PBO 2, yields zero error.

Original: By replacing the expectation over the next action by a max function in (16), we obtain

Original: [Equation] We then get that  theoptimal PBO exists and Γ ∗ ϕ ( Q )
Revised: This means that we have found the PBO in closed-form.

Original: = R + γP max a ′ Q . Prooffor Section 4.3: existence of π -PBO for a low-rank MDP.
Revised: Proof of the closed-form of the PBO for a low-rank MDP.

Original: The proof considers continuous unbounded stateand action spaces.
Revised: The proof considers continuous unbounded state action spaces.

Original: For( s, a ) ∈ S × A , Q ω ∈ Q Ω and w the vector representing Q ω , we write the Bellman iteration as [Equation] = [Equation]
Revised: For Q ω ∈ Q Ω , ω the vector representing Q ω , max a ∈A Q ω ( s, a ) is well defined (here max might not be attained, it should be interpreted as a supremum).

Original: from the low-rank MDP assumption.

Original: = [Equation] s [Equation]
Revised: We have | Q ω

Original: ⟩ from the linearity of the scalar product in its second variable.

Original: = [Equation]
Revised: [Equation]

Original: from the linearity of the scalar product in its first variable.

Original: =

Original: [Equation]

Original: =

Original: [Equation]

Original: = ⟨

Original: [Equation]

Original: ⟩ since ω does not depend on s ′ .

Original: This means that Γ π Q w ∈ Q Ω .

Original: By setting A π = γ (cid:82) s ′ µ ( s ′ )

Original: E a ′ ∈ π ( ·| s ′ )

Original: [ σ ( s ′ , a ′ )] T d s ′ , we obtain that π -PBO exists and

Original: Γ π ϕ ( ω ) =

Original: θ + A π ω .

Original: Proof for Section 4.3: contraction of π -PBO for a low-rank MDP.

Original: We now assume that the MDP has a latent variable representation, which means that the vectors ϕ ( · , · ) are probability distributions over { 1 , . . .

Original: , d } and, for every i ∈ { 1 , . . .

Original: , d } , µ ( · ) i is a probability distribution over S .

Original: This proof is inspired by the proof of the contraction property of the Bellman Operator in Bertsekas (2015).

Original: To show the property, it is enough to show that for any vector w ∈ R d such that ∥ w ∥ ∞ = 1 ,

Original: ∥

Original: A π w ∥ ∞ ≤ γ .

Original: If this is true, then for ( w , w ′ )

Original: ∈ R

Original: d such that w ̸ = w ′ , || Γ π ϕ ( w ) − Γ π ϕ ( w ′ )

Original: ||

Original: ∞ =

Original: || A π ( w − w ′ ) || ∞ = ||
Revised: s, a ) | = |⟨ σ

Original: A π ( w − w ′ )

Original: ||

Original: w − w ′ ||

Original: ∞

Original: || ∞ · || w − w ′ || ∞

Original: ≤ γ || w − w ′ || ∞ .

Original: Considering w ∈ R d , such that || w || ∞

Original: =

Original: 1 , we have

Original: [Equation]

Original: since µ

Original: [Equation]
Revised: [Equation]

Original: and ϕ

Original: [Equation]

Original: · ) are positive.

Original: [Equation]

Original: ≤ γ max i ∈{ 1 ,...,d } (cid:90) s ′ µ ( s ′ )

Original: i E a ′ ∈ π ( ·| s ′ )   (cid:88)

Original: j ∈{ 1 ...,d } σ ( s ′ , a ′ )

Original: j   d s ′ by linearity of the

Original: expectation

Original: and

Original: || w

Original: || ∞

Original: = 1 .

Original: ≤ γ max i ∈{ 1 ,...,d } (cid:90) s ′ µ ( s ′ )

Original: i d s ′ since ϕ ( · , · ) are probability distributions.

Original: ≤ γ since µ ( · ) i is a probability distribution for all i . Proof for Section 4.3: existence of the optimal PBO for a low-rank MDP. The proof considers continuous unbounded state action spaces. For Q w ∈ Q Ω , w the vector representing Q w , max a ∈A Q w ( s, a ) is well defined (here max might not be attained, it should be interpreted as a supremum).
Revised: s, a ) | ω ⟩| ≤ || σ ( s, a ) || · || ω || ≤ || ω

Original: We have | Q w ( s, a ) | = |⟨ σ ( s, a ) | w ⟩| ≤ || σ ( s, a ) || · || w || ≤ || w || , thus max a ∈A Q w ( s, a ) < ∞ .
Revised: || , thus max a ∈A Q ω ( s, a ) < ∞ .

Original: Then, we write the optimal Bellman iteration on the function Q w asΓ ∗ Q w ( s, a ) = r ( s, a ) + γ E s ′ ∼ p ( ·| s,a ) [Equation]
Revised: Then, we write the optimal Bellman iteration on the function Q ω as

Original: = r ( s, a ) + γ

Original: [Equation][Equation]
Revised: [Equation]

Original: s ′ from the definition of the transition probabilities.
Revised: from the definition of the transition probabilities.

Original: [Equation]
Revised: [Equation]

Original: from the linear of the scalar product. This derivation shows thatΓ ∗ Q w ∈ Q Ω and that the optimal PBO exists and can be computed in closed form.
Revised: This derivation shows that the operator ω (cid:55)→ θ + γ (cid:82) s ′ max a ′ ⟨ σ ( s ′ , a ′ ) | ω ⟩ µ ( s ′ ) d s ′ minimizes the objective function presented in the definition of PBO 2 since its yields zero error. This operator is the PBO for a low-rank MDP.

Original: Proof for Section 4.3: class of the optimal PBO for a low-rank MDP.
Revised: Intuition on the architecture of the parameterized PBO for a low-rank MDP.

Original: In the finite case, theoptimal
Revised: In the finite case, the

Original: PBO is
Revised: PBO is

Original: [Equation]
Revised: [Equation]

Original: We can see that, by defining A 1 , A 2 and b as proposed in Section 4.3, the optimal PBO belongs to the class of lin-max-lin PBOs.
Revised: We can see that the most suitable architecture is a composition of a linear layer followed by a maxpooling and finally a last linear layer.

Original: Proof for Section 4.3: contraction of theoptimal PBO  for a low-rank MDP.
Revised: Proof that the PBO is a γ -contraction-mapping for a low-rank MDP.

Original: We now assume that the MDP has a latent variable representation.
Revised: We now assume that the MDP has a latent variable representation.

Original: This proof was inspired by the proof of the contraction property of the Bellman Operator in Bertsekas (2015).
Revised: This proof was inspired by the proof of the contraction property of the Bellman Operator in Bertsekas (2015).

Original: Considering w [Equation] w ′ ∈R d [Equation]
Revised: Considering ω , ω ′ ∈

Original:  we have
Revised: R d , we have

Original: [Equation]
Revised: [Equation]

Original: since µ(
Revised: since µ

Original: [Equation]
Revised: [Equation]

Original: ) is positive

Original: [Equation]
Revised: [Equation]

Original: since ϕ
Revised: positive

Original: [Equation][Equation]
Revised: [Equation]

Original: are positive
Revised: d

Original: [Equation]
Revised: [Equation]

Original: { 1 ,...,d } (cid:90) s ′ µ ( s ′ ) 
Revised: since σ ( · , · ) are positive.

Original: i d s ′ · || w − w ′ || ∞ since ϕ ( · , · ) are probability distributions. ≤ γ || w − w ′ || ∞ since µ ( · ) i is a probability distribution for all i . Proof for Section 4.4: optimal Q function for LQR MDP.
Revised: [Equation]

Original: First, note that the LQR setting provides a closed-form solution for the optimal value function: V ∗ (s ) = P · s 2 with P the solution of the Riccati equation. From there, we can compute the value function in closed-form
Revised: since σ ( · , · ) are probability distributions.

Original: [Equation]
Revised: [Equation]

Original: Proof for Section 4.4: non stability of Q Ω for LQR MDP. Let π be a policy, w ∈ R 3 , the Bellman iteration over Q w is [Equation] by definition of s ′ .
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: Note that we can take, for example, any policy with a constant momentum of order 2 for the state s ′ (i.e. E a ′ ∼ π ( ·| s ′ ) [ a ′ 2 ] is constant) and then Γ π Q w ̸∈ Q Ω .This means that the space of value functions Q Ω is not stable by the Bellman operator associated with the policies that have this property.
Revised: since µ ( · ) i is a probability distribution for all i .

Original: Prooffor Section 4.4: existence of π -PBO for greedy policies for LQR MDP.
Revised: Proof of the closed-form of the PBO for LQR MDP.

Original: Let w =
Revised: Let ω =

Original: ( G, I,
Revised: ( G, I,

Original: M ) ∈ R 2 × R −∗ 5 and let π Q w be a greedy policy with respect to Q w .

Original: Since Q w is a quadratic polynomial with a negative coefficient in front of the monomial of degree 2 , for any s ∈ S , π Q w ( s )

Original: = − I / M · s .

Original: We note C = − I / M .

Original: From the derivations of the previous proof, for w = ( G, I,

Original: M ) ∈ R 2 × R −∗  , the Bellman iteration over Q w is [Equation] [Equation]
Revised: M ) ∈ R 2 × R −∗ 3 , the optimal

Original: since π Q w is deterministic.

Original: [Equation]

Original: This shows the stability of Q Ω by the Bellman operator since Γ π Q [Equation] Q w ∈ Q Ω . We have Γ π Q w ϕ ( w )= (cid:34)
Revised: Bellman iteration over Q ω is Γ ∗ Q ω ( s, a )

Original: QSR (cid:35) +  

Original: A 2 ( G + 2 IC + MC 2 )

Original: AB ( G + 2 IC + MC 2 )

Original: B 2 ( G + 2 IC + MC 2 ) 

Original:  =

Original: b + A π Q w · w or = (cid:34)

Original: QSR (cid:35) + ξ ( w )  

Original: A ABB  , with A π Q w =(cid:2) A AB
Revised: =

Original: B 2 (cid:3) T (cid:2) 1 2 C

Original: C 2 (cid:3) , b = [ Q S R ] T and ξ ( w )= (cid:2) 1 2 C C 2 (cid:3) w .
Revised: r ( s, a ) + max a ′ Q ω ( s ′ , a ′ ) .

Original: Remarks This result makes the evaluation of a greedy policy straightforward.

Original: One only needs to invert a matrix to get the value function corresponding to a greedy policy.

Original: Those parameters w ∗ satisfy the equation Γ π Q w Q w ∗ ( s, a ) =

Original: Q w ∗ ( s, a ) , which means that Γ π Q w ϕ ( w ∗ ) =

Original: w ∗ since Γ π Q w Q w ∗ ( s, a ) =

Original: Q Γ πQ w ϕ ( w ∗ ) ( s, a ) .

Original: Thus, we have Q π Q w = Q ( I 3 − A πQ w ) − 1 b .

Original: This operator can be understood in a geometrical way.

Original: By looking at the rightmost equation, we see that starting from any point in Ω , it projects the points along a line of direction vector (cid:2)

Original: A

Original: AB

Original: B 2 (cid:3) T with an offset [ Q S

Original: R ] T .

Original: The iterations correspond to a linear transformation of the coefficient ξ ( w ) in front of the direction vector.

Original: This means that all π -PBOs are iterating along the same line since the direction vector and the offset are not depending on the policy.

Original: Proof for Section 4.4: existence of the optimal PBO for LQR MDP.

Original: Let w =

Original: ( G, I,

Original: M ) ∈ R 2 ×

Original: R −∗ 6 , be the optimal Bellman iteration over Q

Original: [Equation] Before moving forward, we have to make sure this quantity is properly defined.
Revised: Before moving forward, we have to make sure this quantity is properly defined.

Original: The function a ′ (cid:55)→ Q w ( s ′ , a ′ ) =
Revised: The function a ′ (cid:55)→ Q ω ( s ′ , a ′ ) =

Original: G · s ′ 

Revised: G · s ′

Original: + 2 I · s ′ a ′ + M · a ′ 2 is a polynomial of degree 2 in a ′ .
Revised: + 2 I · s ′ a ′ + M · a ′ 2 is a polynomial of degree 2 in a ′ .

Original: By choosing M < 0 , we ensure that this function has a unique maximum of equation − I / M · s ′ .
Revised: By choosing M < 0 , we ensure that this function has a unique maximum of equation − I / M · s ′ .

Revised: This is why the PBO is only defined on R× R −∗ ∪ R × { 0 } × { 0 } .

Original: This makes
Revised: This makes

Original: [Equation]
Revised: [Equation]

Original: By putting it in the Bellman equation, we get
Revised: By putting it in the Bellman equation, we get

Original: [Equation]
Revised: [Equation]

Original: Then, the following operator is theoptimal PBO
Revised: Then, the following operator is the PBO

Original: [Equation]
Revised: [Equation]

Original: with ξ ( w ) =
Revised: with Λ( ω ) =

Original: G − I 2 / M .
Revised: G − I 2 / M .

Original: We have Γ ∗ Q w ∈ Q Ω , hence the stability of Q Ω .

Original: Remarks Theoptimal PBO can also be understood in a geometrical way.
Revised: Remarks The PBO can also be understood in a geometrical way.

Original: Just like the πPBO, it projects the parameters along a line of direction vector (cid:2)
Revised: It projects the parameters along a line of direction vector (cid:2)

Original: A
Revised: A

Original: AB
Revised: AB

Original: B 2 (cid:3) T with an offset [ Q S
Revised: B 2 (cid:3) T with an offset [ Q S

Original: R ] T .
Revised: R ] T .

Original: Theonly difference is that the iterations correspond to a non-linear transformation of the coefficient ξ ( w ) .
Revised: The iterations correspond to a non-linear transformation of the coefficient in front of the direction vector.

Original: This  means thatall π -PBOs and the optimal PBO are iterating on thesame line. With this geometrical representation, we recover the fact that an optimal Bellman iteration on a value function corresponds to the Bellman iteration of the greedy policy of this value function. Since all the Bellman operators iterate on the same line, their fixed points are also on this line.Those fixed points correspond to the iteration of the optimal Bellman operator and we have just shown that the optimal Bellman operator iterates on this exact same line 7 .
Revised: This also means that the fixed-point, i.e. the optimal parameters are also on this line.

Original: B D ETAILS OF THE BASELINES
Revised: B D ETAILS OF THE BASELINES

Revised: Algorithm

Original:  Inputs:
Revised: 1: Inputs:

Original:  K λ # learning rate  2: Initialize: w 0 # weight that give the null Q function 3: for i in 1 . . .
Revised: Dataset of samples D #Bellman iterations K #fitting steps H learning rate λ 2: Initialize: ω 0 # weight that give the null Q function 3: for i in 1 . . .

Original: K do 4: # Compute the target T 5: for ( s j , a j , r j , s ′ j ) ∈
Revised: K do 4: # Compute the target T 5: for ( s j , a j , r j , s ′ j ) ∈

Original: D do 6:
Revised: D do 6:

Original: T ( s j , a j ) =
Revised: T ( s j , a j ) =

Original: r j + γ max a Q w i − 1 ( s ′ j , a ) 7: end for 8: 9: # Minimize the loss starting from the previous weights 10: w i = w i −11: for in 1 . . .
Revised: r j + γ max a Q ω i − 1 ( s ′ j , a ) 7: end for 8: 9: # Minimize the loss starting from the previous weights 10: ω i = ω i −11: for in 1 . . .

Original: fitting steps do 12: for batch ∈ D do 13: w i ← w i − λ (cid:88)
Revised: H do 12: for batch ∈ D do 13: ω i ← ω i − λ (cid:88)

Original: j ∈ batch ∇ w ( T ( s j , a j ) − Q w i ( s j , a j ))14: end for 15: end for 16: end for 17: return w K
Revised: j ∈ batch ∇ ω ( T ( s j , a j ) − Q ω i ( s j , a j )) 14: end for 15: end for 16: end for 17: return ω K

Original:  Kσ # features  2: Initialize: w 0# weight that give the null Q function 3: for i in 1 . . .
Revised: Algorithm 3 LSPI 1: Inputs: Dataset of samples D #Bellman iterations K features σ 2: Initialize: Weights ω 0 that give the null Q function 3: for i in 1 . . .

Original: K do 4:
Revised: K do 4:

Original: # Critic step: policy improvement 5: for ( , , , s ′ j ) ∈ D do 6: π ( s ′ j ) =
Revised: # Critic step: policy improvement 5: for ( , , , s ′ j ) ∈ D do 6: π ( s ′ j ) =

Original: arg max a σ ( s ′ j , a ) T w i −7: end for 8: 9: # Actor step: policy evaluation 10:
Revised: arg max a σ ( s ′ j , a ) T ω i −7: end for 8: 9: # Actor step: policy evaluation 10:

Original: A ←11: b ←12: for ( s j , a j , r j , s ′ j ) ∈ D do 13:
Revised: A ←11: b ←12: for ( s j , a j , r j , s ′ j ) ∈ D do 13:

Original: A ← A + σ ( s j , a j ) · (cid:0) σ ( s j , a j ) − γ σ ( s ′ j , π ( s ′ j )) (cid:1) T 14: b ← b + r j σ ( s j , a j ) 15: end for 16: w i= A
Revised: A ← A + σ ( s j , a j ) · (cid:0) σ ( s j , a j ) − γ σ ( s ′ j , π ( s ′ j )) (cid:1) T 14: b ← b + r j σ ( s j , a j ) 15: end for 16: ω i

Original: 17: end for 18: return w K
Revised: 18: return [Table]

Original: C A DDITIONAL RESULTS AND DETAILS OF THE EMPIRICAL ANALYSIS
Revised: C A DDITIONAL RESULTS AND DETAILS OF THE EMPIRICAL ANALYSIS

Original: We provide details of the experimental setting and additional empirical analysis to better understand the importance of some parameters.
Revised: We provide details of the experimental setting.

Original: Table 2 summarizes the values of all parameters appearing in the experiments.
Revised: Table 1 summarizes the values of all parameters appearing in the experiments.

Original: For the additional empirical analysis, only the parameter at stake is changed, all [Table] these [Table]
Revised: For the additional empirical analysis, only the parameter at stake is changed, all the others remain the ones shown in Table 1.

Original: [Figure]

Original: [Figure] given as input to PQI on chain-walk. the others remain the ones shown in Table 2. FQI and PQI use an Adam optimizer (Kingma & Ba, 2015) with a linear annealing learning rate.
Revised: FQI and PQI use an Adam optimizer (Kingma & Ba, 2015) with a linear annealing learning rate.

Original: For FQI, the optimizer is reset at each iteration.
Revised: For FQI, the optimizer is reset at each iteration.

Original: C. C HAIN - WALK
Revised: C. C HAIN - WALK

Original: We consider all the possible state-action pairs 10 times as the initial dataset of samples D .
Revised: We consider all the possible state-action pairs 10 times as the initial dataset of samples D .

Original: Theserepetitions help the algorithms to grasp the randomness of the environment.
Revised: Theserepetitions help the algorithms to grasp the randomness of the environment.

Original: The set of parameters W is generated by taking a truncated normal distribution for different seeds.

Original: The validation parameter is a table composed of zeros. The following analyses show that the size of the dataset of samples is not important; conversely, the size of the dataset of parametersis crucial. The analysis also show that PQI behaves well when the number of Bellman iterations changes. Analysis of the size of the dataset of samples D .
Revised: The set of parameters

Original: We consider max-lin operators, with the goal of studying how PQI behaves when the number of samples varies. The distance to the optimal value function is shown in Figure 7a. It can be seen that the distances are not visible, which means that the importance of this parameter is not high for this setting. Analysis of the size of the dataset of parameters W .
Revised: W is generated by taking a truncated normal distribution for different seeds.

Original: Ideally, this dataset should contain parameters close to the initialparameter that we use in validation (the parameters with only zeros), others [Figure] [Figure] || V *
Revised: The initial parameters use for the figures is a table composed of zeros.

Original: V i || PBO linear PBO  linear fixed point PBO deep
Revised: The architecture of the neural PBO is a max-pooling layer followed by a linear layer.

Original: FQI LSPI PBO optimal (a) K = 2 . 35 #Iterations
Revised: This choice has been made so that a perfectly tuned parameterized

Original: || V V i ||(b) K = 15 . close to optimal parameters, and others in between.
Revised: PBO can be mathematically equal to the PBO.

Original: Since we only assume to know the validation parameters, we take parameters close to them, in the sense that the truncated normal distribution used for data collection is centered.

Original: Figure 7b shows the distance to the optimal value function for different sizes of W .

Original: Clearly, the more parameters are given to the algorithm, the better it performs.

Original: This shows the importance of the number of parameters given to linear PBOs in this setting.

Original: Analysis on the number of Bellman iterations K .

Original: We additionally analyze the behavior of PBOs for a different number of Bellman iterations than K = 5 .

Original: We consider K = 2 and 15 , observing that the PBOs still behave well as shown in Figure 8.

Original: C. L INEAR QUADRATIC REGULATOR
Revised: C. L INEAR QUADRATIC REGULATOR

Original: The dynamics of the system is P ( s, a ) =0 .
Revised: The dynamics of the system is P ( s, a ) =

Revised: [Equation]

Original: 72 s − 0 .
Revised: 72 s − 0 .

Original: 53 a and the reward function is R ( s, a )= − 0 .
Revised: 53 a and the reward function is R ( s, a )

Original: 13 s 2 + 0 .
Revised: [Equation]

Original: 26 sa − 0 . 80 a 2 . The set D is collected on a mesh of size 11 by 11 on the state-action space going from − 4 to 4 in both directions, which means that samples belong to [ − 4 , 4] ×
Revised: The set D is collected on a mesh of size 11 by 11 on the state-action space going from − 4 to 4 in both directions, which means that samples belong to [ − 4 , 4] ×

Original: [ − 4 , 4] ⊂
Revised: [ − 4 , 4] ⊂

Original: S × A .
Revised: S × A .

Original:  The set W is composed of onlyone weight which is the validation weight, i.e., (0 , 0 , 0) .
Revised: [Figure] The set W is composed of only the initial parameters, i.e., (0 , 0 , 0) .

Original: This choice is made because during the optimization, the optimal Bellman operator might project the weights in an area where the coefficients M are positive which would lead the next optimal Bellman iterations to diverge.
Revised: This choice is made because during the optimization, the optimal Bellman operator might project the parameters in an area where the coefficients M are positive which would lead the next optimal Bellman iterations to diverge.

Original: For PQI and FQI, we use the optimal Bellman operator, meaning that we should compute the maximum over a value function.
Revised: For PQI and FQI, we use the optimal Bellman operator, meaning that we should compute the maximum over a value function.

Original: We choose to parameterize the value functions in a quadratic way, thus allowing us to compute the maximum in closed-form.
Revised: We choose to parameterize the value functions in a quadratic way, thus allowing us to compute the maximum in closedform.

Original: However, we assume that the algorithms does not have this knowledge since it is not the case in more general settings.
Revised: However, we assume that the algorithms do not have this knowledge since it is not the case in more general settings.

Original: For this reason, we discretize the action space in a set of 200 actions going from − 8 to 8 .
Revised: For this reason, we discretize the action space in a set ofactions going from − 8 to 8 .

Revised: The architecture of the neural

Revised: PBO starts with a function ω (cid:55)→ ω 1 − ω 22 ω 3 and is followed by a linear layer.

Revised: Once again, this choice has been made to ensure that the neural PBO can reproduce the behavior of the PBO if it is well trained.

Original: In the following, we explain how to obtain the metric used in Figure 4b.
Revised: In the following, we explain how to obtain the metric used in Figure 7a.

Original: Then, we show that the size of D is not a crucial parameter, as we did for the chain-walk environment (see Appendix C.1).

Original: We also demonstrate that lowering the standard deviation of the initial parameters of PBO is important.The last analysis confirms that PQI behaves well when the number of Bellman iterations changes.
Revised: Then, we demonstrate that lowering the standard deviation of the initial parameters of PBO is important.

Original: Metric.
Revised: Metric.

Original: In this setting, the value functions associated to a greedy policy can be computed in closed form.
Revised: In this setting, the value functions associated to a greedy policy can be computed in closed form.

Original: For a weight w = ( K, I,
Revised: For a weight ω = ( G, I,

Original: M ) ∈ R 2 × R −∗ ∪ R ×{ 0 }× { 0 } such that A − BI / M ̸ = 1 , the value function associated to the greedy policy π w computed from the value function Q w has the following form, for any s ∈ S :
Revised: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } such that A − BI / M ̸ = 1 , the value function associated to the greedy policy π ω computed from the value function Q ω has the following form, for any s ∈ S :

Original: [Equation]
Revised: [Equation]

Original: where P w = Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 
Revised: where P ω = Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 2 .

Original: Note that K does not influence the result, which is normal since the greedy policy is only influenced by the term linked with the actions.
Revised: Note that G does not influence the result, which is normal since the greedy policy is only influenced by the term linked with the actions.

Original: Proof.
Revised: Proof.

Original: Let w =
Revised: Let ω =

Original: ( K, I,
Revised: ( G, I,

Original: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } such that A − BI / M ̸ = 1 , the greedy policy of the value function parameterized by w is π w ( s ) =
Revised: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } such that A − BI / M ̸ = 1 , the greedy policy of the value function parameterized by ω is π ω ( s ) =

Original: − I / M · s where we take the convention/ 0 = 0 .
Revised: − I / M · s where we take the convention/ 0 = 0 .

Original: To show the result, we show that the function s (cid:55)→ P w · s 2 satisfies the optimal Bellman equation 8 .
Revised: To show the result, we show that the function s (cid:55)→ P ω · s 2 satisfies the optimal Bellman equation 4 .

Original: We set P w = Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 2 , and we have ∀s ∈ S , P w · s 2 = Γ ∗ ( x (cid:55)→ P w · x 2 )( s ) optimal Bellman equation for the function x (cid:55)→ P w · x 2 . ⇐⇒ ∀
Revised: We set P ω = Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 2 , and we have ∀

Original: [Equation]
Revised: [Equation]

Original: , P w · 
Revised: , P ω · s 2 =

Original: [Equation]
Revised: Γ ∗ ( x (cid:55)→ P ω · x 2 )( s ) optimal Bellman equation for the function x (cid:55)→ P ω · x 2 . ⇐⇒ ∀ s ∈ S , P ω · s 2 = r ( s, π ω ( s ))

Original: 2 = r
Revised: + P ω · s ′⇐⇒

Original: [Equation]
Revised: [Equation]

Original: s, π w (
Revised: ⇐⇒ [Equation] ⇐⇒

Original: [Equation]
Revised: [Equation]

Revised: ⇐⇒ P

Original: [Equation]
Revised: [Equation]

Original: P w
Revised: RI

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that if we multiply s to both side of the assumption A − BI / M ̸ = 1 , we get s ′ ̸
Revised: Note that if we multiply s to both side of the assumption A − BI / M ̸ = 1 , we get s ′ ̸

Original: = s where s ′ is the next state if the greedy policy is followed.
Revised: = s where s ′ is the next state if the greedy policy is followed.

Original: This means that we cannot compute the value function of greedy policies that leads to steady states.
Revised: This means that we cannot compute the value function of greedy policies that leads to steady states.

Original: This is normal since the value function is not defined for allthe states because the sum of accumulated rewards diverges except in 0 where it has the value of 0 .
Revised: This is normal since this value function is not defined for all states because the sum of accumulated rewards diverges except in 0 where it is 0 .

Revised: [Figure]

Original: We leverage this result to state that the distance between the P w of a weight w and the solution of the Riccati equation is proportional to the distance between the value function of the greedy policy and the optimal value function.
Revised: We leverage this result to state that the distance between the P ω of a set of parameters ω and the solution of the Riccati equation is proportional to the distance between the value function of the greedy policy and the optimal value function.

Original: Proof.
Revised: Proof.

Original: Let b ∈ R .
Revised: Let b ∈ R .

Original: We consider the norm- 2 over the functions going from [ − b, b ] to R .
Revised: We consider the norm- 2 over the functions going from [ − b, b ] to R .

Original: For w ∈ R× R −∗ ∪ R × { 0 } × { 0 } , we have
Revised: For ω ∈ R× R −∗ ∪ R × { 0 } × { 0 } , we have

Original: [Equation]
Revised: [Equation]

Original: The norm of the difference of the value functions is proportional to the difference of P coefficients.Analysis on the size of the dataset of samples D .
Revised: The norm of the difference of the value functions is proportional to the difference of P coefficients.

Original: We focus on the Λ -lin PBO.

Original: Figure 9a shows the distance to the optimal value function for trainings with a varying number of samples.

Original: When the grid becomes thinner, the distances are decreasing.

Original: However this decrease is not strong. This is why the size of the dataset is not a crucial parameter for PBO. Analysis on the standard deviation of the PBO’s initial parameters.
Revised: Analysis on the standard deviation of parameterized PBO’s initial parameters.

Original: Figure 9b shows the distance to the optimal value function for linear PBOs that have been initialized with different standard deviations.
Revised: Figure 13 shows the distance to the optimal value function for linear PBOs that have been initialized with different standard deviations.

Original: The linear PBO initialized with smaller standard deviation shows better performance.
Revised: The linear PBO initialized with smaller standard deviation shows better performance.

Original: This behavior can be explained by the following two reasons.
Revised: This behavior can be explained by the following two reasons.

Original: First, a linear PBO is very limited in terms of freedom, meaning that if it is initialized in such a way that it goes to the wrong direction, it is very likely that it will stay in the wrong direction during the training; second, by initializing the weights of the  PBO with low value, the operator is already contractive at the beginning of the training.This helps the PBO to have a stable behavior when evaluating on iterations that were not seen during training.
Revised: First, a linear PBO is very limited in terms of freedom, meaning that if it is initialized in such a way that it goes to the wrong direction, it is very likely that it will stay in the wrong direction during the training; second, by initializing the parameters of the parameterized PBO with low value, the operator is already contractive at the beginning of the training.

Original: Analysis on the number of Bellman iterations K .

Original: Since the convergence of all the algorithms is fast in term of iterations, we only show the performance with 4 Bellman iterations instead of 2 .

Original: If we increase the number of Bellman iterations too much, numerical errors arise due to the small distances.

Original: Figures 10a confirms that PBOs perform well for a larger number of Bellman iterations.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] Visualization. This setting allow us to plot the parameters of the action-value functions in a 3D space. The parameters corresponding to the iterations of PQI and FQI are shown in Figure 10b.
Revised: This helps the parameterized PBO to have a stable behavior when evaluating on iterations that were not seen during training.

Original: The training is done with K = 2 .

Original: The first iteration for the three algorithms are on (0 , 0 , 0) and then, the iterations goes towards the optimal parameters on the bottom right of the figure.

Original: In this case, being able to iterate after the training clearly makes the difference, as it is clear that PQI is able to get closer to the optimal parameters after the second iteration.

Original: C. C AR - ON - HILL
Revised: C. C AR - ON - HILL

Original: The goal is to bring a car up a hill.
Revised: The goal is to bring a car up a hill.

Original: The agent chooses between 2 actions: left or right .
Revised: The agent chooses between 2 actions: left or right .

Original: The state space is 2 -dimensional: position in [ − 1 , 1] and velocity [ − 3 , 3] .
Revised: The state space is 2 -dimensional: position in [ − 1 , 1] and velocity [ − 3 , 3] .

Original: If the agent succeeds to bring the car up the hill – at position greater than 1 and velocity in between − 3 and 3 – then the reward is 1 , if the agent exceeds the state space, the reward is − 1 ; otherwise, the reward is 0 .
Revised: If the agent succeeds to bring the car up the hill – at position greater than 1 and velocity in between − 3 and 3 – then the reward is 1 , if the agent exceeds the state space, the reward is − 1 ; otherwise, the reward is 0 .

Original: As our PQI is an offline algorithm, we need to make sure that all the necessary exploration has been done in the dataset of samples.
Revised: As our PQI is an offline algorithm, we need to make sure that all the necessary exploration has been done in the dataset of samples.

Original: For that reason, we first consider a uniform sampling policy to collect episodes starting from the lowest point in the map ( [ − 0 .
Revised: For that reason, we first consider a uniform sampling policy to collect episodes starting from the lowest point in the map ( [ − 0 .

Original: 5 , 0] ) with an horizon of 100 .
Revised: 5 , 0] ) with an horizon of 100 .

Original: This sampling process is stopped when 4 .
Revised: This sampling process is stopped when 4 .

Original: 500 samples are gathered.
Revised: 500 samples are gathered.

Original: To get more samples with positive reward, we sample new episodes starting from a state located between [0 . 1 , 1 . 3] and [0 . 5 , 0 .
Revised: To get more samples with positive reward, we sample new episodes starting from a state located between [0 . 1 , 1 . 3] and [0 . 5 , 0 .

Original: 38] with a uniform policy as well.
Revised: 38] with a uniform policy as well.

Original: In total, 5 . 000 samples are collected.
Revised: In total, 5 . 000 samples are collected.

Original: The sample and reward distributions over the state space is shown in Figure 11.
Revised: The sample and reward distributions over the state space is shown in Figure 14.

Original: The set of weights W is composed of only a single validation weight, because the complexity of the classes of PBOs used in the experiment is not high.
Revised: Inspired by the most suitable architecture for a parameterized [Figure] || Q * Q i ||

Original: The
Revised: [Figure] 25 #Iterations    || V * V i ||

Original: [Figure]
Revised: [Figure]

Revised: PBO, see Appendix A, we set the architecture of the neural PBO with a first block composed of a linear layer followed by a rectified linear unit (ReLU) and another linear layer.

Revised: Then, the first block is followed by a max-pooling before adding a second block that is identical to the first one.

Revised: The set of parameters W is composed of only the initial parameters, because the complexity of the classes of PBOs used in the experiment is not high.

Original: validation weight is a random weight where the weights for the last layer are only zeros, ensuring that the associated value function only outputs zeros.
Revised: The initial parameters are random except for the last layer that is set to zero, ensuring that the associated value function only outputs zeros.

Original: To help the training, the value functions are taking actions in {− 1 , 1 } instead of the usual { 0 , 1 } .
Revised: To help the training, the value functions are taking actions in {− 1 , 1 } instead of the usual { 0 , 1 } .

Original: We also use a low standard deviation for the initial parameters of the PBOs as described in Appendix C.2.
Revised: We also use a low standard deviation for the initial parameters of the PBOs as described in Appendix C.2.

Original: Metric.
Revised: Metric.

Original: Ernst et al.
Revised: Ernst et al.

Original: (2005) measure the performance of the agents by performing rollouts starting from different stateslocated on a grid of size 17 × 17 .
Revised: (2005) measure the performance of the agents by performing roll-outs starting from different states on a grid of size 17 × 17 .

Original: Since the policies, the reward, and the dynamics, are deterministic, the discounted reward obtained at the end of the rollouts corresponds to the value function associated with the policy used during the rollouts.
Revised: Since the policies, the reward, and the dynamics are deterministic, the discounted reward obtained at the end of the roll-outs corresponds to the value function associated with the policy used during the roll-outs.

Original: To compute the optimal value function, they run a brute force algorithm that tries every possible sequence of actions until it finds the shortest way to go up hill.
Revised: To compute the optimal value function, they run a brute force algorithm that tries every possible sequence of actions until it finds the shortest way to go up hill.

Original: In our case, the agents do not have all the transitions in D corresponding to every starting states of the grid they are using, as shown in Figure 11a.
Revised: In our case, the agents do not have all the transitions in D corresponding to every starting states of the grid they are using, as shown in Figure 14a.

Original: This is why we apply a mask on the grid to weight the starting states by the number of time a state has been visited in the dataset.Policies.
Revised: This is why we apply a mask on the grid to weight the starting states by the number of time a state has been visited in the dataset.

Original: Figure 12 shows the policies for FQI and PQI lin-max-lin .

Original: Those policies reflect the average choice among the seeds.

Original: We believe that the reason why PQI lin-max-lin outperforms

Original: FQI is because, on average, the agent chooses to go more to the right when it is on the upper right corner of the state space.

Original: This can be seen in Figure 12a, where a darker green color is shown in that area.

Original: The agent has to choose action right to be able to climb up the hill.

Original: Analysis on the number of Bellman iterations K .
Revised: Analysis on the number of Bellman iterations K .

Original: Figure 13 shows that PQI outperforms FQI at iteration 15 , for K = 15 .
Revised: Figure 15 shows that PQI outperforms FQI at iteration 15 , for K = 15 .

Original: Similarly to the case K = 9 , the linear PBO is too simple to behave well after the training iterations.
Revised: Similarly to the case K = 9 , the linear PBO is too simple to behave well after the training iterations.

Original: Once again, the way the loss is shaped allows PQI to outperform FQI.
Revised: Once again, the way the loss is shaped allows PQI to outperform FQI.

Original: C. B ICYCLE
Revised: C. B ICYCLE

Original: We consider the bicycle problem, as described in Randlov & Alstrøm (1998).
Revised: We consider the bicycle problem, as described in Randlov & Alstrøm (1998).

Original: The state space is composed of 4 dimensions: ( ω, ˙ ω, θ, ˙ θ ) where ω is the angle between the floor and the bike, and θ is the angle between the handle bar and the perpendicular axis to the bike.
Revised: The state space is composed of 4 dimensions: ( ω, ˙ ω, θ, ˙ θ ) where ω is the angle between the floor and the bike, and θ is the angle between the handle bar and the perpendicular axis to the bike.

Original: The goal is to ride a bicycle for 500 seconds ( 50 . 000 steps).
Revised: The goal is to ride a bicycle for 500 seconds ( 50 . 000 steps).

Original: The agent can apply a torque T ∈ {− 2 , 0 , 2 } to the handle bar to make it rotate.
Revised: The agent can apply a torque T ∈ {− 2 , 0 , 2 } to the handle bar to make it rotate.

Original: The agent can also move its center of gravity in the direction d ∈ {− 0 .
Revised: The agent can also move its center of gravity in the direction d ∈ {− 0 .

Original: 02 } perpendicular to the bike.
Revised: 02 } perpendicular to the bike.

Original: As in Lagoudakis & Parr (2003), the agent chooses between applying a torque or moving its center of gravity, resulting in 5 actions instead of 9 .
Revised: As in Lagoudakis & Parr (2003), the agent chooses between applying a torque or moving its center of gravity, resulting in 5 actions instead of 9 .

Original: Usually, a uniform noise in the interval [ − 0 .
Revised: Usually, a uniform noise in the interval [ − 0 .

Original: 02] is added to d .
Revised: 02] is added to d .

Original: For the purpose of this work, we reduce the number of samples by making the magnitude of the noise 10 times smaller.
Revised: For the purpose of this work, we reduce the number of samples by making the magnitude of the noise 10 times smaller.

Original: A reward of − 1 is given when the bike falls down, i.e., | ω | > 12 ◦ .
Revised: A reward of − 1 is given when the bike falls down, i.e., | ω | > 12 ◦ .

Original: We also use reward shaping to have more informative samples, and we add a reward proportional to the change in ω , i.e., 10 4 ( | ω t | − | ω t +1 | ) , as in Lagoudakis &
Revised: We also use reward shaping to have more informative samples, and we add a reward proportional to the change in ω , i.e., 10 4 ( | ω t | − | ω t +1 | ) , as in Lagoudakis &

Original: Parr (2003).
Revised: Parr (2003).

Original: The dataset of samples is composed of 3 .
Revised: The dataset of samples is composed of 3 .

Original: 500 episodes starting from a position close to (0 , 0 , 0 , 0) and cut after 20 steps (Lagoudakis & Parr, 2003).
Revised: 500 episodes starting from a position close to (0 , 0 , 0 , 0) and cut after 20 steps (Lagoudakis & Parr, 2003).

Revised: The architecture of the neural PBO is the same as the one used in the Car-On-Hill experiment.

Original: To evaluate the value functions, we simulate 10 trajectories.
Revised: To evaluate the value functions, we simulate 10 trajectories.

Original: We observed only two kinds of outcomes in our experiments: either the policies deteriorate the performance quickly, or they keep it stable, as shown in Figure 14a.
Revised: We observed only two kinds of outcomes in our experiments: either the policies deteriorate the performance quickly, or they keep it stable, as shown in Figure 11b.

Original: This also explains the big confidence interval for deep PBO in Figure 5b.
Revised: This also explains the big confidence interval for deep PBO in Figure 11a.

Original: Moreover, Figure 14b shows the cumulative discounted reward obtained in the same experiment, where PQI outperforms FQI.