Original: P ARAMETERIZED PROJECTED B ELLMAN OPERATOR
Revised: P ARAMETERIZED PROJECTED B ELLMAN OPERATOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The Bellman operator is a cornerstone of reinforcement learning, widely used in a plethora of works, from value-based methods to modern actor-critic approaches.
Revised: The Bellman operator is a cornerstone of reinforcement learning, widely used in a plethora of works, from value-based methods to modern actor-critic approaches.

Original: In problems with unknown models, the Bellman operator requires transition samples that strongly determine its behavior, as uninformative samples can result in negligible updates or long detours before reaching the fixed point.
Revised: In problems with unknown models, the Bellman operator requires transition samples that strongly determine its behavior, as uninformative samples can result in negligible updates or long detours before reaching the fixed point.

Original: In this work, we introduce the novel idea of obtaining an approximation of the Bellman operator, which we call projected Bellman operator (PBO).
Revised: In this work, we introduce the novel idea of obtaining an approximation of the Bellman operator, which we call projected Bellman operator (PBO).

Original: Our PBO is an operator on the parameter space of a given value function.
Revised: Our PBO is an operator on the parameter space of a given value function.

Original: Given the parameters of a value function, PBO outputs the parameters of a new value function and converges to a fixed point in the limit, as a standard Bellman operator.
Revised: Given the parameters of a value function, PBO outputs the parameters of a new value function and converges to a fixed point in the limit, as a standard Bellman operator.

Original: Notably, our PBO can obtain approximated repeated applications of the true Bellman operator, as opposed to the sequential nature of the standard Bellman operator.
Revised: Notably, our PBO can obtain approximated repeated applications of the true Bellman operator, as opposed to the sequential nature of the standard Bellman operator.

Original: We show how to obtain PBOs for representative classes of RL problems, and how to approximate it resorting to neural network regression.
Revised: We show how to obtain PBOs for representative classes of RL problems, and how to approximate it resorting to neural network regression.

Original: Eventually, we propose an approximate value-iteration algorithm to learn PBOs and empirically evince how it can overcome the limitations of classical methods, opening up multiple research directions as a novel paradigm in reinforcement learning.
Revised: Eventually, we propose an approximate value-iteration algorithm to learn PBOs and empirically evince how it can overcome the limitations of classical methods, opening up multiple research directions as a novel paradigm in reinforcement learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Value-based reinforcement learning (RL) is a popular class of algorithms for solving sequential decision-making problems with unknown dynamics (Sutton & Barto, 2018).
Revised: Value-based reinforcement learning (RL) is a popular class of algorithms for solving sequential decision-making problems with unknown dynamics (Sutton & Barto, 2018).

Original: For a given problem, value-based algorithms aim at obtaining the most accurate estimate of the expected return from each state, i.e., a value function.
Revised: For a given problem, value-based algorithms aim at obtaining the most accurate estimate of the expected return from each state, i.e., a value function.

Original: For instance, the well-known value-iteration algorithm computes value functions by iterated applications of the Bellman operator (Bellman, 1966), of which the true value function is the fixed point.
Revised: For instance, the well-known value-iteration algorithm computes value functions by iterated applications of the Bellman operator (Bellman, 1966), of which the true value function is the fixed point.

Original: Although the Bellman operator can be applied exactly in dynamic programming, it needs to be estimated from samples at each application when dealing with unknown models of RL problems, i.e., empirical Bellman operator (Watkins, 1989;
Revised: Although the Bellman operator can be applied exactly in dynamic programming, it needs to be estimated from samples at each application when dealing with unknown models of RL problems, i.e., empirical Bellman operator (Watkins, 1989;

Original: Bertsekas, 2019).
Revised: Bertsekas, 2019).

Original: Intuitively, the dependence of value iteration on the samples has an impact on the efficiency of the algorithms and on the quality of the obtained estimated value function, which becomes accentuated when solving continuous problems that require value-based methods with function approximation, e.g., approximate value iteration (AVI) (Munos, 2005; Munos &
Revised: Intuitively, the dependence of value iteration on the samples has an impact on the efficiency of the algorithms and on the quality of the obtained estimated value function, which becomes accentuated when solving continuous problems that require value-based methods with function approximation, e.g., approximate value iteration (AVI) (Munos, 2005; Munos &

Original: Szepesv´ari, 2008).
Revised: Szepesv´ari, 2008).

Original: Moreover, in AVI approaches, costly function approximation steps are needed to project the output of the Bellman operator back to the considered action-value functional space.
Revised: Moreover, in AVI approaches, costly function approximation steps are needed to project the output of the Bellman operator back to the considered action-value functional space.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: In this paper, we tackle these limitations by introducing the novel approach of using samples to obtain a new operator, which we call projected Bellman operator (PBO).
Revised: In this paper, we tackle these limitations by introducing the novel approach of using samples to obtain a new operator, which we call projected Bellman operator (PBO).

Original: Our PBO is a function Λ : Ω → Ω defined on parameters ω ∈ Ω of the value function.
Revised: Our PBO is a function Λ : Ω → Ω defined on parameters ω ∈ Ω of the value function.

Original: Contrarily to the standard (empirical) Bellman operator, which uses action-value functions Q ω k to compute targets that are then projected to obtain Q ω k +1 , our PBO uses the parameters of the action-value function to compute updated parameters
Revised: Contrarily to the standard (empirical) Bellman operator, which uses action-value functions Q ω k to compute targets that are then projected to obtain Q ω k +1 , our PBO uses the parameters of the action-value function to compute updated parameters

Original: ω k
Revised: ω k

Original: +

Revised: +

Original: =
Revised: =

Original: Λ( ω k ) directly (Figure 1).
Revised: Λ( ω k ) directly (Figure 1).

Original: The crucial advantages of our approach are twofold: (i) the output of PBO always belongs to the considered action-value functional space, avoiding, therefore, the costly projection step typical when using the Bellman operator, and (ii) once learned, PBO is applicable for an arbitrary number of iterations without using further samples, as visualized in Figure 2.
Revised: The crucial advantages of our approach are twofold: (i) the output of PBO always belongs to the considered action-value functional space, avoiding, therefore, the costly projection step typical when using the Bellman operator, and (ii) once learned, PBO is applicable for an arbitrary number of iterations without using further samples, as visualized in Figure 2.

Original: Starting from initial parameters ω 0 , AVI approaches obtain consecutive approximations of the value function Q ω k by
Revised: Starting from initial parameters ω 0 , AVI approaches obtain consecutive approximations of the value function Q ω k by

Original: [Figure]
Revised: [Figure]

Original: Q ∗ and Q ω ∗ are respectively the optimal value function and its projection on the parametric space applying the Bellman operator iteratively over samples (Figure 2b).
Revised: Q ∗ and Q ω ∗ are respectively the optimal value function and its projection on the parametric space applying the Bellman operator iteratively over samples (Figure 2b).

Original: Instead, our PBO makes use of the samples to learn the operator only.
Revised: Instead, our PBO makes use of the samples to learn the operator only.

Original: Then, starting from initial parameters ω 0 , PBO can produce a chain of updated parameters of arbitrary length (blue lines in Figure 2a) without requiring further samples.
Revised: Then, starting from initial parameters ω 0 , PBO can produce a chain of updated parameters of arbitrary length (blue lines in Figure 2a) without requiring further samples.

Original: In the following, after formally introducing PBO and a novel algorithm for value estimation based on it, we analyze its advantageous properties for different classes of problems.
Revised: In the following, after formally introducing PBO and a novel algorithm for value estimation based on it, we analyze its advantageous properties for different classes of problems.

Original: Thus, our contribution is threefold: (i) we introduce the notion of projected Bellman operator (PBO); (ii) we show how to derive different PBOs according to the class of problems at hand; (iii) we develop a novel algorithm for value estimation based on PBO and show its advantages over related baselines on several RL problems.
Revised: Thus, our contribution is threefold: (i) we introduce the notion of projected Bellman operator (PBO); (ii) we show how to derive different PBOs according to the class of problems at hand; (iii) we develop a novel algorithm for value estimation based on PBO and show its advantages over related baselines on several RL problems.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Our work is, to the best of our knowledge, the first attempt to obtain a variant of the Bellman operator that acts on the parameters of action-value functions.
Revised: Our work is, to the best of our knowledge, the first attempt to obtain a variant of the Bellman operator that acts on the parameters of action-value functions.

Original: Nevertheless, several works in the literature proposed variants of the standard Bellman operator to induce some desired behavior.
Revised: Nevertheless, several works in the literature proposed variants of the standard Bellman operator to induce some desired behavior.

Original: Variants of the Bellman operator are widely studied for entropy-regularized MDPs (Neu et al., 2017; Geist et al., 2019; Belousov & Peters, 2019).
Revised: Variants of the Bellman operator are widely studied for entropy-regularized MDPs (Neu et al., 2017; Geist et al., 2019; Belousov & Peters, 2019).

Original: The softmax (Haarnoja et al., 2017; Song et al., 2019), mellowmax (Asadi & Littman, 2017), and optimistic (Tosatto et al., 2019) operators are all examples of variants of the Bellman operator to obtain maximum-entropy exploratory policies.
Revised: The softmax (Haarnoja et al., 2017; Song et al., 2019), mellowmax (Asadi & Littman, 2017), and optimistic (Tosatto et al., 2019) operators are all examples of variants of the Bellman operator to obtain maximum-entropy exploratory policies.

Original: Besides favoring exploration, other approaches address the limitations of the standard Bellman operator.
Revised: Besides favoring exploration, other approaches address the limitations of the standard Bellman operator.

Original: For instance, the consistent Bellman operator (Bellemare et al., 2016) is a modified operator that addresses the problem of inconsistency of the optimal action-value functions for suboptimal actions.
Revised: For instance, the consistent Bellman operator (Bellemare et al., 2016) is a modified operator that addresses the problem of inconsistency of the optimal action-value functions for suboptimal actions.

Original: The distributional Bellman operator (Bellemare et al., 2017) enables to operate on the whole return distribution, instead of its expectation, i.e., the value function (Bellemare et al., 2023).
Revised: The distributional Bellman operator (Bellemare et al., 2017) enables to operate on the whole return distribution, instead of its expectation, i.e., the value function (Bellemare et al., 2023).

Original: Furthermore, the logistic Bellman operator uses a logistic loss to solve a convex linear programming problem to find optimal value functions (Bas-Serrano et al., 2021).
Revised: Furthermore, the logistic Bellman operator uses a logistic loss to solve a convex linear programming problem to find optimal value functions (Bas-Serrano et al., 2021).

Original: Finally, the Bayesian Bellman operator is a method in Bayesian RL to infer a posterior over Bellman operators centered on the true
Revised: Finally, the Bayesian Bellman operator is a method in Bayesian RL to infer a posterior over Bellman operators centered on the true

Original: Bellman operator (Fellows et al., 2021).
Revised: Bellman operator (Fellows et al., 2021).

Original: We point out that our PBO can be seamlessly applied on an arbitrary variant of the standard Bellman operator with just minor adaptations.
Revised: We point out that our PBO can be seamlessly applied on an arbitrary variant of the standard Bellman operator with just minor adaptations.

Original: Operator learning.
Revised: Operator learning.

Original: Literature in operator learning is mostly focused on supervised learning, with methods for learning operators over vector spaces (Micchelli & Pontil, 2005) and parametric approaches for learning non-linear operators (Chen & Chen, 1995), with a resurgence of recent contributions in deep learning.
Revised: Literature in operator learning is mostly focused on supervised learning, with methods for learning operators over vector spaces (Micchelli & Pontil, 2005) and parametric approaches for learning non-linear operators (Chen & Chen, 1995), with a resurgence of recent contributions in deep learning.

Original: For example, Kovachki et al.
Revised: For example, Kovachki et al.

Original: (2021) learn mappings between infinite function spaces with deep neural networks, or Kissas et al.
Revised: (2021) learn mappings between infinite function spaces with deep neural networks, or Kissas et al.

Original: (2022) apply an attention mechanism to learn correlations in the target function for efficient operator learning.
Revised: (2022) apply an attention mechanism to learn correlations in the target function for efficient operator learning.

Original: The literature about operator learning is much larger, but we consider it out of the scope of this work that, to the best of our knowledge, is the first to deal with the original problem of operator learning in RL.
Revised: The literature about operator learning is much larger, but we consider it out of the scope of this work that, to the best of our knowledge, is the first to deal with the original problem of operator learning in RL.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We consider discounted Markov decision processes (MDPs) M = ⟨S , A , P , R , γ ⟩ , where S is a measurable state space, A is a measurable action space, P : S
Revised: We consider discounted Markov decision processes (MDPs) M = ⟨S , A , P , R , γ ⟩ , where S is a measurable state space, A is a measurable action space, P : S

Original: ×
Revised: ×

Original: A → ∆( S ) 1 is the transition kernel of the dynamics of the system, R : S ×
Revised: A → ∆( S ) 1 is the transition kernel of the dynamics of the system, R : S ×

Original: A → R is a reward function and γ ∈ [0 , 1) is a discount factor (Puterman, 1990).
Revised: A → R is a reward function and γ ∈ [0 , 1) is a discount factor (Puterman, 1990).

Original: A deterministic policy π : S → A is a function mapping a state to an action, inducing a value function
Revised: A deterministic policy π : S → A is a function mapping a state to an action, inducing a value function

Original: V π ( s ) ≜ E [ (cid:80) ∞ t =

Revised: V π ( s ) ≜ E [ (cid:80) ∞ t =

Original: γ t R ( S t , π ( S t )) | S 

Revised: γ t R ( S t , π ( S t )) | S

Original: = s ] , i.e., expected cumulative discounted reward following policy π .
Revised: = s ] , i.e., expected cumulative discounted reward following policy π .

Original: Similarly, the action-value function Q π ( s, a ) ≜
Revised: Similarly, the action-value function Q π ( s, a ) ≜

Original: E [ (cid:80) ∞ t =

Revised: E [ (cid:80) ∞ t =

Original: γ t R ( S t , A t ) | S 

Revised: γ t R ( S t , A t ) | S

Original: = s,
Revised: = s,

Original: A 0 = a, A t = π
Revised: A 0 = a, A t = π

Original: ( S t )] is the expected discounted cumulative reward executing action a in state s , following policy π thereafter.
Revised: ( S t )] is the expected discounted cumulative reward executing action a in state s , following policy π thereafter.

Original: RL aims to find an optimal policy π ∗ yielding the optimal value function V ∗ ( s ) ≜ max π : S→A V π ( s ) , ∀ s ∈ S (Puterman, 1990).
Revised: RL aims to find an optimal policy π ∗ yielding the optimal value function V ∗ ( s ) ≜ max π : S→A V π ( s ) , ∀ s ∈ S (Puterman, 1990).

Original: The (optimal) Bellman operator Γ ∗ is a fundamental tool in RL for obtaining optimal policies, and it is defined as
Revised: The (optimal) Bellman operator Γ ∗ is a fundamental tool in RL for obtaining optimal policies, and it is defined as

Original: [Equation]
Revised: [Equation]

Original: It is well-known that Bellman operators are contraction mappings in L ∞ -norm, such that iterating over an action-value function leads to the fixed point Γ ∗ Q ∗ =
Revised: It is well-known that Bellman operators are contraction mappings in L ∞ -norm, such that iterating over an action-value function leads to the fixed point Γ ∗ Q ∗ =

Original: Q ∗ in the limit (Bertsekas, 2015).
Revised: Q ∗ in the limit (Bertsekas, 2015).

Original: We consider the use of function approximation to represent value functions and denote Ω the space of their parameters.
Revised: We consider the use of function approximation to represent value functions and denote Ω the space of their parameters.

Original: Additionally, we define Q Ω =
Revised: Additionally, we define Q Ω =

Original: { Q ω : S ×
Revised: { Q ω : S ×

Original: A → R | ω ∈ Ω } as the set of value functions representable by parameters of Ω .
Revised: A → R | ω ∈ Ω } as the set of value functions representable by parameters of Ω .

Original:  P ROJECTED B ELLMAN OPERATOR
Revised:  P ROJECTED B ELLMAN OPERATOR

Original: The application of the Bellman operator in RL requires transition samples (Equation 1) (Munos, 2003; 2005; Munos & Szepesv ´ ari, 2008).
Revised: The application of the Bellman operator in RL requires transition samples (Equation 1) (Munos, 2003; 2005; Munos & Szepesv ´ ari, 2008).

Original: The impact of this requirement is twofold: (i) the behavior of the Bellman operator strictly depends on the quality of the samples, thus a poor sample selection can result in negligible updates and long detours before convergence; (ii) the direction of the update is determined exclusively by the current samples, meaning that the empirical Bellman operator has no memory of previously observed samples (Bellemare et al., 2016; Agarwal et al., 2019;
Revised: The impact of this requirement is twofold: (i) the behavior of the Bellman operator strictly depends on the quality of the samples, thus a poor sample selection can result in negligible updates and long detours before convergence; (ii) the direction of the update is determined exclusively by the current samples, meaning that the empirical Bellman operator has no memory of previously observed samples (Bellemare et al., 2016; Agarwal et al., 2019;

Original: Fellows et al., 2021).
Revised: Fellows et al., 2021).

Original: Ideally, we want to obtain an operator that mirrors the behavior of the true
Revised: Ideally, we want to obtain an operator that mirrors the behavior of the true

Original: Bellman operator while being independent of the transition samples and extrapolating from previous experience.
Revised: Bellman operator while being independent of the transition samples and extrapolating from previous experience.

Original: Thus, we introduce the idea of an alternative operator, which we call projected Bellman operator (PBO), defined as follows.
Revised: Thus, we introduce the idea of an alternative operator, which we call projected Bellman operator (PBO), defined as follows.

Original: Definition 1.
Revised: Definition 1.

Original: Let Q Ω =
Revised: Let Q Ω =

Original: { Q ω : S ×
Revised: { Q ω : S ×

Original: A → R | ω ∈ Ω } be a function approximation space for the action-value function, induced by the parameter space Ω .
Revised: A → R | ω ∈ Ω } be a function approximation space for the action-value function, induced by the parameter space Ω .

Original: A projected Bellman operator (PBO) is a function Λ ∗ : Ω → Ω , such that
Revised: A projected Bellman operator (PBO) is a function Λ ∗ : Ω → Ω , such that

Original: [Equation]
Revised: [Equation]

Revised: for some state-action distribution ρ and parameter distribution ν .

Original: Note that the symbol Γ ∗ is the regular optimal Bellman operator on Q Ω .
Revised: Note that the symbol Γ ∗ is the regular optimal Bellman operator on Q Ω .

Original: Conversely, PBO is an operator Λ ∗ acting on the space of parameters Ω .
Revised: Conversely, PBO is an operator Λ ∗ acting on the space of parameters Ω .

Original: All proofs of following results can be found in
Revised: All proofs of following results can be found in

Original: Appendix A.
Revised: Appendix A.

Original: 4. F UNCTION APPROXIMATION FOR PROJECTED BELLMAN OPERATORS
Revised: 4. F UNCTION APPROXIMATION FOR PROJECTED BELLMAN OPERATORS

Original: In the following, we show that PBO can be expressed in closed-form for representative classes of
Revised: In the following, we show that PBO can be expressed in closed-form for representative classes of

Original: RL problems: (i) finite MDPs, (ii) low-rank MDPs (Agarwal et al., 2020; Sekhari et al., 2021), and (iii) linear quadratic regulation problems (Bradtke, 1992; Pang & Jiang, 2021).
Revised: RL problems: (i) finite MDPs, (ii) low-rank MDPs (Agarwal et al., 2020; Sekhari et al., 2021), and (iii) linear quadratic regulation problems (Bradtke, 1992; Pang & Jiang, 2021).

Original: 4.1. F INITE M ARKOV DECISION PROCESSES
Revised: 4.1. F INITE M ARKOV DECISION PROCESSES

Original: Let us consider finite state and action spaces of cardinality N and M respectively, and a tabular setting with Ω = R N · M .
Revised: Let us consider finite state and action spaces of cardinality N and M respectively, and a tabular setting with Ω = R N · M .

Original: Intuitively, there is a trivial bijection between Q Ω and Ω , which allows us to write the parameters of the action-value function as Q instead of ω .
Revised: Intuitively, there is a trivial bijection between Q Ω and Ω , which allows us to write the parameters of the action-value function as Q instead of ω .

Original: Proposition 1.
Revised: Proposition 1.

Original: The PBO exists and it is equal to the optimal Bellman operator
Revised: The PBO exists and it is equal to the optimal Bellman operator

Original: [Equation]
Revised: [Equation]

Original: Note that, since the optimal Bellman operator is a γ -contraction mapping for the sup -norm, PBO for finite MDPs (Equation 3) is also a γ -contraction mapping.
Revised: Note that, since the optimal Bellman operator is a γ -contraction mapping for the sup -norm, PBO for finite MDPs (Equation 3) is also a γ -contraction mapping.

Original: 4.1. L OW - RANK M ARKOV DECISION PROCESSES
Revised: 4.1. L OW - RANK M ARKOV DECISION PROCESSES

Original: Low-rank MDPs is a class of problems with two feature maps σ : S ×
Revised: Low-rank MDPs is a class of problems with two feature maps σ : S ×

Original: A → R d and µ : S → R d , such that P ( s ′ | s, a ) = ⟨ σ ( s, a ) , µ ( s ′ ) ⟩ and R ( s, a ) = ⟨ σ ( s, a ) , θ ⟩ , for all ( s, a, s ′ )
Revised: A → R d and µ : S → R d , such that P ( s ′ | s, a ) = ⟨ σ ( s, a ) , µ ( s ′ ) ⟩ and R ( s, a ) = ⟨ σ ( s, a ) , θ ⟩ , for all ( s, a, s ′ )

Original: ∈ S ×
Revised: ∈ S ×

Original: A × S and for θ ∈ R d .
Revised: A × S and for θ ∈ R d .

Original: We assume, without loss of generality, that for all ( s, a ) ∈ S × A , ∥ σ ( s, a ) ∥ 1 ≤and max {∥ µ ( s ) ∥ 1 , ∥ θ ∥ 1 } ≤ √ d .
Revised: We assume, without loss of generality, that for all ( s, a ) ∈ S × A , ∥ σ ( s, a ) ∥ 1 ≤and max {∥ µ ( s ) ∥ 1 , ∥ θ ∥ 1 } ≤ √ d .

Original: We set the space of action-value functions to be the space of linear functions in the parameters, i.e., Q Ω = {⟨ σ ( · , · ) , ω ⟩| ω ∈ R d } .
Revised: We set the space of action-value functions to be the space of linear functions in the parameters, i.e., Q Ω = {⟨ σ ( · , · ) , ω ⟩| ω ∈ R d } .

Original: Proposition 2.
Revised: Proposition 2.

Original: In case of continuous state and action spaces and for ω ∈ R d , the PBO is
Revised: In case of continuous state and action spaces and for ω ∈ R d , the PBO is

Original: [Equation]
Revised: [Equation]

Original: The closed-form is again a γ -contraction mapping under the assumption that the MDP has a latent variable representation.
Revised: The closed-form is again a γ -contraction mapping under the assumption that the MDP has a latent variable representation.

Original: 4.1. L INEAR QUADRATIC REGULATION
Revised: 4.1. L INEAR QUADRATIC REGULATION

Original: Eventually, we consider the continuous MDPs class of linear quadratic regulator (LQR) with
Revised: Eventually, we consider the continuous MDPs class of linear quadratic regulator (LQR) with

Original: S =
Revised: S =

Original: A = R .
Revised: A = R .

Original: The transition model P ( s, a ) =
Revised: The transition model P ( s, a ) =

Original: As + Ba is deterministic and the reward function R ( s, a ) =
Revised: As + Ba is deterministic and the reward function R ( s, a ) =

Original: Qs 2 + 2 Ssa + Ra 2 is quadratic, where A , B Q , S and R , are constants inherent to the MDP.
Revised: Qs 2 + 2 Ssa + Ra 2 is quadratic, where A , B Q , S and R , are constants inherent to the MDP.

Original: As commonly done for LQR, we consider an undiscounted setting, thus we do not analyze the contraction property of PBO.
Revised: As commonly done for LQR, we consider an undiscounted setting, thus we do not analyze the contraction property of PBO.

Original: We choose to parameterize the action-value functions with a 3 -dimensional parameter vector
Revised: We choose to parameterize the action-value functions with a 3 -dimensional parameter vector

Original: [Equation]
Revised: [Equation]

Original: | ( G, I, M ) ∈ R 3 } , for visualization purposes.
Revised: | ( G, I, M ) ∈ R 3 } , for visualization purposes.

Original: Proposition 3.
Revised: Proposition 3.

Original: The PBO exists and for any ω ∈ R 2 × R −∗ 2 its closed form is
Revised: The PBO exists and for any ω ∈ R 2 × R −∗ 2 its closed form is

Original: [Equation]
Revised: [Equation]

Original: A geometrical interpretation of the behavior of this PBO is available in Appendix A.
Revised: A geometrical interpretation of the behavior of this PBO is available in Appendix A.

Original: 4. A PPROXIMATING PROJECTED B ELLMAN OPERATORS
Revised: 4. A PPROXIMATING PROJECTED B ELLMAN OPERATORS

Original: For generic RL problems that do not fall into the previously described categories, the true PBO is unknown and has to be estimated from samples.
Revised: For generic RL problems that do not fall into the previously described categories, the true PBO is unknown and has to be estimated from samples.

Original: We propose to approximate the unknown PBO with a parameterized PBO Λ ϕ differentiable w.r.t.
Revised: We propose to approximate the unknown PBO with a parameterized PBO Λ ϕ differentiable w.r.t.

Original: its parameters ϕ ∈ Φ , enabling the use of gradient descent on a loss function.
Revised: its parameters ϕ ∈ Φ , enabling the use of gradient descent on a loss function.

Original: A first idea would be to learn an operator that minimizes the loss in the definition of PBO (Equation 2).
Revised: A first idea would be to learn an operator that minimizes the loss in the definition of PBO (Equation 2).

Original: While this approach has the benefit of allowing to sample as many parameters as needed from Ω , it does not leverage the crucial fact that approximating an operator allows to iterate over it as many times as desired.
Revised: While this approach has the benefit of allowing to sample as many parameters as needed from Ω , it does not leverage the crucial fact that approximating an operator allows to iterate over it as many times as desired.

Original: Thus, our key insight is to include multiple iterations of the operator to the loss.
Revised: Thus, our key insight is to include multiple iterations of the operator to the loss.

Original: The function to minimize becomes
Revised: The function to minimize becomes

Original: [Equation]
Revised: [Equation]

Original: 6) for all ω ∈ Ω , s ∈ S , and a ∈ A , and where K is an arbitrary number of optimal Bellman operator iterations.
Revised: 6) for all ω ∈ Ω , s ∈ S , and a ∈ A , and where K is an arbitrary number of optimal Bellman operator iterations.

Original: An additional idea is to add a term that corresponds to an infinite number of iterations, i.e., the fixed point, when possible.
Revised: An additional idea is to add a term that corresponds to an infinite number of iterations, i.e., the fixed point, when possible.

Original: Now, the loss becomes
Revised: Now, the loss becomes

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for all ω ∈ Ω , s ∈ S , and a ∈ A , where Λ ωϕ is the fixed point of the parameterized PBO.
Revised: for all ω ∈ Ω , s ∈ S , and a ∈ A , where Λ ωϕ is the fixed point of the parameterized PBO.

Original: Note that the addition of the fixed point term is only possible for classes of parameterized PBOs where the fixed point can be computed and the computation is differentiable in the parameters ϕ .
Revised: Note that the addition of the fixed point term is only possible for classes of parameterized PBOs where the fixed point can be computed and the computation is differentiable in the parameters ϕ .

Original: [Figure]
Revised: [Figure]

Original: Algorithm
Revised: Algorithm

Original: 1: Inputs: samples D = {⟨ s j , a j , r j , s ′ j ⟩} J j =1 , parameters W = { ω l } Ll =1 , #Bellman iterations K , #training steps I , #fitting steps H , operator Λ ϕ to learn with initial parameters ϕ
Revised: 1: Inputs: samples D = {⟨ s j , a j , r j , s ′ j ⟩} J j =1 , parameters W = { ω l } Ll =1 , #Bellman iterations K , #training steps I , #fitting steps H , operator Λ ϕ to learn with initial parameters ϕ

Original: = ϕ 0 , learning rate λ 2: for i ∈ 1 . . .
Revised: = ϕ 0 , learning rate λ 2: for i ∈ 1 . . .

Original: I do 3: # Compute the target T 4: for k ∈ 1 . . .
Revised: I do 3: # Compute the target T 4: for k ∈ 1 . . .

Original: K do 5: for ( ω l , ( s j , a
Revised: K do 5: for ( ω l , ( s j , a

Original: [Equation]
Revised: [Equation]

Original: , r
Revised: , r

Original: [Equation]
Revised: [Equation]

Original: , s
Revised: , s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: W × D do
Revised: W × D do

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: 7: end for 8: end for 9: # Minimize the loss starting from the previous parameters 10: ϕ i = ϕ i −11: for ∈ 1 . . .
Revised: 7: end for 8: end for 9: # Minimize the loss starting from the previous parameters 10: ϕ i = ϕ i −11: for ∈ 1 . . .

Original: H do 12: for j ∈ 1 . . .
Revised: H do 12: for j ∈ 1 . . .

Original: J do 13: ϕ i ← ϕ
Revised: J do 13: ϕ i ← ϕ

Original: i − λ
Revised: i − λ

Original: L (cid:88) l =
Revised: L (cid:88) l =

Original: J (cid:88) j
Revised: J (cid:88) j

Original: =
Revised: =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = ∇
Revised: = ∇

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: kl ( s
Revised: kl ( s

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( ω
Revised: ( ω

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:17)14: end for 15: end for 16: end for 17: return Parameters
Revised: (cid:17)14: end for 15: end for 16: end for 17: return Parameters

Original: ϕ
Revised: ϕ

Original: I of operator Λ
Revised: I of operator Λ

Original: A PPROXIMATE VALUE ITERATION FOR PROJECTED B ELLMAN OPERATORS
Revised: A PPROXIMATE VALUE ITERATION FOR PROJECTED B ELLMAN OPERATORS

Original: We devise an approximate value iteration algorithm to learn parameterized PBOs by solving the optimization problems (6) and (7).
Revised: We devise an approximate value iteration algorithm to learn parameterized PBOs by solving the optimization problems (6) and (7).

Original: The general structure of the algorithm closely resembles the one of Fitted Q -Iteration (Ernst et al., 2005), alternating one phase in which we compute the targets, and another one where we apply gradient descent to fit the parameters w.r.t.
Revised: The general structure of the algorithm closely resembles the one of Fitted Q -Iteration (Ernst et al., 2005), alternating one phase in which we compute the targets, and another one where we apply gradient descent to fit the parameters w.r.t.

Original: the targets (Algorithm 1).
Revised: the targets (Algorithm 1).

Original: For this reason, we call our algorithm Projected Q -Iteration (PQI).
Revised: For this reason, we call our algorithm Projected Q -Iteration (PQI).

Original: When possible, the fixed point of the parameterized PBO is added to the loss.
Revised: When possible, the fixed point of the parameterized PBO is added to the loss.

Original: An interesting property of our method is that it enables to build a rich loss by including multiple optimal Bellman iterations at the same time, as opposed to FQI that learns each Bellman iteration sequentially:
Revised: An interesting property of our method is that it enables to build a rich loss by including multiple optimal Bellman iterations at the same time, as opposed to FQI that learns each Bellman iteration sequentially:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Figureillustrates this advantage by comparing the way PQI and FQI behave in the space of all possible action-value functions.
Revised: Figureillustrates this advantage by comparing the way PQI and FQI behave in the space of all possible action-value functions.

Original: The ability of iterating for an arbitrary number of times enables us to derive theoretical advantages of PQI by leveraging established results in approximation error for AVI.
Revised: The ability of iterating for an arbitrary number of times enables us to derive theoretical advantages of PQI by leveraging established results in approximation error for AVI.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Theorem 3.4 of Farahmand (2011)) Let K ∈ N ∗ , ρ, ν two distribution probabilities over S×A .
Revised: (Theorem 3.4 of Farahmand (2011)) Let K ∈ N ∗ , ρ, ν two distribution probabilities over S×A .

Original: For any sequence ( Q k ) K k =

Revised: For any sequence ( Q k ) K k =

Original: ⊂ B ( S × A , R γ ) where R γ depends on the reward function and the discount factor, we have
Revised: ⊂ B ( S × A , R γ ) where R γ depends on the reward function and the discount factor, we have

Original: [Equation]
Revised: [Equation]

Original: where α k and C K,γ,R γ do not depend on the sequence ( Q k )
Revised: where α k and C K,γ,R γ do not depend on the sequence ( Q k )

Original: Kk =0 .
Revised: Kk =0 .

Original: F ( r ; K, ρ, γ ) relies on the concentrability coefficients of the greedy policies w.r.t.
Revised: F ( r ; K, ρ, γ ) relies on the concentrability coefficients of the greedy policies w.r.t.

Original: the value functions.
Revised: the value functions.

Original: This theorem shows that the distance between the value function and the optimal value function depends on the approximation errors for all the iterations.
Revised: This theorem shows that the distance between the value function and the optimal value function depends on the approximation errors for all the iterations.

Original: The loss of FQI contains only one term of the sum, while the loss of PQI contains the entire sum.
Revised: The loss of FQI contains only one term of the sum, while the loss of PQI contains the entire sum.

Original: In PQI, the terms are summed up with equal weights, which is not the case for the error propagation.
Revised: In PQI, the terms are summed up with equal weights, which is not the case for the error propagation.

Original: Future works could investigate how to weight those terms for increased performance.
Revised: Future works could investigate how to weight those terms for increased performance.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We empirically analyze our PQI algorithm with different classes of parameterized PBOs w.r.t.
Revised: We empirically analyze our PQI algorithm with different classes of parameterized PBOs w.r.t.

Original: FQI (Ernst et al., 2005) and, when possible, the additional baseline least-squares policy iteration (LSPI) (Lagoudakis & Parr, 2003).
Revised: FQI (Ernst et al., 2005) and, when possible, the additional baseline least-squares policy iteration (LSPI) (Lagoudakis & Parr, 2003).

Original: The fixed point is always added to the loss for the linear
Revised: The fixed point is always added to the loss for the linear

Original: PBO, unless stated otherwise.
Revised: PBO, unless stated otherwise.

Original: The pseudocodes of FQI and LSPI are provided in Appendix B.
Revised: The pseudocodes of FQI and LSPI are provided in Appendix B.

Original: Experimental details.
Revised: Experimental details.

Original: Each algorithm receives a fixed dataset of samples D and has access to the same number of optimal Bellman iterations K .
Revised: Each algorithm receives a fixed dataset of samples D and has access to the same number of optimal Bellman iterations K .

Original: We define the initial parameters for both FQI and
Revised: We define the initial parameters for both FQI and

Original: LSPI such that they result in a zero-function.
Revised: LSPI such that they result in a zero-function.

Original: We store the parameters after each iteration of the algorithms; then, we compute the action-value function for each iteration and the value function associated with the greedy policy computed from the action-value function at each iteration.
Revised: We store the parameters after each iteration of the algorithms; then, we compute the action-value function for each iteration and the value function associated with the greedy policy computed from the action-value function at each iteration.

Original: PQI can access the same functions by iterating over the initial parameters for more than K iterations, thanks to our parameterized PBO.
Revised: PQI can access the same functions by iterating over the initial parameters for more than K iterations, thanks to our parameterized PBO.

Original: When possible, we include the parameters corresponding to an infinite number of iterations, i.e., the fixed point of the parameterized PBO.
Revised: When possible, we include the parameters corresponding to an infinite number of iterations, i.e., the fixed point of the parameterized PBO.

Original: We learn parameterized PBOs both with linear function approximators ( linear PBO ) and neural networks ( neural PBO ).
Revised: We learn parameterized PBOs both with linear function approximators ( linear PBO ) and neural networks ( neural PBO ).

Original: We also report the performance of the fixed point of the linear PBO (linear PBO fixed point).
Revised: We also report the performance of the fixed point of the linear PBO (linear PBO fixed point).

Original: For problems when PBO is known in closed-form, we include it in the analysis as a reference ( PBO in the figures).
Revised: For problems when PBO is known in closed-form, we include it in the analysis as a reference ( PBO in the figures).

Original: Additional details on the experiments are in Appendix C.
Revised: Additional details on the experiments are in Appendix C.

Original: 5. C HAIN - WALK
Revised: 5. C HAIN - WALK

Original: [Figure]
Revised: [Figure]

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: We initially consider the chain-walk environment in Figure 4, with a chain of length N = 20 .
Revised: We initially consider the chain-walk environment in Figure 4, with a chain of length N = 20 .

Original: We parameterize value functions as tables to leverage our theoretical results on finite
Revised: We parameterize value functions as tables to leverage our theoretical results on finite

Original: MDPs (Section 4.1.1).
Revised: MDPs (Section 4.1.1).

Original: We plot the ℓ 2 -norm between the iterated value functions and the optimal ones in Figure 5a.
Revised: We plot the ℓ 2 -norm between the iterated value functions and the optimal ones in Figure 5a.

Original: For all methods, the Bellman operator is iterated for K = 5 times during training.
Revised: For all methods, the Bellman operator is iterated for K = 5 times during training.

Original: The vertical black dotted line denotes the split between the iterations that are learnt and the iteration that can be only reached by parameterized PBO.
Revised: The vertical black dotted line denotes the split between the iterations that are learnt and the iteration that can be only reached by parameterized PBO.

Original: Notably, we can iterate the learnt PBO for an arbitrary number of times.
Revised: Notably, we can iterate the learnt PBO for an arbitrary number of times.

Original: This property is crucial to nullify the distance between the value function computed with the parameterized PBOs and the optimal value function.
Revised: This property is crucial to nullify the distance between the value function computed with the parameterized PBOs and the optimal value function.

Original: The superior performance of PQI comes from the fact that both classes of operators are close to the class of the PBO.
Revised: The superior performance of PQI comes from the fact that both classes of operators are close to the class of the PBO.

Original: Since in this setting the PBO perfectly replicates the behavior of the optimal Bellman operator, we can foresee that any algorithm minimizing the approximation error by learning parameters of value functions would perform in the best case as good as the PBO, meaning that their performance would coincide with the one of PBO at iteration 5 .
Revised: Since in this setting the PBO perfectly replicates the behavior of the optimal Bellman operator, we can foresee that any algorithm minimizing the approximation error by learning parameters of value functions would perform in the best case as good as the PBO, meaning that their performance would coincide with the one of PBO at iteration 5 .

Original: For this reason, any algorithm designed as such would perform worse than or equal to FQI in this setting.
Revised: For this reason, any algorithm designed as such would perform worse than or equal to FQI in this setting.

Original: Note that LSPI is not aiming at reducing the approximation error, this is why the reasoning does not apply to this baseline.
Revised: Note that LSPI is not aiming at reducing the approximation error, this is why the reasoning does not apply to this baseline.

Original: Analysis of the size of the dataset of parameters.
Revised: Analysis of the size of the dataset of parameters.

Original: Ideally, the dataset of parameters should contain parameters close to the initial ones (the parameters with only zeros), others close to optimal
Revised: Ideally, the dataset of parameters should contain parameters close to the initial ones (the parameters with only zeros), others close to optimal

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: parameters, and others in between.
Revised: parameters, and others in between.

Original: Since we only assume to know the initial parameters, we take parameters close to them
Revised: Since we only assume to know the initial parameters, we take parameters close to them

Original: [Figure]
Revised: [Figure]

Original: in the sense that the truncated normal distribution used for data collection is centered.
Revised: in the sense that the truncated normal distribution used for data collection is centered.

Original: Figure 5b shows the distance to the optimal value function for different number of parameters.
Revised: Figure 5b shows the distance to the optimal value function for different number of parameters.

Original: Reasonably, the more parameters are given to the algorithm, the better the performance.
Revised: Reasonably, the more parameters are given to the algorithm, the better the performance.

Original: Analysis on the number of Bellman iterations K .
Revised: Analysis on the number of Bellman iterations K .

Original: We additionally analyze the behavior of PBOs for a different number of Bellman iterations than K = 5 .
Revised: We additionally analyze the behavior of PBOs for a different number of Bellman iterations than K = 5 .

Original: We consider K = 2 and K = 15 , observing that the PBOs still behave well as shown in Figure 6.
Revised: We consider K = 2 and K = 15 , observing that the PBOs still behave well as shown in Figure 6.

Original: 5. L INEAR QUADRATIC REGULATOR
Revised: 5. L INEAR QUADRATIC REGULATOR

Original: We consider the LQR setting in Section 4.1.3, leveraging our theoretical analysis by parameterizing value functions accordingly.
Revised: We consider the LQR setting in Section 4.1.3, leveraging our theoretical analysis by parameterizing value functions accordingly.

Original: Again, we consider a neural PBO and a linear PBO.
Revised: Again, we consider a neural PBO and a linear PBO.

Original: Figure 7a shows a metric proportional to the distance between the optimal value function and the value functions obtained at each iteration.
Revised: Figure 7a shows a metric proportional to the distance between the optimal value function and the value functions obtained at each iteration.

Original: For all methods, we set K = 2 .
Revised: For all methods, we set K = 2 .

Original: As in the previous experiment, the chosen class of operators is close to the class of the optimal PBO.
Revised: As in the previous experiment, the chosen class of operators is close to the class of the optimal PBO.

Original: We see that, at iteration 2 , the performance of FQI and the PQI are on par, while afterward, PQI keeps improving until the fixed point is reached.
Revised: We see that, at iteration 2 , the performance of FQI and the PQI are on par, while afterward, PQI keeps improving until the fixed point is reached.

Original: Again, we can state that any algorithm that learns parameters of value functions trying to minimize the approximation error, would perform worse than FQI.
Revised: Again, we can state that any algorithm that learns parameters of value functions trying to minimize the approximation error, would perform worse than FQI.

Original: FQI follows the curve of the PBO, meaning that the training cannot be improved.
Revised: FQI follows the curve of the PBO, meaning that the training cannot be improved.

Original: LSPI performs badly because the first policy for the evaluation step is a uniform policy, and the value function associated with the uniform policy is not a quadratic function.
Revised: LSPI performs badly because the first policy for the evaluation step is a uniform policy, and the value function associated with the uniform policy is not a quadratic function.

Original: Analysis on the number of Bellman iterations K .
Revised: Analysis on the number of Bellman iterations K .

Original: Since the convergence of all the algorithms is fast in term of iterations, we additionally show the performance with 4 Bellman iterations instead of 2 .
Revised: Since the convergence of all the algorithms is fast in term of iterations, we additionally show the performance with 4 Bellman iterations instead of 2 .

Original: Figures 7b confirms that PBOs perform well for a larger number of Bellman iterations.
Revised: Figures 7b confirms that PBOs perform well for a larger number of Bellman iterations.

Original: Analysis on the size of the dataset of samples.
Revised: Analysis on the size of the dataset of samples.

Original: We focus on neural PBO.
Revised: We focus on neural PBO.

Original: Figure 8a shows the distance to the optimal value function for trainings with a varying number of samples.
Revised: Figure 8a shows the distance to the optimal value function for trainings with a varying number of samples.

Original: When the grid becomes thinner, the distances are decreasing.
Revised: When the grid becomes thinner, the distances are decreasing.

Original: However this decrease is not strong.
Revised: However this decrease is not strong.

Original: This is why
Revised: This is why

Original: [Figure]
Revised: [Figure]

Original: V i |
Revised: V i |

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the size of the dataset is not a crucial parameter for learning PBO.
Revised: the size of the dataset is not a crucial parameter for learning PBO.

Original: Visualization.
Revised: Visualization.

Original: This setting allow us to plot the parameters of the action-value functions in a 3D space.
Revised: This setting allow us to plot the parameters of the action-value functions in a 3D space.

Original: The parameters corresponding to the iterations of PQI and FQI are shown in Figure 8b.
Revised: The parameters corresponding to the iterations of PQI and FQI are shown in Figure 8b.

Original: The training is done with K = 2 .
Revised: The training is done with K = 2 .

Original: The first parameters for the three algorithms are on (0 , 0 , 0) and then, the iterations goes towards the optimal parameters on the bottom right of the figure.
Revised: The first parameters for the three algorithms are on (0 , 0 , 0) and then, the iterations goes towards the optimal parameters on the bottom right of the figure.

Original: In this case, being able to iterate after the training clearly makes the difference, as it is clear that PQI is able to get closer to the optimal parameters after the second iteration
Revised: In this case, being able to iterate after the training clearly makes the difference, as it is clear that PQI is able to get closer to the optimal parameters after the second iteration

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: || V
Revised: || V

Original: * V i ||
Revised: * V i ||

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5. N ON - LINEAR PROJECTED B ELLMAN OPERATORS
Revised: 5. N ON - LINEAR PROJECTED B ELLMAN OPERATORS

Original: We consider a more advanced setting where the value function is approximated by a neural network, thus the space of value functions is not stable by the optimal Bellman operator.
Revised: We consider a more advanced setting where the value function is approximated by a neural network, thus the space of value functions is not stable by the optimal Bellman operator.

Original: We focus on the caron-hill MDP (Ernst et al., 2005), as implemented in MushroomRL (D’Eramo et al., 2021)
Revised: We focus on the caron-hill MDP (Ernst et al., 2005), as implemented in MushroomRL (D’Eramo et al., 2021)

Original: [Figure]
Revised: [Figure]

Original: and the bicycle MDP in Randlov & Alstrøm (1998).
Revised: and the bicycle MDP in Randlov & Alstrøm (1998).

Original: We parameterize value functions with one hidden layer of 30 neurons
Revised: We parameterize value functions with one hidden layer of 30 neurons

Original: [Figure]
Revised: [Figure]

Original: and we compare PQI only to FQI, since LSPI is not suitable for this setting.
Revised: and we compare PQI only to FQI, since LSPI is not suitable for this setting.

Original: We train a linear PBO and a neural PBO.
Revised: We train a linear PBO and a neural PBO.

Original: We set K = 9 for car-on-hill and K = 8 for bicycle.
Revised: We set K = 9 for car-on-hill and K = 8 for bicycle.

Original: Figure 9 shows the policies on car-on-hill for FQI and PQI trained with the neural PBO
Revised: Figure 9 shows the policies on car-on-hill for FQI and PQI trained with the neural PBO

Original: [Figure]
Revised: [Figure]

Original: where PQI is able to obtain a significantly more accurate policy than FQI.
Revised: where PQI is able to obtain a significantly more accurate policy than FQI.

Original: Figures 10 and 11a show that PBOs outperform FQI in terms of performance too.
Revised: Figures 10 and 11a show that PBOs outperform FQI in terms of performance too.

Original: Interestingly, iterating further produces a drop in performance for linear
Revised: Interestingly, iterating further produces a drop in performance for linear

Original: PBO, indicating that the class of linear operator is too simple.
Revised: PBO, indicating that the class of linear operator is too simple.

Original: Remarkably, neural PBO either shows no deterioration (Figure 10a), or a slight deterioration due to opposite performance in different seeds (Figure 11a and Appendix C.4).
Revised: Remarkably, neural PBO either shows no deterioration (Figure 10a), or a slight deterioration due to opposite performance in different seeds (Figure 11a and Appendix C.4).

Original: Further tuning should substantially reduce the number of failing seeds.
Revised: Further tuning should substantially reduce the number of failing seeds.

Original: This experiment evinces the limits of FQI where errors made at any iteration propagates until the last iteration (Theorem 1).
Revised: This experiment evinces the limits of FQI where errors made at any iteration propagates until the last iteration (Theorem 1).

Original: Conversely, PQI can change the path to the optimal parameters at each optimization step.
Revised: Conversely, PQI can change the path to the optimal parameters at each optimization step.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced the original idea of obtaining an approximation of the true Bellman operator, which we call projected Bellman operator (PBO).
Revised: We introduced the original idea of obtaining an approximation of the true Bellman operator, which we call projected Bellman operator (PBO).

Original: We defined PBO for different classes of problems, studied their theoretical properties of stability and convergence, and devised a novel approximate value iteration algorithm that provably reduces the error propagation over Bellman iterations.
Revised: We defined PBO for different classes of problems, studied their theoretical properties of stability and convergence, and devised a novel approximate value iteration algorithm that provably reduces the error propagation over Bellman iterations.

Original: Finally, we have empirically shown the benefit of PBO over related baselines on a variety of problems.
Revised: Finally, we have empirically shown the benefit of PBO over related baselines on a variety of problems.

Original: We consider this work a seminal study of PBO as a novel paradigm of RL, that can lead to multiple research directions, among which investigating its use in online settings is a first foreseeable avenue.
Revised: We consider this work a seminal study of PBO as a novel paradigm of RL, that can lead to multiple research directions, among which investigating its use in online settings is a first foreseeable avenue.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: We aim to guarantee the reproducibility of our work by providing the proofs of our theoretical results and additional details of the conducted experiments in the appendix.
Revised: We aim to guarantee the reproducibility of our work by providing the proofs of our theoretical results and additional details of the conducted experiments in the appendix.

Original: Further, we provide the code for reproducing our results in the supplementary material, and we will make it public upon acceptance
Revised: Further, we provide the code for reproducing our results in the supplementary material, and we will make it public upon acceptance

Original: A P ROOFS
Revised: A P ROOFS

Original: Proof of the closed-form of the PBO for a finite MDP.
Revised: Proof of the closed-form of the PBO for a finite MDP.

Original: We compute the optimal Bellman iteration over a table Q ∈ R N · M
Revised: We compute the optimal Bellman iteration over a table Q ∈ R N · M

Original: [Equation]
Revised: [Equation]

Original: where P ∈ R N · M × N is the transition probability matrix of the environment.
Revised: where P ∈ R N · M × N is the transition probability matrix of the environment.

Original: From these equations, the operator Q (cid:55)→ R + γP max a ′ ∈A Q ( · , a ′ ) , evaluated on the objective function from the definition of PBO 2, yields zero error.
Revised: From these equations, the operator Q (cid:55)→ R + γP max a ′ ∈A Q ( · , a ′ ) , evaluated on the objective function from the definition of PBO 2, yields zero error.

Original: This means that we have found the PBO in closed-form.
Revised: This means that we have found the PBO in closed-form.

Original: Proof of the closed-form of the PBO for a low-rank MDP.
Revised: Proof of the closed-form of the PBO for a low-rank MDP.

Original: The proof considers continuous unbounded state action spaces.
Revised: The proof considers continuous unbounded state action spaces.

Original: For Q ω ∈ Q Ω , ω the vector representing Q ω , max a ∈A Q ω ( s, a ) is well defined (here max might not be attained, it should be interpreted as a supremum).
Revised: For Q ω ∈ Q Ω , ω the vector representing Q ω , max a ∈A Q ω ( s, a ) is well defined (here max might not be attained, it should be interpreted as a supremum).

Original: We have | Q ω
Revised: We have | Q ω

Original: [Equation]
Revised: [Equation]

Original: s, a ) | = |⟨ σ
Revised: s, a ) | = |⟨ σ

Original: [Equation]
Revised: [Equation]

Original: s, a ) | ω ⟩| ≤ || σ ( s, a ) || · || ω || ≤ || ω
Revised: s, a ) | ω ⟩| ≤ || σ ( s, a ) || · || ω || ≤ || ω

Original: || , thus max a ∈A Q ω ( s, a ) < ∞ .
Revised: || , thus max a ∈A Q ω ( s, a ) < ∞ .

Original: Then, we write the optimal Bellman iteration on the function Q ω as
Revised: Then, we write the optimal Bellman iteration on the function Q ω as

Original: [Equation]
Revised: [Equation]

Original: from the definition of the transition probabilities.
Revised: from the definition of the transition probabilities.

Original: [Equation]
Revised: [Equation]

Original: This derivation shows that the operator ω (cid:55)→ θ + γ (cid:82) s ′ max a ′ ⟨ σ ( s ′ , a ′ ) | ω ⟩ µ ( s ′ )
Revised: This derivation shows that the operator ω (cid:55)→ θ + γ (cid:82) s ′ max a ′ ⟨ σ ( s ′ , a ′ ) | ω ⟩ µ ( s ′ )

Original: d s ′ minimizes the objective function presented in the definition of PBO 2 since its yields zero error.
Revised: d s ′ minimizes the objective function presented in the definition of PBO 2 since its yields zero error.

Original: This operator is the PBO for a low-rank MDP.
Revised: This operator is the PBO for a low-rank MDP.

Original: Intuition on the architecture of the parameterized PBO for a low-rank MDP.
Revised: Intuition on the architecture of the parameterized PBO for a low-rank MDP.

Original: In the finite case, the
Revised: In the finite case, the

Original: PBO is
Revised: PBO is

Original: [Equation]
Revised: [Equation]

Original: We can see that the most suitable architecture is a composition of a linear layer followed by a maxpooling and finally a last linear layer.
Revised: We can see that the most suitable architecture is a composition of a linear layer followed by a maxpooling and finally a last linear layer.

Original: Proof that the PBO is a γ -contraction-mapping for a low-rank MDP.
Revised: Proof that the PBO is a γ -contraction-mapping for a low-rank MDP.

Original: We now assume that the
Revised: We now assume that the

Original: MDP has a latent variable representation.
Revised: MDP has a latent variable representation.

Original: This proof was inspired by the proof of the contraction property of the Bellman Operator in Bertsekas (2015).
Revised: This proof was inspired by the proof of the contraction property of the Bellman Operator in Bertsekas (2015).

Original: Considering ω , ω ′ ∈
Revised: Considering ω , ω ′ ∈

Original: R d , we have
Revised: R d , we have

Original: [Equation]
Revised: [Equation]

Original: since µ
Revised: since µ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: positive
Revised: positive

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: since σ ( · , · ) are positive.
Revised: since σ ( · , · ) are positive.

Original: [Equation]
Revised: [Equation]

Original: since σ ( · , · ) are probability distributions.
Revised: since σ ( · , · ) are probability distributions.

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: since µ ( · ) i is a probability distribution for all i .
Revised: since µ ( · ) i is a probability distribution for all i .

Original: Proof of the closed-form of the PBO for LQR MDP.
Revised: Proof of the closed-form of the PBO for LQR MDP.

Original: Let ω
Revised: Let ω

Original: =
Revised: =

Original: ( G, I,
Revised: ( G, I,

Original: M ) ∈ R 2 × R −∗ 3 , the optimal
Revised: M ) ∈ R 2 × R −∗ 3 , the optimal

Original: Bellman iteration over Q ω is Γ ∗ Q ω ( s, a )
Revised: Bellman iteration over Q ω is Γ ∗ Q ω ( s, a )

Original: =
Revised: =

Original: r ( s, a ) + max a ′ Q ω ( s ′ , a ′ ) .
Revised: r ( s, a ) + max a ′ Q ω ( s ′ , a ′ ) .

Original: Before moving forward, we have to make sure this quantity is properly defined.
Revised: Before moving forward, we have to make sure this quantity is properly defined.

Original: The function a ′ (cid:55)→ Q ω ( s ′ , a ′ ) =
Revised: The function a ′ (cid:55)→ Q ω ( s ′ , a ′ ) =

Original: G · s ′ 

Revised: G · s ′

Original: + 2 I · s ′ a ′ + M · a ′ 2 is a polynomial of degree 2 in a ′ .
Revised: + 2 I · s ′ a ′ + M · a ′ 2 is a polynomial of degree 2 in a ′ .

Original: By choosing M < 0 , we ensure that this function has a unique maximum of equation − I / M · s ′ .
Revised: By choosing M < 0 , we ensure that this function has a unique maximum of equation − I / M · s ′ .

Original: This is why the PBO is only defined on R× R −∗ ∪ R × { 0 } × { 0 } .
Revised: This is why the PBO is only defined on R× R −∗ ∪ R × { 0 } × { 0 } .

Original: This makes
Revised: This makes

Original: [Equation]
Revised: [Equation]

Original: By putting it in the Bellman equation, we get
Revised: By putting it in the Bellman equation, we get

Original: [Equation]
Revised: [Equation]

Original: Then, the following operator is the PBO
Revised: Then, the following operator is the PBO

Original: [Equation]
Revised: [Equation]

Original: with Λ( ω ) =
Revised: with Λ( ω ) =

Original: G − I 2 / M .
Revised: G − I 2 / M .

Original: Remarks The PBO can also be understood in a geometrical way.
Revised: Remarks The PBO can also be understood in a geometrical way.

Original: It projects the parameters along a line of direction vector (cid:2)
Revised: It projects the parameters along a line of direction vector (cid:2)

Original: A
Revised: A

Original: AB
Revised: AB

Original: B 2 (cid:3) T with an offset [ Q S
Revised: B 2 (cid:3) T with an offset [ Q S

Original: R ] T .
Revised: R ] T .

Original: The iterations correspond to a non-linear transformation of the coefficient in front of the direction vector.
Revised: The iterations correspond to a non-linear transformation of the coefficient in front of the direction vector.

Original: This also means that the fixed-point, i.e. the optimal parameters are also on this line.
Revised: This also means that the fixed-point, i.e. the optimal parameters are also on this line.

Original: B D ETAILS OF THE BASELINES
Revised: B D ETAILS OF THE BASELINES

Original: Algorithm
Revised: Algorithm

Original: 1: Inputs:
Revised: 1: Inputs:

Original: Dataset of samples D
Revised: Dataset of samples D

Original: #Bellman iterations K #fitting steps H learning rate λ 2: Initialize: ω 0 # weight that give the null Q function 3: for i in 1 . . .
Revised: #Bellman iterations K #fitting steps H learning rate λ 2: Initialize: ω 0 # weight that give the null Q function 3: for i in 1 . . .

Original: K do 4: # Compute the target T 5: for ( s j , a j , r j , s ′ j ) ∈
Revised: K do 4: # Compute the target T 5: for ( s j , a j , r j , s ′ j ) ∈

Original: D do 6:
Revised: D do 6:

Original: T ( s j , a j ) =
Revised: T ( s j , a j ) =

Original: r j + γ max a Q ω i − 1 ( s ′ j , a ) 7: end for 8: 9: # Minimize the loss starting from the previous weights 10: ω i = ω
Revised: r j + γ max a Q ω i − 1 ( s ′ j , a ) 7: end for 8: 9: # Minimize the loss starting from the previous weights 10: ω i = ω

Original: i −11: for in 1 . . .
Revised: i −11: for in 1 . . .

Original: H do 12: for batch ∈ D do 13: ω
Revised: H do 12: for batch ∈ D do 13: ω

Original: i ← ω i − λ (cid:88)
Revised: i ← ω i − λ (cid:88)

Original: j ∈ batch ∇ ω ( T ( s j , a j ) − Q ω i ( s j , a j ))
Revised: j ∈ batch ∇ ω ( T ( s j , a j ) − Q ω i ( s j , a j ))

Original: 14: end for 15: end for 16: end for 17: return ω K
Revised: 14: end for 15: end for 16: end for 17: return ω K

Original: Algorithm 3 LSPI 1: Inputs:
Revised: Algorithm 3 LSPI 1: Inputs:

Original: Dataset of samples D
Revised: Dataset of samples D

Original: #Bellman iterations K features σ 2: Initialize:
Revised: #Bellman iterations K features σ 2: Initialize:

Original: Weights ω 0 that give the null Q function 3: for i in 1 . . .
Revised: Weights ω 0 that give the null Q function 3: for i in 1 . . .

Original: K do 4:
Revised: K do 4:

Original: # Critic step: policy improvement 5: for ( , , , s ′ j ) ∈ D do 6: π ( s ′ j ) =
Revised: # Critic step: policy improvement 5: for ( , , , s ′ j ) ∈ D do 6: π ( s ′ j ) =

Original: arg max a σ ( s ′ j , a ) T ω i −7: end for 8: 9: # Actor step: policy evaluation 10:
Revised: arg max a σ ( s ′ j , a ) T ω i −7: end for 8: 9: # Actor step: policy evaluation 10:

Original: A ←11: b ←12: for ( s j , a j , r j , s ′ j ) ∈ D do 13:
Revised: A ←11: b ←12: for ( s j , a j , r j , s ′ j ) ∈ D do 13:

Original: A ← A + σ ( s j , a j ) · (cid:0) σ ( s j , a j ) − γ σ ( s ′ j , π ( s ′ j )) (cid:1) T 14: b ← b + r j σ ( s j , a j ) 15: end for 16: ω i
Revised: A ← A + σ ( s j , a j ) · (cid:0) σ ( s j , a j ) − γ σ ( s ′ j , π ( s ′ j )) (cid:1) T 14: b ← b + r j σ ( s j , a j ) 15: end for 16: ω i

Original: 18: return
Revised: 18: return

Original: [Table]
Revised: [Table]

Original: C A DDITIONAL RESULTS AND DETAILS OF THE EMPIRICAL ANALYSIS
Revised: C A DDITIONAL RESULTS AND DETAILS OF THE EMPIRICAL ANALYSIS

Original: We provide details of the experimental setting.
Revised: We provide details of the experimental setting.

Original: Table 1 summarizes the values of all parameters appearing in the experiments.
Revised: Table 1 summarizes the values of all parameters appearing in the experiments.

Original: For the additional empirical analysis, only the parameter at stake is changed, all the others remain the ones shown in Table 1.
Revised: For the additional empirical analysis, only the parameter at stake is changed, all the others remain the ones shown in Table 1.

Original: FQI and PQI use an Adam optimizer (Kingma & Ba, 2015) with a linear annealing learning rate.
Revised: FQI and PQI use an Adam optimizer (Kingma & Ba, 2015) with a linear annealing learning rate.

Original: For FQI, the optimizer is reset at each iteration.
Revised: For FQI, the optimizer is reset at each iteration.

Original: C.
Revised: C.

Original: C HAIN -
Revised: C HAIN -

Original: WALK
Revised: WALK

Original: We consider all the possible state-action pairs 10 times as the initial dataset of samples D .
Revised: We consider all the possible state-action pairs 10 times as the initial dataset of samples D .

Original: Theserepetitions help the algorithms to grasp the randomness of the environment.
Revised: Theserepetitions help the algorithms to grasp the randomness of the environment.

Original: The set of parameters
Revised: The set of parameters

Original: W is generated by taking a truncated normal distribution for different seeds.
Revised: W is generated by taking a truncated normal distribution for different seeds.

Original: The initial parameters use for the figures is a table composed of zeros.
Revised: The initial parameters use for the figures is a table composed of zeros.

Original: The architecture of the neural PBO is a max-pooling layer followed by a linear layer.
Revised: The architecture of the neural PBO is a max-pooling layer followed by a linear layer.

Original: This choice has been made so that a perfectly tuned parameterized
Revised: This choice has been made so that a perfectly tuned parameterized

Original: PBO can be mathematically equal to the PBO.
Revised: PBO can be mathematically equal to the PBO.

Original: C. L INEAR QUADRATIC REGULATOR
Revised: C. L INEAR QUADRATIC REGULATOR

Original: The dynamics of the system is P ( s, a ) =
Revised: The dynamics of the system is P ( s, a ) =

Original: [Equation]
Revised: [Equation]

Original: 72 s − 0 .
Revised: 72 s − 0 .

Original: 53 a and the reward function is R ( s, a )
Revised: 53 a and the reward function is R ( s, a )

Original: [Equation]
Revised: [Equation]

Original: The set D is collected on a mesh of size 11 by 11 on the state-action space going from − 4 to 4 in both directions, which means that samples belong to [ − 4 , 4] ×
Revised: The set D is collected on a mesh of size 11 by 11 on the state-action space going from − 4 to 4 in both directions, which means that samples belong to [ − 4 , 4] ×

Original: [ − 4 , 4] ⊂
Revised: [ − 4 , 4] ⊂

Original: S × A .
Revised: S × A .

Original: [Figure]
Revised: [Figure]

Original: The set W is composed of only the initial parameters, i.e., (0 , 0 , 0) .
Revised: The set W is composed of only the initial parameters, i.e., (0 , 0 , 0) .

Original: This choice is made because during the optimization, the optimal Bellman operator might project the parameters in an area where the coefficients M are positive which would lead the next optimal Bellman iterations to diverge.
Revised: This choice is made because during the optimization, the optimal Bellman operator might project the parameters in an area where the coefficients M are positive which would lead the next optimal Bellman iterations to diverge.

Original: For
Revised: For

Original: PQI and FQI, we use the optimal Bellman operator, meaning that we should compute the maximum over a value function.
Revised: PQI and FQI, we use the optimal Bellman operator, meaning that we should compute the maximum over a value function.

Original: We choose to parameterize the value functions in a quadratic way, thus allowing us to compute the maximum in closedform.
Revised: We choose to parameterize the value functions in a quadratic way, thus allowing us to compute the maximum in closedform.

Original: However, we assume that the algorithms do not have this knowledge since it is not the case in more general settings.
Revised: However, we assume that the algorithms do not have this knowledge since it is not the case in more general settings.

Original: For this reason, we discretize the action space in a set ofactions going from − 8 to 8 .
Revised: For this reason, we discretize the action space in a set ofactions going from − 8 to 8 .

Original: The architecture of the neural
Revised: The architecture of the neural

Original: PBO starts with a function ω (cid:55)→ ω 1 − ω 22 ω 3 and is followed by a linear layer.
Revised: PBO starts with a function ω (cid:55)→ ω 1 − ω 22 ω 3 and is followed by a linear layer.

Original: Once again, this choice has been made to ensure that the neural PBO can reproduce the behavior of the PBO if it is well trained.
Revised: Once again, this choice has been made to ensure that the neural PBO can reproduce the behavior of the PBO if it is well trained.

Original: In the following, we explain how to obtain the metric used in Figure 7a.
Revised: In the following, we explain how to obtain the metric used in Figure 7a.

Original: Then, we demonstrate that lowering the standard deviation of the initial parameters of PBO is important.
Revised: Then, we demonstrate that lowering the standard deviation of the initial parameters of PBO is important.

Original: Metric.
Revised: Metric.

Original: In this setting, the value functions associated to a greedy policy can be computed in closed form.
Revised: In this setting, the value functions associated to a greedy policy can be computed in closed form.

Original: For a weight ω
Revised: For a weight ω

Original: = ( G, I,
Revised: = ( G, I,

Original: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } such that A − BI / M ̸ = 1 , the value function associated to the greedy policy π ω computed from the value function Q ω has the following form, for any s ∈ S :
Revised: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } such that A − BI / M ̸ = 1 , the value function associated to the greedy policy π ω computed from the value function Q ω has the following form, for any s ∈ S :

Original: [Equation]
Revised: [Equation]

Original: where P ω =
Revised: where P ω =

Original: Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 2 .
Revised: Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 2 .

Original: Note that G does not influence the result, which is normal since the greedy policy is only influenced by the term linked with the actions.
Revised: Note that G does not influence the result, which is normal since the greedy policy is only influenced by the term linked with the actions.

Original: Proof.
Revised: Proof.

Original: Let ω =
Revised: Let ω =

Original: ( G, I,
Revised: ( G, I,

Original: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } such that A − BI / M ̸ = 1 , the greedy policy of the value function parameterized by ω is π ω ( s )
Revised: M ) ∈ R 2 × R −∗ ∪ R × { 0 } × { 0 } such that A − BI / M ̸ = 1 , the greedy policy of the value function parameterized by ω is π ω ( s )

Original: =
Revised: =

Original: − I / M · s where we take the convention/ 0 = 0 .
Revised: − I / M · s where we take the convention/ 0 = 0 .

Original: To show the result, we show that the function s (cid:55)→ P ω · s 2 satisfies the optimal Bellman equation 4 .
Revised: To show the result, we show that the function s (cid:55)→ P ω · s 2 satisfies the optimal Bellman equation 4 .

Original: We set P ω
Revised: We set P ω

Original: = Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 2 , and we have ∀
Revised: = Q − 2 SIM + RI 2 M 2 1 − ( A − BIM ) 2 , and we have ∀

Original: [Equation]
Revised: [Equation]

Original: , P ω · s 2 =
Revised: , P ω · s 2 =

Original: Γ ∗ ( x (cid:55)→ P ω · x 2 )( s ) optimal Bellman equation for the function x (cid:55)→ P ω · x 2 .
Revised: Γ ∗ ( x (cid:55)→ P ω · x 2 )( s ) optimal Bellman equation for the function x (cid:55)→ P ω · x 2 .

Original: ⇐⇒ ∀ s ∈ S , P ω · s 2 = r
Revised: ⇐⇒ ∀ s ∈ S , P ω · s 2 = r

Original: ( s, π ω ( s ))
Revised: ( s, π ω ( s ))

Original: + P ω · s ′⇐⇒
Revised: + P ω · s ′⇐⇒

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒
Revised: ⇐⇒

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒
Revised: ⇐⇒

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒ P
Revised: ⇐⇒ P

Original: [Equation]
Revised: [Equation]

Original: RI
Revised: RI

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that if we multiply s to both side of the assumption A − BI / M ̸ = 1 , we get s ′ ̸
Revised: Note that if we multiply s to both side of the assumption A − BI / M ̸ = 1 , we get s ′ ̸

Original: = s where s ′ is the next state if the greedy policy is followed.
Revised: = s where s ′ is the next state if the greedy policy is followed.

Original: This means that we cannot compute the value function of greedy policies that leads to steady states.
Revised: This means that we cannot compute the value function of greedy policies that leads to steady states.

Original: This is normal since this value function is not defined for all states because the sum of accumulated rewards diverges except in 0 where it is 0 .
Revised: This is normal since this value function is not defined for all states because the sum of accumulated rewards diverges except in 0 where it is 0 .

Original: [Figure]
Revised: [Figure]

Original: We leverage this result to state that the distance between the P ω of a set of parameters ω and the solution of the Riccati equation is proportional to the distance between the value function of the greedy policy and the optimal value function.
Revised: We leverage this result to state that the distance between the P ω of a set of parameters ω and the solution of the Riccati equation is proportional to the distance between the value function of the greedy policy and the optimal value function.

Original: Proof.
Revised: Proof.

Original: Let b ∈ R .
Revised: Let b ∈ R .

Original: We consider the norm- 2 over the functions going from [ − b, b ] to R .
Revised: We consider the norm- 2 over the functions going from [ − b, b ] to R .

Original: For ω ∈ R× R −∗ ∪ R × { 0 } × { 0 } , we have
Revised: For ω ∈ R× R −∗ ∪ R × { 0 } × { 0 } , we have

Original: [Equation]
Revised: [Equation]

Original: The norm of the difference of the value functions is proportional to the difference of P coefficients.
Revised: The norm of the difference of the value functions is proportional to the difference of P coefficients.

Original: Analysis on the standard deviation of parameterized PBO’s initial parameters.
Revised: Analysis on the standard deviation of parameterized PBO’s initial parameters.

Original: Figure 13 shows the distance to the optimal value function for linear PBOs that have been initialized with different standard deviations.
Revised: Figure 13 shows the distance to the optimal value function for linear PBOs that have been initialized with different standard deviations.

Original: The linear PBO initialized with smaller standard deviation shows better performance.
Revised: The linear PBO initialized with smaller standard deviation shows better performance.

Original: This behavior can be explained by the following two reasons.
Revised: This behavior can be explained by the following two reasons.

Original: First, a linear PBO is very limited in terms of freedom, meaning that if it is initialized in such a way that it goes to the wrong direction, it is very likely that it will stay in the wrong direction during the training; second, by initializing the parameters of the parameterized PBO with low value, the operator is already contractive at the beginning of the training.
Revised: First, a linear PBO is very limited in terms of freedom, meaning that if it is initialized in such a way that it goes to the wrong direction, it is very likely that it will stay in the wrong direction during the training; second, by initializing the parameters of the parameterized PBO with low value, the operator is already contractive at the beginning of the training.

Original: This helps the parameterized PBO to have a stable behavior when evaluating on iterations that were not seen during training.
Revised: This helps the parameterized PBO to have a stable behavior when evaluating on iterations that were not seen during training.

Original: C. C AR - ON - HILL
Revised: C. C AR - ON - HILL

Original: The goal is to bring a car up a hill.
Revised: The goal is to bring a car up a hill.

Original: The agent chooses between 2 actions: left or right .
Revised: The agent chooses between 2 actions: left or right .

Original: The state space is 2 -dimensional: position in [ − 1 , 1] and velocity [ − 3 , 3] .
Revised: The state space is 2 -dimensional: position in [ − 1 , 1] and velocity [ − 3 , 3] .

Original: If the agent succeeds to bring the car up the hill – at position greater than 1 and velocity in between − 3 and 3 – then the reward is 1 , if the agent exceeds the state space, the reward is − 1 ; otherwise, the reward is 0 .
Revised: If the agent succeeds to bring the car up the hill – at position greater than 1 and velocity in between − 3 and 3 – then the reward is 1 , if the agent exceeds the state space, the reward is − 1 ; otherwise, the reward is 0 .

Original: As our PQI is an offline algorithm, we need to make sure that all the necessary exploration has been done in the dataset of samples.
Revised: As our PQI is an offline algorithm, we need to make sure that all the necessary exploration has been done in the dataset of samples.

Original: For that reason, we first consider a uniform sampling policy to collect episodes starting from the lowest point in the map ( [ − 0 .
Revised: For that reason, we first consider a uniform sampling policy to collect episodes starting from the lowest point in the map ( [ − 0 .

Original: 5 , 0] ) with an horizon of 100 .
Revised: 5 , 0] ) with an horizon of 100 .

Original: This sampling process is stopped when 4 .
Revised: This sampling process is stopped when 4 .

Original: 500 samples are gathered.
Revised: 500 samples are gathered.

Original: To get more samples with positive reward, we sample new episodes starting from a state located between [0 . 1 , 1 . 3] and [0 . 5 , 0 .
Revised: To get more samples with positive reward, we sample new episodes starting from a state located between [0 . 1 , 1 . 3] and [0 . 5 , 0 .

Original: 38] with a uniform policy as well.
Revised: 38] with a uniform policy as well.

Original: In total, 5 . 000 samples are collected.
Revised: In total, 5 . 000 samples are collected.

Original: The sample and reward distributions over the state space is shown in Figure 14.
Revised: The sample and reward distributions over the state space is shown in Figure 14.

Original: Inspired by the most suitable architecture for a parameterized
Revised: Inspired by the most suitable architecture for a parameterized

Original: [Figure]
Revised: [Figure]

Original: || Q *
Revised: || Q *

Original: Q i ||
Revised: Q i ||

Original: [Figure]
Revised: [Figure]

Original: 25 #Iterations    || V
Revised: 25 #Iterations    || V

Original: * V i ||
Revised: * V i ||

Original: [Figure]
Revised: [Figure]

Original: PBO, see Appendix A, we set the architecture of the neural PBO with a first block composed of a linear layer followed by a rectified linear unit (ReLU) and another linear layer.
Revised: PBO, see Appendix A, we set the architecture of the neural PBO with a first block composed of a linear layer followed by a rectified linear unit (ReLU) and another linear layer.

Original: Then, the first block is followed by a max-pooling before adding a second block that is identical to the first one.
Revised: Then, the first block is followed by a max-pooling before adding a second block that is identical to the first one.

Original: The set of parameters W is composed of only the initial parameters, because the complexity of the classes of PBOs used in the experiment is not high.
Revised: The set of parameters W is composed of only the initial parameters, because the complexity of the classes of PBOs used in the experiment is not high.

Original: The initial parameters are random except for the last layer that is set to zero, ensuring that the associated value function only outputs zeros.
Revised: The initial parameters are random except for the last layer that is set to zero, ensuring that the associated value function only outputs zeros.

Original: To help the training, the value functions are taking actions in {− 1 , 1 } instead of the usual { 0 , 1 } .
Revised: To help the training, the value functions are taking actions in {− 1 , 1 } instead of the usual { 0 , 1 } .

Original: We also use a low standard deviation for the initial parameters of the PBOs as described in Appendix C.2.
Revised: We also use a low standard deviation for the initial parameters of the PBOs as described in Appendix C.2.

Original: Metric.
Revised: Metric.

Original: Ernst et al.
Revised: Ernst et al.

Original: (2005) measure the performance of the agents by performing roll-outs starting from different states on a grid of size 17 × 17 .
Revised: (2005) measure the performance of the agents by performing roll-outs starting from different states on a grid of size 17 × 17 .

Original: Since the policies, the reward, and the dynamics are deterministic, the discounted reward obtained at the end of the roll-outs corresponds to the value function associated with the policy used during the roll-outs.
Revised: Since the policies, the reward, and the dynamics are deterministic, the discounted reward obtained at the end of the roll-outs corresponds to the value function associated with the policy used during the roll-outs.

Original: To compute the optimal value function, they run a brute force algorithm that tries every possible sequence of actions until it finds the shortest way to go up hill.
Revised: To compute the optimal value function, they run a brute force algorithm that tries every possible sequence of actions until it finds the shortest way to go up hill.

Original: In our case, the agents do not have all the transitions in D corresponding to every starting states of the grid they are using, as shown in Figure 14a.
Revised: In our case, the agents do not have all the transitions in D corresponding to every starting states of the grid they are using, as shown in Figure 14a.

Original: This is why we apply a mask on the grid to weight the starting states by the number of time a state has been visited in the dataset.
Revised: This is why we apply a mask on the grid to weight the starting states by the number of time a state has been visited in the dataset.

Original: Analysis on the number of Bellman iterations K .
Revised: Analysis on the number of Bellman iterations K .

Original: Figure 15 shows that PQI outperforms FQI at iteration 15 , for K = 15 .
Revised: Figure 15 shows that PQI outperforms FQI at iteration 15 , for K = 15 .

Original: Similarly to the case K = 9 , the linear PBO is too simple to behave well after the training iterations.
Revised: Similarly to the case K = 9 , the linear PBO is too simple to behave well after the training iterations.

Original: Once again, the way the loss is shaped allows PQI to outperform FQI.
Revised: Once again, the way the loss is shaped allows PQI to outperform FQI.

Original: C. B ICYCLE
Revised: C. B ICYCLE

Original: We consider the bicycle problem, as described in Randlov & Alstrøm (1998).
Revised: We consider the bicycle problem, as described in Randlov & Alstrøm (1998).

Original: The state space is composed of 4 dimensions: ( ω, ˙ ω, θ, ˙ θ ) where ω is the angle between the floor and the bike, and θ is the angle between the handle bar and the perpendicular axis to the bike.
Revised: The state space is composed of 4 dimensions: ( ω, ˙ ω, θ, ˙ θ ) where ω is the angle between the floor and the bike, and θ is the angle between the handle bar and the perpendicular axis to the bike.

Original: The goal is to ride a bicycle for 500 seconds ( 50 . 000 steps).
Revised: The goal is to ride a bicycle for 500 seconds ( 50 . 000 steps).

Original: The agent can apply a torque T ∈ {− 2 , 0 , 2 } to the handle bar to make it rotate.
Revised: The agent can apply a torque T ∈ {− 2 , 0 , 2 } to the handle bar to make it rotate.

Original: The agent can also move its center of gravity in the direction d ∈ {− 0 .
Revised: The agent can also move its center of gravity in the direction d ∈ {− 0 .

Original: 02 } perpendicular to the bike.
Revised: 02 } perpendicular to the bike.

Original: As in Lagoudakis & Parr (2003), the agent chooses between applying a torque or moving its center of gravity, resulting in 5 actions instead of 9 .
Revised: As in Lagoudakis & Parr (2003), the agent chooses between applying a torque or moving its center of gravity, resulting in 5 actions instead of 9 .

Original: Usually, a uniform noise in the interval [ − 0 .
Revised: Usually, a uniform noise in the interval [ − 0 .

Original: 02] is added to d .
Revised: 02] is added to d .

Original: For the purpose of this work, we reduce the number of samples by making the magnitude of the noise 10 times smaller.
Revised: For the purpose of this work, we reduce the number of samples by making the magnitude of the noise 10 times smaller.

Original: A reward of − 1 is given when the bike falls down, i.e., | ω | > 12 ◦ .
Revised: A reward of − 1 is given when the bike falls down, i.e., | ω | > 12 ◦ .

Original: We also use reward shaping to have more informative samples, and we add a reward proportional to the change in ω , i.e., 10 4 ( | ω t | − | ω t +1 | ) , as in Lagoudakis &
Revised: We also use reward shaping to have more informative samples, and we add a reward proportional to the change in ω , i.e., 10 4 ( | ω t | − | ω t +1 | ) , as in Lagoudakis &

Original: Parr (2003).
Revised: Parr (2003).

Original: The dataset of samples is composed of 3 .
Revised: The dataset of samples is composed of 3 .

Original: 500 episodes starting from a position close to (0 , 0 , 0 , 0) and cut after 20 steps (Lagoudakis & Parr, 2003).
Revised: 500 episodes starting from a position close to (0 , 0 , 0 , 0) and cut after 20 steps (Lagoudakis & Parr, 2003).

Original: The architecture of the neural PBO is the same as the one used in the Car-On-Hill experiment.
Revised: The architecture of the neural PBO is the same as the one used in the Car-On-Hill experiment.

Original: To evaluate the value functions, we simulate 10 trajectories.
Revised: To evaluate the value functions, we simulate 10 trajectories.

Original: We observed only two kinds of outcomes in our experiments: either the policies deteriorate the performance quickly, or they keep it stable, as shown in Figure 11b.
Revised: We observed only two kinds of outcomes in our experiments: either the policies deteriorate the performance quickly, or they keep it stable, as shown in Figure 11b.

Original: This also explains the big confidence interval for deep PBO in Figure 11a.
Revised: This also explains the big confidence interval for deep PBO in Figure 11a.