Proximal Point Imitation Learning
Abstract
This work develops new algorithms with rigorous efficiency guarantees for infinite horizon imitation learning (IL) with linear function approximation without restrictive coherence assumptions.
We begin with the minimax formulation of the problem and then outline how to leverage classical tools from optimization, in particular, the proximal-point method (PPM) and dual smoothing, for online and offline IL, respectively.
Thanks to PPM, we avoid nested policy evaluation and cost updates for online IL appearing in the prior literature.
In particular, we do away with the conventional alternating updates by the optimization of a single convex and smooth objective over both cost and Q -functions.
When solved inexactly, we relate the optimization errors to the suboptimality of the recovered policy.
As an added bonus, by re-interpreting PPM as dual smoothing with the expert policy as a center point, we also obtain an offline IL algorithm enjoying theoretical guarantees in terms of required expert trajectories.
Finally, we achieve convincing empirical performance for both linear and neural network function approximation.
Introduction
This work is concerned with the prototypical setting of imitation learning (IL) where
An expert provides demonstrations of state-action pairs in an environment.
The expert could be optimal or suboptimal with respect to an unknown cost/reward function.
The learner chooses distance measure between its policy to be learned and the expert empirical distribution estimated from demonstrations.
The learner employs an algorithm, which additionally may or may not use interactions with the environment, to minimize the chosen distance.
In IL, the central goal of the learner is to recover a policy competitive with expert with respect to the underlying unknown cost function.
IL is important for several real world applications like driving [62], robotics [88], and economics/finance [27] at the expense of following resources: (R1) expert demonstrations, (R2)
(optional) interactions with the environment where the expert collected the demonstrations, and (R3) computational resources for solving the problem template.
Interestingly, while there is a vast amount of literature using optimization ideas on the IL problem template, i.e. Lagrangian duality [51, 38, 59, 63, 64], resource guarantees are still widely missing since the optimization literature focuses on the resource (R3) where IL literature mainly focuses on the first two resources (R1) and (R2).
Our work leverages deeper connections between optimization tools and IL by showing how classical optimization tools can be applied in a linear programming formulation of IL problem guaranteeing efficiency in all (R1), (R2), (R3).
Our contributions: This work aims at designing an algorithm enjoying both theoretical guarantees and convincing empirical performance.
Our methodology is rooted in classical optimization tools and the LP approach to MDPs.
More precisely, the method uses the recently repopularized overparameterization technique to obtain the Q-function as a Lagrangian multiplier [77, 14] and solves the associated program using a PPM update with appropriately chosen Bregman divergences.
This results to an actor-critic algorithm, with the key feature that the policy evaluation step involves optimization of a single concave and smooth objective over both cost and Q -functions.
In this way, we avoid instability or poor convergence due to adversarial training [51, 122, 70, 105], and can also recover an explicit cost along with Q-function.
We further account for potential optimization errors, presenting an error propagation analysis that leads to rigorous guarantees for both online and offline setting.
For the context of linear MDPs [14, 121, 55, 22, 116, 7, 84], we provide explicit convergence rates and error bounds for the suboptimality of the learned policy, under mild assumptions, significantly weaker than those found in the literature until now.
To our knowledge, such guarantees in this setting are provided for the first time.
Finally, we demonstrate that our approach achieves convincing empirical performance for both linear and neural network function approximation.
Related Literature.
The first algorithm addressing the imitation learning problem is behavioral cloning [93].
Due to the covariate shift problem [98, 99], it has low efficiency in terms of expert trajectories (R1).
To address this issue, [100, 87, 4, 95, 111, 85, 123, 5, 68, 69] proposed to cast the problem as inverse reinforcement learning (IRL).
IRL improves the efficiency in terms of expert trajectories, at the cost of introducing the need of running reinforcement learning (RL) repetitively, which can be prohibitive in terms of environment samples (R2) and computation (R3).
A successive line of work started with [112] highlights that repeated calls to an RL routine can be avoided.
This work inspired generative adversarial imitation learning (GAIL) [51] and other follow-up works [38, 59, 63, 64] that leveraged optimization tools like primal-dual algorithms but did not try to deepen the optimization connections to derive efficiency guarantees in terms of all (R1),(R2),(R3).
Finally, a recent line of work [40, 57] in IL bypasses the need of optimizing over cost functions and thus avoids instability due to adversarial training.
Although these algorithms achieve impressive empirical performance in challenging high dimensional benchmark tasks, they are hampered by limited theoretical understanding.
This is the fundamental difference from our work, which enjoys both favorable practical performance and strong theoretical guarantees.
Existing model-free IL theoretical papers with global convergence guarantees assume either a finite horizon episodic
MDP setting [70], or tabular MDPs [105], or the infinite horizon case but with restrictive assumptions, such as linear quadratic regulator setting [21], continuous kernelized nonlinear regulator [26, 56], generative model, coherence assumption [58, 14], or a linear transition law that can be completely specified by a finite-dimensional matrix [122, 70].
On the other hand, we provide convergence guarantees and error bounds for the context of linear MDPs [14, 121, 55, 22, 116, 7, 84].
Despite being linear, the transition law can still have infinite degrees of freedom.
To our knowledge, such guarantees in this setting are provided for the first time.
Our work applies the technique known as regularization in the online learning literature [6, 103] and Bregman proximal-point or smoothing in optimization literature [97, 82] to the LP formulation for MDPs [73, 35, 36, 17, 48, 49, 33, 34, 102, 91, 92, 1, 65, 30, 79, 115, 67, 13, 31, 55, 106].
From this perspective, we can see Deep Inverse Q-Learning [57] and IQ-Learn [40] that consider entropy regularization in the objective as smoothing using uniform distribution as center point.
In our case, we instead use as center point the previous iteration of the algorithm (for the online case) or the expert (for the offline case).
From the technical point of view, the most important related works are the analysis of REPS/QREPS [90, 14, 89] and O-REPS [124] that first pointed out the connection between REPS and
PPM.
We build on their techniques with some important differences.
In particular, while in the
LP formulation of RL, PPM and mirror descent [15, 47] are equivalent, recognizing that they are not equivalent in IL is critical for stronger empirical performance.
As an independent interest, our techniques can be used to improve upon the best rate for REPS in the tabular setting [89] and to extend the guarantees to linear MDPs.
In order to discuss in more detail our research questions and situate them among prior related theoretical and practical works, we provide in Appendix A an extended literature review.
Background 2.1 Markov Decision Processes
The RL environment and its underlying dynamics are typically abstracted as an MDP given by a tuple ( S , A , P, ν 0 , c , γ ) , where S is the state space, A is the action space, P : S×A → ∆ S is the transition law, ν 0 ∈ ∆ S is the initial state distribution, c ∈ [0 , 1] |S||A| is the cost, and γ ∈ (0 , 1) is the discount factor.
For simplicity, we focus on problems where S and A are finite but too large to be enumerated.
A stationary Markov policy π : S → ∆ A interacts with the environment iteratively, starting with an initial state s 0 ∼ ν 0 .
At round t , if the system is at state s t , an action a t ∼ π ( ·| s t ) is sampled and applied to the environment.
Then a cost c ( s, a ) is incurred, and the system transitions to the next state s t
+1 ∼ P ( ·| s, a ) .
The goal of RL is to solve the optimal control problem ρ ⋆ c ≜ min π ρ c ( π ) , where ρ c ( π ) ≜ (1 − γ ) ⟨ ν 0 , V π c ⟩ is the normalized total discounted expected cost of π .
The state value function V π c ∈
R |S| of π , given cost c , is defined by V π c ( s ) ≜
E π s (cid:104) (cid:80) ∞ t =
γ t c ( s t , a t ) (cid:105) , where E π s denotes the expectation with respect to the trajectories generated by π starting from s 0 = s .
The optimal value function V ⋆ c ∈ R |S| is defined by
V ⋆ c ( s ) ≜ min π V π c ( s ) .
The optimal state-action value function Q ⋆ c ∈ R |S||A| , given by Q ⋆ c ( s, a ) ≜ c ( s, a ) + γ (cid:80) s ′
V ⋆ c ( s ′ )
P ( s ′ | s, a ) , is known to characterize optimal behaviors.
Indeed V ⋆ c is the unique solution to the Bellman optimality equation V ⋆ c ( s ) =
min a Q ⋆ c ( s, a ) .
In addition, any deterministic policy π ⋆ c ( s ) = arg min a Q ⋆ c ( s, a ) is known to be optimal.
For every policy π , we define the normalized state-action occupancy measure µ π ∈ ∆ S×A , by µ π ( s, a ) ≜ (1 − γ ) (cid:80) ∞ t =
γ t P π ν
[ s t
=
s, a t = a ] , where P π ν 0 [ · ] denotes the probability of an event when following π starting from s 0 ∼ ν 0 .
The occupancy measure can be interpreted as the discounted visitation frequency of state-action pairs.
This allows us to write ρ c ( π ) = ⟨ µ π , c ⟩ .
2.2 Imitation Learning
Similarly to RL, the IL problem is posed in the MDP formalism, with the critical difference that the true cost c true is unknown.
Instead, we have access to a finite set of truncated trajectories sampled i.i.d.
by executing an expert policy π E in the environment.
The goal is to learn a policy that performs better than π E with respect to the unknown c true .
To this end, we adopt the apprenticeship learning formalism [4, 112, 50, 51, 105], which carries the assumption that c true belongs to a class of cost functions C .
We then seek an apprentice policy π A that outperforms the expert across C by solving the following optimization problem
[Equation]
(1) where d C ( π, π E ) ≜ max c ∈C (cid:0) ρ c ( π ) − ρ c ( π E ) (cid:1) defines the C -distance between π and π E [51, 28, 122, 70].
Then, π A satisfies the goal of IL, since it holds that ρ c true ( π A ) − ρ c true ( π E ) ≤ ζ ⋆ ≤ 0 .
Intuitively, the cost class C distinguishes the expert from other policies.
The maximization in (1) assigns high total cost to non-expert policies and low total cost to π E [51], while the minimization aims to find the policy that matches the expert as close as possible with respect to d C .
By writing d C in its dual form ¯ d C ( µ π , µ π E ) ≜ max c ∈C (cid:0) ⟨ µ π , c ⟩ − (cid:10) µ π E , c (cid:11) (cid:1) , it can be interpreted as an integral probability metric [80, 60] between the occupancy measures µ π and µ π E .
Depending on how C is chosen, d C turns to a different metric of probability measures like the 1 -Wasserstein distance [117, 32] for C
= Lip 1 ( S ×A ) , the total variation for C = { c | ∥ c ∥ ∞ ≤ 1 } , or the maximum mean discrepancy for C
= { c | ∥ c
∥ H ≤ 1 } , where Lip 1 ( S × A ) denotes the space of 1 -Lipschitz functions on S × A , and ∥·∥ H denotes the norm of a reproducing kernel Hilbert space H [104].
In our theoretical analysis, we focus on linearly parameterized cost classes [111, 112, 51, 70, 105] of the form C ≜ { c w ≜ (cid:80) mi
=1 w i ϕ i | w ∈ W} , where { ϕ i } mi =1 ⊂ R |S||A| + are fixed feature vectors, such that ∥ ϕ i ∥ 1 ≤ 1 for all i ∈ [ m ] , and W is a a convex constraint set for the cost weights w .
This assumption is not necessarily restrictive as usually in practice the true cost depends on just a few key properties, but the desirable weighting that specifies how different desiderata should be traded-off is unknown [4].
Moreover, the cost features can be complex nonlinear functions that can be obtained via unsupervised learning from raw state observations [20, 29].
The matrix Φ ≜ [ ϕ. . .
ϕ m ] gives rise a feature expectation vector (FEV) ρ Φ ( π ) ≜ ( ρ ϕ 1 ( π E ) , . . .
, ρ ϕ m ( π E ))
T ∈ R m of a policy π .
Then, by choosing W to be the ℓ 2 unit ball B m 1 ≜ { w ∈ R m | ∥ w ∥ 2 ≤ 1 } [4], we get a feature expectation matching objective d C ( π, π π E ) =
∥ ρ Φ ( π ) −
ρ Φ ( π E ) ∥ 2 , while for W being the probability simplex ∆ [ m ] [111, 112] we have a worst-case excess cost objective d C ( π, π π E ) =
max i ∈ [ m ] (cid:0)
ρ ϕ i ( π ) − ρ ϕ
i ( π E ) (cid:1) .
For clarity, we will replace c by w in the notation of the quantities defined in Section 2.1.
A Q -Convex-Analytic Viewpoint
Our methodology builds upon the convex-analytic approach to AL, first introduced by [112], with the key difference that we consider a different convex formulation that introduces Q -functions as slack variables.
This allows to design a practical scalable model-free algorithm with theoretical guarantees.
Let F ≜ { µ ∈ R |S||A| | ( B − γ P ) ⊺ µ = (1 − γ ) ν 0 , µ ≥ 0 } be the state-action polytope , where P is the vector form of P , i.e., P ( s,a ) ,s ′ ≜ P ( s ′ | s, a ) , and B is a binary matrix defined by B ( s,a ) ,
s ′ ≜if s = s ′ , and B ( s,a ) ,s ′ ≜ 0 otherwise.
The linear constraints that define the set F , also known as
Bellman flow constraints , precisely characterize the set of state-action occupancy measures.
Proposition 1 (94) .
We have that µ ∈ F if and only if there exists a unique stationary Markov policy π such that µ = µ π .
If µ ∈ F then the policy π µ ( a | x ) ≜ µ ( x,a ) (cid:80) a ′∈A µ ( x,a ′ ) has occupancy measure µ .
Using Proposition 1 and the dual form of the C -distance ¯ d C ( µ , µ π E ) = max w ∈W (cid:10) µ − µ π E , c w (cid:11) , it follows that (1) is equivalent to the primal convex program ζ
⋆ = min µ { ¯ d C ( µ , µ π E ) | µ ∈ F } .
In particular for W = ∆ [ m ] and by using an epigraphic transformation, we end up with an LP program [112], while for W =
B m 1 we get a quadratic objective with linear constraints [4].
A slight variation of the above reasoning is to introduce a mirror variable d and split the Bellman flow constraints in the definition of F .
We then get the primal convex program
[Equation]
where the new polytope is given by M ≜ { ( µ , d ) | B ⊺ d = γ P ⊺ µ + (1 − γ ) ν 0 , µ = d , d ≥ 0 } .
This overparameterization trick has been first introduced by Mehta and Meyn [76] and has been recently revisited by [14, 84, 67, 83, 77, 71].
A salient feature of this equivalent formulation is that it introduces a Q -function as Lagrange multiplier to the equality constraint d = µ , and so lends itself to data-driven algorithms.
To motivate further this new formulation, in Appendix C, we shed light to its dual and provide an interpretation of the dual optimizers.
In particular, when W = B m 1 , we show that ( V ⋆ w true , Q ⋆ w true , w true ) is a dual optimizer.
For our theoretical analysis we focus on the linear MDP setting [55], i.e., we assume that the transition law is linear in the feature mapping.
We denote by ϕ ( s, a ) the ( s, a ) -th row of Φ .
Assumption 1 (Linear MDP) .
There exists a collection of m probability measures ω
= ( ω 1 , . .
. , ω m ) on S , such that P ( ·| s, a ) = ⟨ ω ( · ) , ϕ ( s, a ) ⟩ , for all ( s, a ) .
Moreover ϕ
( s, a ) ∈ ∆ [ m ] , for all ( s, a ) .
Assumption 1 essentialy says that the transition matrix P has rank at most m , and P = ΦM for some matrix M ∈ R m ×|S| .
It is worth noting that in the case of continuous MDPs, despite being linear, the transition law P ( ·| s, a ) can still have infinite degrees of freedom.
This is a substantial difference from the recent theoretical works on IL [122, 70, 105] which consider either a linear quadratic regulator, or a transition law that can be completely specified by a finite-dimensional matrix such that the degrees of freedom are bounded.
Assumption 1 enables us to consider a relaxation of (Primal).
In particular, we aggregate the constraints µ = d by imposing Φ ⊺ µ = Φ ⊺ d instead, and introduce a variable λ =
Φ ⊺ µ .
It follows that λ lies in the m -dimensional simplex ∆ [ m ] .
Then, we get the following convex program
[Equation]
where M Φ ≜ { ( λ , d ) | B ⊺ d = γ M ⊺ λ + (1 − γ )
ν 0 , λ =
Φ ⊺ d , λ ∈ ∆ [ m ] , d ∈ ∆ S×A } .
As shown in [84, 14, 83], for linear MDPs, the set of occupancy measures F can be completely characterized by the set M Φ (c.f., Proposition 2).
While the number of constraints and variables in (Primal ′ ) is intractable for large scale MDPs, in the next paragraph, we show how this problem can be solved using a proximal point scheme.
Proximal Point Imitation Learning
By using a Lagrangian decomposition, we have that (Primal ′ ) is equivalent to the following bilinear saddle-point problem
[Equation]
where A ∈ R (2 m + |S| ) × ( m + |S||A| ) , and b ∈ R ( m + |S| + |S||A| ) are appropriately defined (see Appendix D), x ≜ [ λ ⊺ , d ⊺ ] ⊺ , y ≜ [ w ⊺ , V ⊺ , θ ⊺ ] ⊺ , X ≜ ∆ [ m ] × ∆ S×A , and Y ≜ W × R |S| × R m .
Since in practice we do not have access to the whole policy π E , but instead can observe a finite set of i.i.d.
sample trajectories D E ≜ { ( x ( l ) 0 , a ( l ) 0 , x ( l ) 1 , a ( l ) 1 , . . .
, x ( l ) H , a ( l ) H ) } n E l =1 ∼ π E , we define the vector (cid:98) b by replacing ρ Φ ( π E ) with its empirical counterpart
ρ Φ ( (cid:99) π E ) (by taking sample averages) in the definition of b .
We then consider the empirical objective f ( x ) ≜ max y ∈Y (cid:68) y , Ax + (cid:98) b (cid:69) and apply PPM on the decision variable x .
For the λ -variable we use the relative entropy D ( λ || λ ′ ) ≜ (cid:80) mi =1 λ ( i ) log λ ( i )
λ ′ ( i ) , while for the occupancy measure d we use the conditional relative entropy
H ( d || d ′ ) ≜ (cid:80) s,a d ( s, a ) log π d ( a | s ) π d ′ ( a | s ) .
With this choice we can rewrite the PPM update as ( λ k +1 , d k +1 ) =
[Equation]
where we used primal feasibility to replace λ k with Φ ⊺ d k as the center point of the relative entropy.
PPM is implicit, meaning that it requires the evaluation of the gradient at the next iterate x k +1 .
Such a requirement makes it not implementable in general.
However, in the following, we describe a procedure to apply proximal point to our specific f ( x ) .
The following Proposition summarizes the result.
Proposition 2.
For a parameter θ ∈ R m , we define the logistic state-action value function Q θ ∈
R |S||A| by Q θ ≜ Φ θ , and the k -step logistic state value function V k θ ∈ R |S| by
[Equation]
Moreover, we define the k -step reduced Bellman error function δ k w , θ ∈ R m by δ k w , θ ≜ w + γ MV k θ − θ .
Then, the PPM update ( λ ⋆k , d ⋆k ) in 2 is given by
[Equation]
where ( w ⋆k , θ ⋆k ) is the maximizer over W × R m of the k -step logistic policy evaluation objective
[Equation]
Moreover, it holds that G k
[Equation]
1 α
[Equation]
⋆k ||
d k − 1 ) .
If in addition Assumption 1 holds, then d ⋆k is a valid occupancy measure, i.e., d ⋆k ∈ F and so d ⋆k = µ π d ⋆k .
The proof of Proposition 2 is broken down into a sequence of lemmas and is presented in Appendix E.
It employs an analytical-oracle g : Y
→ X given by
[Equation]
and a max-oracle h : X → Y given by h ( x ) ≜ arg max y ∈Y ⟨ y , Ag ( y ; x ) ⟩ + 1 τ D Ω ( g ( y ; x ) || x ) , where we used D Ω to compact the two divergences.
By noting that the PPM update Equation (2) can be rewritten as x k +
=
g ( h ( x k ); x k ) , its analytical computation is reduced to the characterization of the two aforementioned oracles.
In particular, the updates (3)–(4) come from the analytical-oracle while (5) is the objective of the max-oracle .
The choice of conditional entropy as Bregman divergence for the λ variable living in the probability simplex is standard in the optimization literature and is known to mitigate the effect of dimension.
In particular, as noted in [85], the classic REPS algorithm [90] can be seen as mirror descent with relative entropy regularization.
On the other hand, the choice of conditional entropy as Bregman divergence for the d variable is less standard and has been popularized by Q-REPS [14].
Such particular divergence leads to an actor-critic algorithm that comes with several merits.
By Proposition 2, it is apparent that we get analytical softmin updates for the policy π d rather than the occupancy measure d .
Moreover, these softmin updates are expressed in terms of the logistic Q -function and do not involve the unknown transition matrix P .
Consequently, we avoid the problematic occupancy measure approximation and the restrictive coherence assumption on the choice of features needed in [13, 58], as well as the biased policy updates appearing in REPS [90, 89].
In addition, the newly introduced logistic policy evaluation objective G k ( w , θ ) has several desired properties.
It is concave and smooth in ( w , θ ) and has bounded gradients.
Therefore, it does not suffer from the pathologies of the squared
Bellman error [78] and does not require heuristic gradient clipping techniques.
Moreover, unlike [58] it allows a model-free implementation without the need for a generative model (see Section 4.1)
We stress the fact that the max-oracle of our proximal point scheme performs the cost update and policy evaluation phases jointly.
This is a rather novel feature of our algorithm that differs from the separate cost update and policy evaluation step used in recent theoretical imitation learning works [122, 105, 70].
Our joint optimization over cost and Q -functions avoids instability due to adversarial training and can also recover an explicit cost along with the Q -function without requiring knowledge or additional interaction with the environment (see Section 5).
It is worth noting that application of primal-dual mirror descent to (SPP) does not have this favorable property.
While in the standard MDP setting, proximal point and mirror descent coincide because of the linear objective, in imitation learning proximal point optimization makes a difference.
In Appendix K, we include a more detailed discussion and numerical comparison between PPM and mirror descent updates.
4.1 Practical Implementation
Exact optimization of the logistic policy evaluation objective is infeasible in practical scenarios, due to unknown dynamics and limited computation power.
In this section, we design a practical algorithm that uses only sample transitions by obtaining stochastic (albeit biased) gradient estimators.
Proposition 2 gives rise to Proximal Point Imitation Learning ( P 2 IL ), a model-free actor-critic IRL algorithm described in Algorithm 1.
The key feature of P 2 IL is that the policy evaluation step involves optimization of a single smooth and concave objective over both cost and state-action value function parameters.
In this way, we avoid instability or poor convergence in optimization due to nested policy evaluation and cost updates, as well as the undesirable properties of the widely used squared Bellman error.
In particular, the k th iteration of P 2 IL consists of the following two steps : (i)
( Critic Step ) Computation of an approximate maximizer ( w k , θ k )
≈ arg max w , θ G k ( w , θ ) of the concave logistic policy evaluation objective, by using a biased stochastic gradient ascent subroutine; (ii) ( Actor Step ) Soft-min policy update π k ( a | s ) ∝ π k − 1 ( a | s ) e − αQ θ k
( s,a ) expressed in terms of the logistic Q -function.
The domain Θ in Algorithm 1 is the ℓ ∞ -ball with appropriately chosen radius D .
Moreover, Π Θ ( x ) ≜ arg min y ∈ Θ ∥ x − y ∥ 2 (resp.
Π W ( w ) ) denotes the Euclidean projection of x (resp.
w ) onto Θ (resp.
W ).
The following proposition ensures that max w , θ ∈W× R m G k ( w , θ ) = max w ,
[Equation]
∈W×
[Equation]
G
[Equation]
Therefore, this constraint does not change the problem optimality, but will considerably accelerate the convergence of the algorithm by considering smaller domains.
Proposition 3.
For all the maximizers θ ⋆k , the span seminorm [94] is bounded, ∥ θ ⋆k ∥ sp ≤ 1+ | log β | 1 − γ ≜
D .
In particular, there exists a maximizer such that ∥ θ ⋆k ∥ ∞ ≤ D .
In order to estimate the gradients ∇ θ G k ( w , θ ) and ∇ w G k ( w , θ ) we invoke the Biased Stochastic
Gradient Estimator subroutine (BSGE) (Algorithm 2) given in Appendix H. By using the linear
Input: Feature matrix Φ , expert demonstrations D E , number of iterations K , step sizes η and α , number of SGD iterations T, SGD learning rates β
= { β t } T − 1 t =0 , number-of-samples function n : N → N
Initialize π 0 as uniform distribution over A
Compute the empirical FEV ρ Φ ( (cid:99) π E ) using expert demonstrations D E for k = 1 , . . .
K do //
Critic-step (policy evaluation)
Initialize θ k, 0 = 0 and w k, 0 =
Run π k − 1 and collect i.i.d.
samples B k = { ( s ( n ) k − 1 , a ( n ) k − 1 , s ′ ( n )
k − 1 ) } n ( T ) n =1 such that ( s ( n ) k − 1 , a ( n ) k − 1 ) ∼ µ π k − 1 and s ′ ( n )
k − 1 ∼ P ( ·| x ( n ) k − 1 , a ( n ) k − 1 ) for t = 0 , . . .
T − 1 do
Compute biased stochastic gradient estimators ( (cid:98) ∇ w G k ( w k , θ k ) , (cid:98) ∇ θ G k ( w k , θ k ))
=
BSGE ( k, w t , θ t , n ( t ))
w k,t +1 =
Π W ( w k,t + β t (cid:98) ∇ w G k ( w k , θ k ))
θ k,t +1 = Π Θ ( θ k,t + β t (cid:98) ∇ θ G k ( w k , θ k )) end for ( w k , θ k ) =
( 1 T (cid:80)
Tt =1 w k,t , 1 T (cid:80)
Tt
=1 θ k,t ) //
Actor-step (policy update)
Policy update: π k ( a | s ) ∝ π k − 1 ( a | s ) e − αQ θ k
( s,a ) end for
Output: Mixed policy (cid:98) π K of { π k } k ∈ [ K ]
MDP Assumption 1 and leveraging ridge regression and plug-in estimators, the proposed stochastic gradients can be computed via simple linear algebra with computational complexity poly ( m, n ( t )) , independent of the size of the state space.
4.2 Theoretical Analysis
The first step in our theoretical analysis is to study the propagation of optimization errors made by the algorithm on the true policy evaluation objective.
In particular at each iteration step k , the ideal policy evaluation update ( w ⋆k , θ ⋆k ) and the ideal policy update π ⋆k are given by ( w ⋆k , θ ⋆k ) =
arg max w , θ G k ( w , θ ) , and π ⋆k ( a | s ) =
π k − 1 ( a | s ) e − α
( Q θ ⋆k ( s,a ) − V k θ ⋆k ( s )) .
On the other hand, consider the realised policy evaluation update ( w k , θ k ) such that G k ( w ⋆k , θ ⋆k ) − G k ( w k , θ k )
=
ϵ k , the corresponding policy π k given by π k = π k − 1 ( a | s ) e − α
( Q θ k ( s,a ) − V k θ k ( s )) , and let d k ≜ µ π k .
We denote by (cid:98) π K the extracted mixed policy of { π k } Kk =1 .
We are interested in upper-bounding the suboptimality gap
d C ( (cid:98) π K , π E ) of Algorithm 1 as a function of ε k .
To this end, we need the following assumption.
Assumption 2.
It holds that λ min ( E ( s,a ) ∼ d k ϕ ( s, a ) ϕ
( s, a ) T ) ≥ β , for all k ∈
[ K ] .
Assumption 2 states that every occupancy measure d k induces a positive definite feature covariance matrix, and so every policy π k explores uniformly well in the feature space.
This assumption is common in the RL theory literature [2, 46, 37, 66, 3, 7].
It is also related to the condition of persistent excitation from the control literature [81].
We can now state our error propagation theorem.
Theorem 1.
Let (cid:98) π K be the output of running Algorithm 1 for K iterations, with n E ≥ 2log( 2 m δ ) εexpert trajectories of length H ≥ 11 − γ log( 1 ε ) .
Let C ≜ 1 βη
(cid:0)(cid:113) 2 α 1 − γ + √ 8 η (cid:1) + (cid:113) 18 α 1 − γ .
Then, with probability at least 1 − δ , it holds that d C ( (cid:98) π K , π E ) ≤ 1 K (cid:16) D ( λ ∗ ||
Φ T d 0 )
η
[Equation]
By Theorem 1, whenever the policy evaluation errors ε k , as well as the estimation error ε can be kept small, Algorithm 1 ouputs a policy (cid:98) π K with small suboptimality gap ρ c true (
[Equation]
π
[Equation]
−
[Equation]
c true
[Equation]
E
[Equation]
Notably, there is no direct dependence on the size of the state space or the dimension of the feature space.
In the ideal case, where ε k = 0 for all k , the convergence rate is O (1 /K ) .
The provided error propagation analysis still holds with general function approximation, i.e., in the context of deep RL.
[Figure]
Indeed, by choosing Φ =
I , Assumption 1 is trivially satisfied and the θ variable in the objective G k is replaced by a Q -function.
In practice, the estimation error ε can be made arbitrary small, by increasing the number of expert demonstrations n E .
Moreover, the next theorem ensures that under Assumptions 1 and 2 the biased stochastic gradient ascent (BSGA) subroutine has sublinear convergence rate.
Theorem 2.
Let ( w k , θ k ) be the output of the BSGA subroutine in Algorithm 1 for T iterations, with n ( t ) ≥ max (cid:16) O
[Equation]
sample transitions, and learning rates
β t
=
O ( 1 √ t ) .
Then, ϵ k = G k ( w ⋆k , θ ⋆k ) − G k ( w k , θ k ) ≤
O ( max { η, 1 } mD β √ T ) , with probability 1 − δ .
Corollary 1 (Resource guarantees) .
Choose η = α = 1 and let K
= Ω (cid:0) ϵ − 1 (cid:1) , T = Ω (cid:0) ϵ − 4 (cid:1) .
Then for Ω ( KT ) =
Ω (cid:0) ϵ − 5 (cid:1) sample transitions, Ω (cid:0) ε − 2 (cid:1) expert trajectories and approximately solving Ω (cid:0) ϵ − 1 (cid:1) concave maximization problems, we can ensure d C ( (cid:98) π, π E ) ≤
O ( ϵ + ε ) , with high probability.
Offline Setting.
Finally, we notice that using Φ ⊺ µ π E as the reference distribution for the relative entropy we can obtain an offline algorithm that does not require environment interactions.
By reinterpreting smoothing [82] as one step of proximal point, and using similar arguments as in the proof of Theorem 1, we can provide similar theoretical guarantees for the offline setting.
The formal statement of the theoretical result as well as the optimization of the empirical policy evaluation objective are presented in Appendix J (see Theorems 4 and 6).
Experiments
In this section, we demonstrate that our approach achieves convincing empirical performance in both online and offline IL settings on several environments.
The precise setting is detailed in Appendix L.
Online Setting.
We first present results in various tabular environments where we can implement our algorithm without any practical relaxation outperforming GAIL [51], AIRL [38] and IQ-Learn [40].
Results are given in Figure 1.
Good performance but inferior to IQ-Learn is observed also for continuous states environments (CartPole and Acrobot) where we used neural networks function approximation.
Offline Setting.
Figures 2a to 2c shows that our method is competitive with the state-of-the-art offline IL methods IQLearn [40] and AVRIL [25] that recently showed performances superior to other methods like [54][64].
We also tried our algorithm in the complex image-based Pong task from the Atari suite.
Figure 2d shows that the algorithm reaches the expert level after observing 2 e 5 expert samples.
We did not find AVRIL competitive in this setting, and skip it for brevity.
In these settings, we verified that the algorithmic performance is convincing even for costs parameterized by neural networks.
Continuous control experiments.
We attain the expert performance also in 2 MuJoCo environments:
Ant , HalfCheetah , Hopper , and Walker (see Figures 2e to 2h).
The additional difficulty in implementing the algorithm in continuous control experiments is that the analytical form of the policy improvement step is no longer computationally tractable because this would require to compute an integral over the continuous action space.
Therefore, we approximated this update using the Soft
Actor Critic (SAC) [44] algorithm.
SAC requires environment samples making the algorithm online.
The good empirical result opens the question of analyzing policy improvement errors as in [41].
[Figure]
2d
Recovered Costs.
A unique algorithmic feature of the proposed methodology is that we can explicitly recover a cost along with the Q -function without requiring adversarial training.
In Figure 3, we visualize our recovered costs in a simple 5x5 Gridworld .
Most importantly, we verify that the recovered costs induce nearly optimal policies w.r.t.
the unknown true cost function.
Compared to
I Q -Learn [40], we do not require knowledge or further interaction with the environment.
Therefore, the recovered cost functions show promising transfer capability to new dynamics.
[Figure]
t
[Figure]
t e I n d e x
[Figure]
Cost Transfer Setting.
We experimented with a transfer cost setting on a Gridworld (Figure 4).
We consider two different Gridworld MDP environments, say M and (cid:102) M , with opposite action effects.
This means that action Down in (cid:102)
M corresponds to action Left in M and vice versa.
Similarly, the effects of Up and Right are swapped between (cid:102) M and M .
We denote by V π (cid:102)
M, c true (resp.
V ⋆ (cid:102) M, c true ) the value function of policy π (resp.
optimal value function) in the MDP environment (cid:102)
M with cost function c true .
Moreover, we denote by π ⋆M, c
the optimal policy in the MDP environment M under cost function c .
Figure (a) gives the corresponding optimal value function.
Figure (b) presents the value function of the expert policy π E
= π ⋆M, c true used as target by P 2 IL .
Figure (d) shows the value function of the learned imitating policy π K from P 2 IL .
Finally, Figure (b) depicts the value function of the optimal policy π ⋆ (cid:102) M, c K for the environment (cid:102)
M endowed with the recovered cost function c K by
P 2 IL (with access to samples from M ).
We conclude that the policy π ⋆ (cid:102) M, c K is optimal in (cid:102) M with cost c true .
By contrast, the expert policy π E = π ⋆M, c true used as target by P 2 IL performs poorly and as a consequence also the imitating policy π K does so.
All in all, we notice that the recovered cost induces an optimal policy for the new dynamics while the imitating policy fails.
Albeit, cost transfer is successful in this experiment we do not expect this fact to be true in general because we do not tackle the issue of cost shaping [87].
[Figure]
Discussion and Outlook
In this work, we studied a Proximal Point Imitation Learning ( P 2 IL ) algorithm with both theoretical guarantees and convincing empirical performance.
Our methodology is rooted in classical optimization tools and the LP approach to MDPs.
The most significant merits of P 2 IL are the following: (i) It optimizes a convex and smooth logistic Bellman evaluation objective over both cost and Q-functions.
In particular, it avoids instability due to adversarial training and can also recover an explicit cost along with Q function; (ii)
In the context of linear MDPs, it comes with efficient resource guarantees and error bounds for the suboptimality of the learned policy (Theorem 2 and Corollary 1).
In particular, given poly(1 /ε, log(1 /δ ) , m ) many samples , it recovers an ε -optimal policy, with probability 1 − δ .
Notably, the bound is independent of the size of the state-action space; (iii) Beyond the linear MDP setting, it can be implemented in a model-free manner, for both online and offline setups, with general function approximation without losing its theoretical specifications.
This is justified by providing an error propagation analysis (Theorems 1 and 4), guaranteeing that small optimization errors lead to high-quality output policy; (iv) It enjoys not only strong theoretical guarantees but also favorable empirical performance.
At the same time, our newly introduced methods bring challenges and open questions.
One interesting question is whether one can accelerate the PPM updates and improve the convergence rate.
Another direction for future work is to provide rigorous arguments for the near-optimality of the recovered cost function.
On the practical side, we plan to conduct experiments in more challenging environments than MuJoCo and Atari.
We hope our new techniques will be useful to future algorithm designers and lay the foundations for overcoming current limitations and challenges.
In Appendix B, we point out in detail a few interesting future directions.
Code repository
The code is available at the following link https://github.com/lviano/P2IL .
Acknowledgements
The authors would like to thank one anonymous reviewer for their suggestions to improve the presentation and for motivating us to inspect the recovered cost function.
This work has received financial support from the Enterprise for Society Center (E4S).
This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme grant agreement OCAL, No. 787845.
This work received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement n° 725594 - time-data).
This work was supported by the
Swiss National Science Foundation (SNSF) under grant number 200021_205011. Gergely Neu was supported by the European Research Council (ERC) under the European Union’s Horizonresearch and innovation programme (Grant agreement No. 950180). Luca Viano acknowledges travel support from ELISE (GA no 951847).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We reflect the contribution described in the introduction with the theoretical results in Section 4, Section 4.
(b) Did you describe the limitations of your work?
[Yes] We state the assumptions needed for the analysis in Sections 3 and 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The work is mainly theoretical, we do not foresee potential negative impacts on the society.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We acknowledge habing read the review guidelines.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We state the assumptions needed for the analysis in Section 3 Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] The main results are stated in Section 4.2 and the proofs are included as supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code is included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Training details are provided in the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We averaged multiple seeds whenever possible computationally.
We ran a single seed for the computational expensive Pong environment.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We specified the resource in the Supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite [40] for using their codebase and their expert data.
(b) Did you mention the license of the assets?
[Yes] We mention the license in the Supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We attach the code to the supplementary.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] We discuss this in the supplemnetary material (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
We do not use personal data.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not involve human partecipants.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
We did not involve human partecipants.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We did not involve human partecipants.
A Related Literature (Extended)
In order to state our research questions and situate them among prior related theoretical and practical works, we provide an extended literature review.
Theoretical Imitation Learning.
Our work is related to recent actor-critic IL schemes with theoretical guarantees for different MDP models, and different policy evaluation objectives (e.g., minimizing the squared Bellman error)
Contrary to these actor-critic schemes, in our proximal-point imitation learning algorithm, the policy evaluation step involves optimization of a single objective over both cost and Q -functions.
In this way, we avoid instability or poor convergence due to nested policy evaluation and cost update steps [40] as well as the undesirable properties of the widely used squared Bellman error [78].
Moreover, for the context of linear MDPs [14, 121, 55, 22, 115, 7, 84], we provide guarantees and convergence rates for the suboptimality of the learned policy, under mild assumptions, significantly weaker than those found in the literature until now.
To our knowledge, such guarantees in this setting are provided for the first time.
It is worth noting that in the case of continuous MDPs, despite being linear, the transition law can still have infinite degrees of freedom.
This is a substantial difference from the recent theoretical works on IL [21, 122, 26, 70, 105] which consider either tabular MDPs [105], or a linear quadratic regulator [21], or a linear transition law that can be completely specified by a finite-dimensional matrix [122, 70].
In the last case, the degrees of freedom are bounded, and thus mitigate the challenges in estimating the transition model.
Indeed, the linear MDP setting studied in [122, 70] reduces the unknown dynamics problem to estimating an unknown finite-dimensional matrix, which differs from our nonparametric approach.
We also note that [122, 118] require the restrictive assumption of bounded concentrability coefficients, while this is not the case for the analysis in this paper.
The convergence and generalization of actor-critic IL schemes for general MDPs has been studied in [28].
However, the authors in [28] only provide local optimality convergence guarantees, i.e., convergence to a stationary point.
On the contrary, our algorithm provides global convergence guarantees for the linear MDP setting.
Moreover, we account for potential policy evaluation errors , presenting an error propagation analysis that leads to rigorous guarantees for both online and offline setting, beyond the linear MDP assumption.
Indeed, it is worth noting that the provided error propagation analysis justifies using our derived actor-critic scheme with general function approximation.
A scalable deep reinforcement learning implementation is possible, without losing the theoretical guarantees of Theorem 1.
The work [26] studies offline IL for the continuous kernelized nonlinear regulator and Gaussian process setting [56].
We notice that this setting is different from the linear MDP model studied in this paper, and each one does not imply the other.
Finally, a recent theoretical IL work that is rooted in the LP approach to MDPs is [58].
The authors consider a Lagrangian reformulation of the problem and design a stochastic primal-dual algorithm with explicit performance bounds on the quality of the extracted policy.
The most important limitations of the primal-dual algorithm [58] are (i) the need of a generative oracle, (ii) restricted coherence assumptions on the choice of features, as well as (iii) the problematic occupancy measure approximation.
These limitations lead to poor practical performance for challenging high-dimensional and model-free IL setups.
On the other hand, our algorithm overcomes these difficulties by applying a proximal point update to an alternative Q -LP formulation [77, 83].
This results to a model-free actor-critic scheme with explicit tractable softmax policy updates.
Compared with the setting in [58], where access to a generative-model oracle is assumed, we only have the ability to execute learned policies in the underlying MDP to generate trajectories.
This assumption is considerably weaker that having a simulator-based MDP, however it is stronger than having ”irreversible experience”, where the learner must follow a single trajectory without having access to a reset action , that obtains a new trajectory from the initial state distribution.
Most importantly our algorithm enjoys not only strong theoretical guarantees, but also favorable practical performance.
Approximate Linear Programming.
There is an emerging body of literature [33, 1, 30, 109, 65, 79, 115, 67, 10, 16, 74, 31, 55, 106, 14] that studies ALP for the forward RL.
While this approach dates back to 1960s [73], it has recently witnessed an interesting renaissance for its potential to provide a solid formal framework for newly derived methods, as well as a deeper understanding of existing empirically successful algorithms.
In this paper, we present scalable imitation learning algorithms with theoretical guarantees rooted in the LP approach, highlighting how historical key limitations have been eliminated.
Prior approximate linear programming (ALP) approaches developed algorithms for solving large-scale and/or continuous MDPs on a low-dimensional subspace by reducing the number of constraints (e.g., by constraint sampling)
However, these prior works either scale badly with the size of the state-action spaces or require access to samples from a distribution that depends on the optimal policy.
Moreover, they focus mainly on the approximation of the optimal value but not so much on extracting a near optimal policy.
On the other hand, a recent line of works [30, 67, 115, 55, 106] solve the problem for large-scale MDPs by employing stochastic primal-dual methods, in light of Lagrangian duality.
Although this approach achieves state-of-the-art sample complexity guarantees, it shows poor performance in practice.
First, current primal-dual algorithms need access to a simulator, mitigating implicitly the problem of exploration, Second, when dealing with linear relaxations of MDPs [14, 58] one needs to impose a restrictive coherence assumptions to ensure that small duality gap for the linearly relaxed LP implies small suboptimality gap for the extracted policy.
Finally, while their is enough intuition behind the use of linear function approximation for value functions, this is not the case for occupancy measure approximation.
A new breed of algorithms that seem to overcome these difficulties is based on an alternative Q -LP formulation of RL.
This approach has been first introduced by Mehta and Meyn [76] and has been recently revisited by [14, 84, 67, 83, 77, 71].
A salient feature of this equivalent formulation is that it introduces a Q -function as slack variables, and so lends itself to data-driven algorithms.
Our work is inspired by these line of works.
The most related works are the analysis of REPS/Q-REPS [90, 14, 89] and O-REPS [124] that first pointed out the connection between REPS and PPM.
We build on their techniques with some important differences.
In particular, while in the LP formulation of RL, PPM and mirror descent [15, 47] are equivalent, recognizing that they are not equivalent in IL is critical for stronger empirical performance.
Moreover, our techniques can be used to improve upon the best rate for REPS in the tabular setting [89] and to extend their guarantees to Linear MDPs.
State-of-the-art Imitation Learning.
Generative adversarial imitation learning (GAIL) [51] and other follow-up works [38, 59, 63, 64] formulate the IL as a minimax adversarial problem similar to a
GAN [43] and leverage primal-dual optimization tools.
In particular, GAIL solves IL with alternating updates of both policy and cost functions.
On the other hand, a recent line of work [40, 11, 96] bypasses the need of optimizing over cost functions and thus avoids instability due to adversarial training.
Although these algorithms achieve impressive empirical performance in challenging high dimensional benchmark tasks, they are hampered by limited theoretical understanding.
This is the fundamental difference from our work, which enjoys both favorable practical performance and strong theoretical guarantees.
Moreover, a unique algorithmic feature of our proposed methodology is a convex and smooth logistic policy evaluation objective that optimizes jointly cost and Q -functions.
As a result, our algorithm has the additional practical benefit that can also recover an explicit cost along with the Q-function without requiring knowledge or further interaction with the environment (as in [40, 11, 96]).
Therefore, the recovered cost functions show promising transfer capability to new dynamics.
In addition, unlike IQ-Learn [40], in our online IL algorithm, instead of regularizing the IL objective, the key idea is to penalize the divergence between the current policy and the policy obtained at the previous iteration.
We do so by employing a Bregman proximal point update.
Most importantly, as we have already highlighted, the convergence properties of [40, 11, 96]) remain largely elusive in the function approximation and model-free regime.
It is unclear whether the sampling-based variants of their algorithms converge to a global optimum or if they converge at all, even for the simple tabular setting.
B Future directions
In this work, we studied a proximal point imitation learning algorithm with both theoretical guarantees and convincing empirical performance in challenging benchmark tasks.
Our methodology is rooted in classical stochastic optimization tools and in the LP approach to MDPs.
We hope that our new techniques will be useful for future algorithm designers and lay foundations for overcoming current limitations and challenges.
We point out a few interesting directions.
Accelerated proximal point.
An appealing possibility is to study an accelerated proximal point scheme with inexact updates and achieve faster convergence rates.
While there has been an effort in this direction [119, 120], the acceleration relies on the triangle/quadrangle scaling property assumption [45] that does not hold for KL divergence over the simplex.
Understanding if it is possible to accelerate PPM without such an assumption is an open question, whose solution has direct application to the LP formulation of RL and imitation learning.
Primal-dual methods with conditional relative entropy.
Recent primal-dual RL algorithms rooted in the LP approach to MDPs achieve state-of-the-art sample complexity guarantees.
See, for example, [13] for exact gradients, [23, 55] for stochastic gradients, and [58] for the imitation learning problem.
The most important disadvantages of primal-dual RL algorithms are (i) the need of a generative oracle, (ii) restricted coherence assumptions on the choice of features, as well as (iii) the problematic occupancy measure approximation.
Unfortunately, these limitations lead to poor practical performance for challenging high-dimensional and model-free RL and IL setups.
On the other hand, our algorithm overcomes these difficulties but requires to approximately solve a small-dimensional convex program repetitively.
It is also challenging to account for the biased gradient estimates beyond the linear MDP setting.
It is promising to investigate if by combining the alternative Q -LP formulation and the conditional relative entropy as Bregman divergence in a primal-dual mirror descent scheme, one can avoid the current practical limitations of primal-dual RL methods.
It is also interesting that in this case, the action-value parameters will be updated by taking one gradient step each time, instead of solving a small-dimensional convex program.
Inexact policy improvement update.
The error propagation presented in this work accommodates for errors only in the policy evaluation phase, while it assumes that the policy improvement step can be implemented exactly.
This happens in other related works like [14, 114].
In contrast, the error propagation analysis in [41] takes into account an error in the policy improvement step but unfortunately it does not provide a way to ensure that such an error is small.
Future research effort will aim to include in our error propagation analysis a term given by inexact policy improvement steps, ensure that such errors are small and characterizing the deterioration in the sample complexity under policy improvement errors.
This kind of analysis would be important for continuous actions environment where the softmax policy update can not be computed in closed-form.
C Dual Program Interpretation
To motivate further the Primal formulation, we shed light to its dual and provide an interpretation of the dual optimizers.
For brevity, we focus on the case W =
B
m 1 .
The proof can be found in
Appendix C.1 and is based on strong duality between the two convex programs.
Proposition 4.
The dual convex program is given by
[Equation]
Moreover, for W = B m 1 , a triple ( V A , Q A , w A ) is dual optimal if and only if (i) π E is optimal for the
RL problem with cost c = c w A , (ii) V A
= V ⋆ w A , (iii) Q A = Q ⋆ w A , and (iv) w A ∈ W .
In particular, ( V ⋆ w true , Q ⋆ w true , w true ) is a dual optimizer.
Proposition 4 states that the set of dual optimal costs c w A is the set of costs in C for which the expert is optimal.
In this case, the optimal V A coincides with the corresponding optimal value function 1 , while the optimal Q A coincides with the corresponding optimal state-action value function.
In particular, the true weights w true , the true optimal value function V ⋆ w true and the true optimal state-action value function Q ⋆ w true are dual optimizers.
Therefore, the presented Q -convex approach allows to recover an optimal solution to the original problem (1) from both the (Primal) and (Dual) formulations: it can be obtained either as the induced policy of a primal optimal occupancy measure or as a greedy policy associated to a dual optimal Q -function.
In Section 4, we generalize the later observation to implement PPM using softmin updates in terms of Q -functions.
C.1 Proof of Proposition
We recall the alternative Q -LP approach to MDPs [76, 77, 84, 83, 14].
Let c ∈ R
|S||A| be a cost function.
The forward RL problem is equivalent to the following linear programs
[Equation]
(Primal
[Equation]
We have that if π ⋆ is an optimal policy for the forward RL problem with cost c , then ( µ π ⋆ , µ π ⋆ ) is optimal for (Primal Q -LP) and conversely if ( µ ⋆ , d ⋆ ) is optimal for (Primal Q -LP), then π µ ⋆ is an optimal policy for the forward RL problem with cost c .
Moreover, ( V ⋆ c , Q ⋆ c ) is an optimal solution to (Dual Q -LP) and it is the unique optimizer when ν 0 ∈ R |S| ++ .
For the following results, we will assume without loss of generality that ν 0 ∈ R |S| ++ .
Proof of Proposition 4.
We first derive the dual convex program.
We have,
[Equation]
where the second equality follows by Sion’s minimax theorem [107], since M is convex and compact, W is convex and the objective is bilinear, the third equality follows by introducing Lagrange multipliers V and Q , and the fifth equality follows by linear duality.
Note that the derivations hold for any convex set W .
From now on we consider the case W =
B m 1 = { w ∈ R m | ∥ w ∥ 2 ≤ 1 } .
Then, the (Primal) program can be written in the form
[Equation]
where in the last equality we used that the ℓ 2 -norm is self-dual, that is, the dual norm of the ℓ 2 -norm is still the ℓ 2 -norm.
Therefore, when W = B m 1 , we get a quadratic objective with linear constraints [4].
Assume first that ( V A , Q A , w A ) is optimal for (Dual).
Then,
[Equation] [Equation]
where (6) holds because ( V A , Q A , w A ) is feasible to (Dual), and (7) holds by optimality.
Therefore, ( V A , Q A ) is feasible for (Dual Q -LP) with cost c
= c w A .
Moreover, ( µ π E , µ π E ) is feasible for (Primal
Q -LP) with cost c =
c w A .
Therefore,
[Equation]
However, by (7) we get that (1 − γ ) ⟨ ν 0 , V A ⟩ = (cid:10) µ π E , c
w A (cid:11) .
Thus, ( µ π E , µ π E ) is optimal for (Primal
Q -LP) with cost c = c w A and ( V A , Q A ) is optimal for (Dual Q -LP) with cost c
= c w A .
Thus π E is optimal for the forward RL problem with cost c w A , V A
= V ⋆ c w A , and Q A
= Q ⋆ c w A
Conversely, assume that w A ∈ W , π E is optimal for c w A , V A
= V ⋆ w A , and Q A = Q ⋆ w A .
Then, we have that ( µ π E , µ π E ) is optimal for (Primal Q -LP) with cost c w A , and ( V A , Q A ) is optimal for (Dual
Q -LP) with cost c w A .
By dual feasibility, we get
[Equation]
Moreover, by primal-dual optimality, we have
[Equation]
From (9), we get that ( V A , Q A , w A ) is feasible to (Dual).
Since ζ ⋆ = 0 , by (10), we conclude that ( V A , Q A , w A ) is optimal for (Dual).
D Saddle-Point Formulation
By using a compact notation, we have that Primal ′ is equivalent to the following bilinear saddle-point problem
[Equation]
where
[Equation]
E Proof of Proposition
Proof of Proposition 2.
We break the proof in three parts.
In the first two parts, we introduce and compute the explicit forms of the oracles, while in the third part we derive the proximal point updates.
Analytical oracle.
We characterize the analytical-oracle by employing the first-order optimality conditions for λ and d .
In particular, at each iteration step k , for any [ w ⊺ , V ⊺ , θ ⊺ ] ⊺ , we have that the
Lagrangian of the optimization problem in the definition of the analytical-oracle has the form ⟨
[Equation]
, where we considered a Lagrangian multiplier τ for the simplex constraint (cid:80) i λ ( i )
= 1 .
Now taking the derivatives with respect to to λ and d , we obtain the following first order optimality conditions:
[Equation]
Therefore, we obtain
[Equation]
where δ k w , θ ≜ w + γ MV k θ − θ .
In addition, the simplex constraint (cid:80) i λ ( i )
= 1 is satisfied by choosing τ = τ k w , θ , where
[Equation]
Moreover, by setting Q θ =
Φ θ , we get
[Equation]
Equation (4) follows by noting that the constraint (cid:80) a π d ( a | x ) implies that V equals the logistic value function V k θ given in Proposition 2.
Finally, since ( λ k , d k ) are ideal updates, they are primal feasible.
Hence, we can use the constraint λ k =
Φ ⊺ d k in Equation (11) to obtain Equation (3).
All in all, for any y
=
[ w ⊺ , V ⊺ , θ ⊺ ] ⊺ the analytical-oracle outputs g ( y ; x k ) =
[ λ ⊺ , d ⊺ ] ⊺ with
[Equation]
Note that the derivatives with respect to λ and d differ from the ones in Logistic Q-Learning [14].
In our case, δ k w , θ depends on both cost weights w and logistic action-value parameters θ .
In addition, δ k w , θ is the reduced Bellman error in the feature space rather than in the high dimensional state-action space.
Max oracle.
Since the objective in (2) is convex in x and linear in y , X is convex and compact, and Y is convex, by virtue of Sion’s minimax theorem [107], we can exchange the min and max in
Equation (2).
We then have
[Equation]
Therefore, we get
[Equation]
Proximal point updates via max and analytical oracles.
It remains to prove the closed-form expressions for π d ⋆ and λ ⋆ given in Equation (3) and Equation (4), respectively.
We start rewriting the objective of the max-oracle as a function of λ and d .
In particular, we have
[Equation]
The minimizers of the previous expression are characterized via the analytical-oracle .
In particular, plugging in the analytical forms for λ , d and V , we obtain
[Equation]
≜
[Equation]
This is the objective of the max-oracle in Proposition 2.
Given that the max-oracle returns ( w ⋆k , θ ⋆k ) , the corresponding primal variables ( d ⋆k , λ ⋆k ) satisfy ( d ⋆k , λ ⋆k ) =
g ([ w ⋆k , V θ ⋆k , θ ⋆k ]; x k − 1 ) .
This completes the proof of the first part of Proposition 2.
It remains to show the dual form of the max-oracle objective G k ( w , θ ) .
In particular, we will show that
[Equation]
We first recall that
[Equation]
Then, by taking the maximum over y =
[ w , V , θ ] on both sides and using Sion’s minimax theorem, we get
[Equation]
where in the last equality we used the definition of proximal point update in Equation (2).
Finally, by LP strong duality, we have that max w ⟨ λ k +1 , w ⟩ − ⟨ ρ Φ ( (cid:99) π E ) , w ⟩ = max y ∈Y
[Equation]
+
[Equation]
b
[Equation]
Hence, we conclude that (16) holds.
F Proof of Proposition
Proof of Proposition 3.
From first order optimality conditions for λ
⋆k , we get
[Equation] [Equation]
We define the regularized cost weights by (cid:101) w ⋆k ≜ w ⋆k + 1 η log λ
⋆k ( i ) λ
k − 1 ( i ) , and the costant (wrt the vector index i ) c ≜ − τ k w ⋆k , θ ⋆k + 1 η .
This gives for all i ∈ [ m ]
[Equation]
We define the span norm as ∥ x ∥ sp = inf c ∈ R ∥ x − c 1 ∥ ∞ .
Then multiplying by Φ from the left, we have that Φ (cid:101) w ⋆k + γ PV k θ ⋆k − Φ θ ⋆k =
c 1 .
Moreover, we can write
[Equation]
We set ( T V k θ ⋆ k )
[Equation]
s ) ≜ − 1 α log
[Equation]
a π d k
[Equation]
1 ( a | s ) e − α
[Equation]
Φ
[Equation]
w ⋆k + γ PV k θ ⋆k )( s,a ) (cid:19) .
Note that T is the softBellman operator [86, 41] that is a γ -contraction with respect to ∥·∥ ∞ -norm.
It follows that
[Equation]
[Table]
Therefore,
[Equation]
≤
[Equation]
+log 1 β
[Equation]
− γ .
Moreover, using the relation (cid:101) w ⋆k + γ MV k
[Equation]
− θ ⋆k
= c 1 , we have that
[Equation]
This proves that for every maximizer the span norm is bounded.
Finally, for showing that there exists a maximizer with bounded infinity norm, we want to prove that the negative logistic Bellman error is shift invariant in θ .
That is, G k ( w , θ + c 1 ) =
G k ( w , θ ) .
Towards this goal, we start proving that
V k θ + c 1 =
V k
θ + c for any constant c ∈ R .
Indeed,
[Equation]
At this point
[Equation]
we can show the shift invariance of G k .
[Equation]
It follows that there exists a maximizer θ ⋆k for which ∥ θ ⋆k ∥ sp =
∥ θ ⋆k ∥ ∞ .
To see this, we show that we can find a value of c for which the span seminorm equals the ℓ ∞ -norm, that is ∥ θ ⋆ k + c 1 ∥ ∞ =
∥ θ ⋆ k ∥ sp .
By definition of the span norm (and assuming that the infimum is attained), the equality is attained for c =
arg min c ∥ θ ⋆k + c 1 ∥ ∞ =
max i ∈
[ m ] θ ⋆k ( i )+min i ∈ [ m ] θ ⋆k ( i ).
Then, choosing the shift for which max i ∈ [ m ] θ ⋆k ( i ) = − min i ∈ [ m ] θ ⋆k ( i ) , gives the maximizer for which ∥ θ ⋆k ∥ sp =
∥ θ ⋆k ∥ ∞ .
This concludes the proof for the bound on the ℓ ∞ -norm.
G Proof of Theorem
We will analyze the proximal point method applied to SPP.
We use a similar error propagation analysis as in [14].
By Proposition 2, the ideal updates ( θ ⋆k , w ⋆k , π ⋆k , λ ⋆k , d ⋆k ) are given by
[Equation]
where τ k θ ⋆k , w ⋆k is a normalization constant.
By feasibility of the ideal updates we also have λ ⋆k =
Φ ⊺ d ⋆k
On the other hand, the realized updates ( θ k , w k , π k , λ k , d k ) are given by
[Equation]
k
[Equation]
where τ k θ k , w k is a normalization constant, and the notation ( w k , θ k )
=
arg max w , θ G ϵ k k ( w , θ ) means that G k ( w ⋆ k , θ ⋆ k ) − G k ( w k , θ k ) =
ϵ k .
We start by introducing some auxiliary results
Lemma 1.
For any occupancy measures d 1 , d 2 ∈ F , and for any cost vectors c , c ′ ∈ C , we have:
[Equation]
Proof.
We have that
[Equation]
Corollary 2.
Let d ⋆ = argmin d ∈ F max c ∈C ⟨ d , c ⟩−⟨ µ π E , c ⟩ .
Setting c
=
Φw k , d 1 = d ⋆ , d 2 =
d k , we ge that ⟨ µ π E − d ⋆ , Φw k ⟩ − min c ∈C ⟨ µ π E − d
[Equation] [Equation]
≥ d
[Equation]
π
[Equation] [Equation]
d k
[Equation]
d
[Equation]
, π
[Equation]
Lemma 2.
It holds that D ( λ ⋆k ||
λ k )
η
[Equation]
Proof.
The proof is analogous to Lemma 1 in [14].
Lemma 3 (First order optimality conditions for G k ) .
For all k ∈ [ K ] , it holds that
[Equation]
Proof.
We start by taking the gradient of G k ( w k , θ k ) with respect to w .
In particular, the partial derivative with respect to the i th component is given by
[Equation]
Therefore,
[Equation]
Then, by using the first-order optimality conditions for a concave function, we have
[Equation]
By replacing the expression for ∇ w G k
w ⋆k , θ ⋆k ) , we obtain
[Equation]
We also need the following auxiliary result
[Equation]
Lemma 4.
For all k ∈ [ K ] , it holds that
[Equation]
Proof.
By introducing ¯ w ⋆k = arg min w ∈W ⟨ ρ Φ ( (cid:99) π E ) − Φ ⊺ d k , w ⟩
[Equation]
and applying triangular inequality
[Equation]
we obtain
[Equation]
Moreover
[Equation]
we have
[Equation]
The first equality holds because the term in w ⋆k is a constant wrt w , the variable of the max.
In the last inequality follows from (A) being zero as we show next:
[Equation]
Lemma 5 (Lower Bound on feature expectation vectors) .
Let Assumption 2 hold.
We then have ( Φ ⊺ d k )( j ) ≥ β for all j ∈ [ m ] .
Proof.
Let e j ∈ R m the vector with zeros everywhere but in position j where it takes the value of 1 .
Then, we observe that
[Equation]
Theorem 3 (Error propagation with empirical expert feature expectation vector) .
Let d ⋆ = argmin d ∈ F max c ∈C ⟨ d , c ⟩ − ⟨ µ π E , c ⟩
[Equation]
and let λ ⋆ be any state-action occupancy measure such that
( λ ⋆ , d ⋆ ) ∈
M Φ .
Moreover, let C ≜ 1 βη
(cid:0)(cid:113) 2 α 1 − γ + √ 8 η (cid:1) + (cid:113) 18 α 1 − γ .
Then, we have that
[Equation]
Proof.
We have that
[Equation]
⋆ , V k θ k ⟩ − η G k ( θ k , w k ) − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ B ⊺ d ⋆ , V k θ k ⟩ − η G k
( θ ⋆k , w ⋆k )
+ ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ ||
Φ ⊺ d k − 1 ) + η ⟨ λ ⋆ , w k − θ k ⟩ + η ⟨ B ⊺ d ⋆ , V k θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − λ
⋆k , w ⋆k ⟩ − D ( λ ⋆k ||
Φ ⊺ d k − 1 ) − η H ( d ⋆k || d k − 1 ) α + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ || Φ ⊺
d
k − 1 )
+ η ⟨ λ ⋆ , w k ⟩ + η ⟨ d ⋆ , BV k θ k − Φ θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − λ
⋆k , w ⋆k ⟩ + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ || Φ ⊺ d k − 1 ) + η ⟨ d ⋆ , Φw k ⟩ + η ⟨ d ⋆ , BV k θ k − Φ θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − Φ ⊺ d k , w ⋆k ⟩ + η ⟨ d k − d ⋆k , Φw ⋆k ⟩ + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ ≤ D ( λ ⋆ || Φ ⊺ d k − 1 ) + η ⟨ d ⋆ , Φw k ⟩ + η ⟨ d ⋆ , BV k θ k − Φ θ k ⟩ + η ⟨ ρ Φ ( (cid:99) π E ) − Φ ⊺ d k , w ⋆k ⟩ + η ∥
d
k − d ⋆k ∥ 1 + ηϵ k − η ⟨ ρ Φ ( (cid:99) π E ) , w k ⟩ Using Lemma≤ D ( λ ⋆
[Equation]
Therefore
[Equation]
it follows that
[Equation]
Then, by using H ( d ⋆ || d k ) =
H ( d ⋆ || d k − 1 ) − α ⟨ d ⋆
[Equation]
Φ θ k − BV k θ k ⟩
[Equation]
we obtain
[Equation]
Summing over iteration indices k and dividing by the total number of iterations K , we obtain
K
[Equation]
Moreover, by a telescoping sum
[Equation]
we get
[Equation]
η
Combining this derivation with (21), we get
K (cid:88)
k
[Equation]
In order to bound the term D ( λ ⋆ || Φ ⊺ d k ) − D ( λ ⋆ || λ k ) , we introduce the Bregman projection to the space of feature expectation vectors induced by valid occupancy measures ˜ λ k = arg min { λ = Φd | d ∈F} D ( λ || λ k ) .
We then have
[Equation]
where in the second inequality
[Equation]
we used Lemma 11.3 in [24].
Furthermore,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
, where we used max i λ ⋆ ( i )
Φ ⊺ d k ( i ) ≤ 1 β thanks to Lemma 5 while in the last line we use the fact that
H ( d ⋆k || d k ) is positive and the equality in Lemma 2.
To bound the ℓ 1 -norm, we apply Pinkser’s inequality and Lemma 2 in [14] to get that
[Equation]
Plugging the last derivation in Equation (22) gives
K
[Equation]
Finally, using Lemma 3 we have that the term ⟨ ρ Φ ( (cid:99) π E ) − λ
⋆k , w ⋆k − w k ⟩ is non positive.
Therefore,
[Equation]
where
[Equation]
Finally, we need a Lemma that provides a concentration for the estimated expert feature expectation vector.
Lemma 6 ([111]) .
Let D π E ≜ { ( s ℓ 0 , a ℓ 0 , s ℓ 1 , a ℓ 1 , . . .
, s ℓH , a ℓH ) } n E ℓ =1 ∼ π E be a finite set of i.i.d.
truncated sample trajectories.
We consider the empirical expert feature expectation vector ρ Φ ( (cid:99) π E ) by taking sample averages, i.e.,
[Equation]
Suppose the trajectory length is H ≥
− γ log( 1 ε ) , and the number of of expert trajectories is n
[Equation]
Then, with probability at least 1 − δ , it holds that ∥ ρ Φ
[Equation]
π
[Equation]
At this point, Theorem 1 is proven from the results of Theorem 3,Lemma 6 and Lemma 1.
H Biased Stochastic Gradients and their Properties
In order to estimate the gradient ∇ θ G ( w , θ ) , we define the policy π k, θ ( a | s ) ∝ π
[Equation]
( a | s ) e − αQ θ ( s,a ) , for all k ∈ N , and for all θ ∈ R m .
Then, by standard computations we get that for all ( w , θ ) , and for all j ∈ [ m ] ,
[Equation]
where B k w , θ ( i ) ≜ exp (cid:0) − η δ k w , θ ( i )
[Equation]
Z k , Z k ≜ (cid:80) mi =1 exp (cid:0) − η δ k w , θ ( i ) (cid:1) ρ Φ ( π k − 1 )( i ) , and Γ k ( i, j ) ≜ (cid:80) s ′ ,a ′ M i,s ′ π k − 1 , θ ( a ′ | s ′ )
ϕ
j ( s ′ , a ′ ) , .
Similarly, for the gradient ∇ w G ( w , θ ) , we can write
[Equation]
Note that the following estimators of ∇ θ G k ( w , θ ) and ∇ w G k ( w , θ ) are unbiased: Sample ( s ′ , a ′ ) ∼ d k − 1 , i ′ ∼ ϕ
( s ′ , a ′ ) , s 0 ∼ ν 0 , and a 0 ∼ π k − 1 , θ ( ·| s 0 ) , then define
[Equation]
[Equation]
These expressions give rise to the Biased Stochastic Gradient Estimator subroutine (BSGE) given in Algorithm 2, where we plug-in estimators (cid:98) B k w , θ ∈ R m and (cid:98) Γ k ∈ R m × m to Equations (24) and (25).
It remains to show how to maintain good estimators (cid:98) B k w , θ and (cid:98)
Γ k by using the linear MDP
Assumption 1.
While the estimator (cid:98) Γ k ∈ R m is a standard ridge regression estimator, the construction of (cid:98) B k w , θ is more involved.
In particular, we first need to build an estimator for the product MV kθ via ridge regression.
Then, the estimator for (cid:98) B k w , θ is derived by plugging-in the estimator of MV kθ , and the estimator for the feature expectation vector ρ Φ ( π k − 1 ) to equation B k w , θ (
[Equation]
The reasoning and analysis is inspired by [52, 89].
H.1 Ridge estimators
This section leverages ridge regression [53] to build estimators (cid:98) B k w , θ and (cid:98) Γ k ∈ R m × m .
We work under the Assumption 2 which ensures that every iterate covers the features space.
We recall that by
Lemma 5, Assumption 2 implies that Φ ⊺ d k ( s, a ) ≥ β , for all k ∈ [ K ] .
Input: Policy evaluation step k , reference points ( w , θ ) , number of samples N
Compute empirical estimators (cid:98) δ k w , θ ∈ R m , (cid:98) Γ k ∈ R m × m , ρ Φ ( (cid:91) π k − 1 ) ∈
R m using the first N samples { ( s ( n ) k − 1 , a ( n ) k − 1 , s ′ ( n ) k − 1 ) }
Nn =1 from the buffer B k for i = 1 , . .
. , m do
Compute (cid:98) B k w , θ ( i ) = exp (cid:0) − η (cid:98) δ k w , θ ( i ) (cid:1) (cid:98) Z k , Where (cid:98)
Z k = (cid:80) mi =1 exp (cid:0) − η (cid:98) δ k w , θ ( i ) (cid:1) ρ Φ ( (cid:91)
π k − 1 )( i ) end for
Sample ( s ( N +1)
k − 1 , a ( N +1)
k − 1 ) ∼ d k − 1 , i ( N +1)
k −∼ ϕ ( s ( N +1)
k − 1 , a ( N +1)
k − 1 )
Sample s (0) k − 1 ∼ ν 0 , and a (0) k − 1 ∼ π k − 1 , θ ( ·| s 0 )
Compute (cid:98) ∇ w ,j G k ( w , θ ) = − ρ Φ ( (cid:99) π E )( j ) + (cid:98) B k w , θ ( i ( N +1)
k − 1 ) 1 { i ( N +1) k −= j } (cid:98) ∇ θ ,j G k ( w , θ )
= (cid:98) B k w , θ ( i ( N +1)
k − 1 ) (cid:104) γ (cid:98) Γ k
( i ( N +1)
k − 1 , j ) − 1 { i ( N +1) k −= j } (cid:105) +
[Equation] [Equation]
Output: ( (cid:98) ∇ w G k ( w , θ ) , (cid:98) ∇ θ G k ( w , θ ))
H.1.1 Estimator for MV kθ
We first construct an estimator for M k V kθ .
We can start noticing that we can rewrite M k V kθ using the feature covariance matrix ¯ Λ k ≜
E ( s,a ) ∼ d k −(cid:2) ϕ ( s, a ) ϕ ( s, a ) T (cid:3) as showed by the next lemma.
Lemma 7.
It holds that MV
[Equation]
Proof.
[Equation]
It follows that MV kθ
[Equation]
min
[Equation]
Now, we move to the problem of estimating (cid:92) MV kθ with a finite amount of environment interactions sampled i.i.d from d k − 1 .
We define
[Equation]
By optimality conditions, we can obtain a closed-form expression for (cid:92) MV kθ .
Lemma 8.
It holds that
[Equation]
where Λ k,N ≜ 1 N (cid:80) Nn =1 ϕ ( s ( n ) k − 1 , a ( n ) k − 1 ) ϕ
( s ( n ) k − 1 , a ( n ) k − 1 ) T is the empirical covariance matrix.
Proof.
Let L ( z ) ≜ 1 N (cid:80)
Nn =1 (cid:16) ϕ
( s ( n ) k − 1 , a ( n ) k − 1 )
T z − V kθ ( s ′ ( n ) k − 1 ) (cid:17) 2 + χ ∥ z ∥ 22 .
The first derivative is given by
[Equation] [Equation]
Since L ( · ) is convex in z , by first-order optimality conditions, we get
[Equation]
The statement follows from rearranging the terms.
Remark 1.
Note that when χ = 0 , and ϕ ( s, a ) is one-hot vector for every ( s, a ) , then we obtain the tabular estimators W v proposed in [89].
We invoke Theorem 2 in [53] to derive an upper bound for (cid:13)(cid:13)(cid:13)
[Equation]
Lemma 9.
Fix some χ > 0 and take N ≥ O
( log( mδ ) χβ ) .
Then, with probability at least 1 − δ , we have
[Equation]
, where D ≜ 1+log ( 1 β ) 1 − γ ≥ 1 is the upper bound of (cid:13)(cid:13) V kθ (cid:13)(cid:13) ∞ derived in Proposition 3.
Proof.
We introduce the following auxiliary quantities:
[Equation]
and the conditional expectation ¯
[Equation]
s ′ ) (cid:3) with F n being the filtration F n = { s ( i ) k − 1 , a ( i ) k − 1 } ni =0 .
Then applying the general random design decomposition in ([53], Proposition 3) we obtain:
[Equation]
where similarly to [53], we define ϵ rg as the ridge error, ϵ bs the ridge estimator bias and with ϵ vr the ridge estimator variance.
By choosing N ≥ O (6 ρ 2 χ d 1 ,χ (log max (1 , d 1 ,χ )+log 1 δ ))
=
O ( 1 βχ log mδ ) , we ensure that the conditions in Theorem 2 in [53] are satisfied.
We next bound each term separately.
Ridge error.
In [53], the bound derived for the ridge error is a function of the regularization parameter χ , the eigenvalues of the covariance matrix ¯ Λ k denoted as { σ j } mj =1 and the corresponding eigenvectors { v j } mj =1 .
In particular, we have
[Equation]
where in the first inequality we used bullet (3) of Theorem 2 in [53].
Bias.
It holds that
[Equation]
where we used the notation
[Equation]
Moreover,
[Equation] [Equation]
Finally, according to Remark 2 in [53], we have that ρ χ is bounded as follows
[Equation]
where the last inequality follows from noticing that d 1 ,χ ≥ βm 1+ χ .
Therefore, we can conclude that:
[Equation]
N log (cid:18)
[Equation]
δ (cid:19)(cid:33) =
O (cid:18)
[Equation]
(cid:18)δ (cid:19)(cid:19) =
O (cid:18)
N mχ β 4 D 2 log (cid:18)δ (cid:19)(cid:19) ,
Variance.
From the bullet (5) in [53] it follows that
[Equation]
O
[Equation]
δ
[Equation]
We have Var (cid:2)
V kθ ( s ′ ) | s, a (cid:3) ≤ (cid:13)(cid:13)
V kθ (cid:13)(cid:13) 2 ∞ ≤ D 2 .
Finally, bounding d 2 ,χ we obtain that
[Equation]
Hence we can conclude
[Equation]
Final bound.
By combining the above bounds with Equation (27), we get the final bound
[Equation]
The bound above is minimized by choosing χ as small as allowed.
This is made precise in the next corollary.
Corollary 3.
Let χ =
O ( log mδ βN ) .
With probability at least 1 − δ , it holds that
[Equation]
2 + mD
N log (cid:18)δ (cid:19)(cid:19) .
In order to upper bound (cid:13)(cid:13)(cid:13) MV kθ − (cid:92) MV kθ (cid:13)(cid:13)(cid:13) 2
we need the next lemma.
Hence, to bound (cid:13)(cid:13)(cid:13) MV kθ − (cid:92) MV kθ (cid:13)(cid:13)(cid:13) 2 2 , we can directly apply Theorem 2 in [53] that leads to the following lemma.
Lemma 10.
Given a matrix A ∈ R m × m and a vector x ∈ R m , we have that ∥ x ∥
A ≥ λ min ( A ) ∥ x ∥ 2 .
Proof.
We have that A − λ min ( A ) I ≥ 0 that implies x ⊺ Ax ≥ λ min ( A ) x ⊺ x .
Corollary 4.
Let χ =
O ( log mδ βN ) .
With probability at least 1 − δ , it holds that
[Equation]
Corollary 5.
Let χ = O (cid:16) log mδ βN (cid:17) , and N ≥ max (cid:16) γ 2 mD 2 βϵlog(1 /δ ) , γ √ mDβ 3 ϵ log( m/δ ) (cid:17) .
Then, with probability at least 1 − δ , it holds that (cid:13)(cid:13)(cid:13) MV kθ − (cid:92) MV kθ (cid:13)(cid:13)(cid:13) 2 ≤ ϵγ .
H.1.
2 Estimators for Γ k
Recall that we introduced Γ k ( i, j ) ≜ (cid:80) s ′ ,a ′ M i,s ′
π k − 1 , θ ( a ′ | s ′ )
ϕ
j ( s ′ , a ′ ) .
We can equivalently rewrite it as
[Equation]
where the last equality is obtained with manipulations analogous to Lemma 7.
Similarly, We can estimate Γ k
( i, j ) with a finite amount of environment interactions sampled i.i.d.
from d k − 1 , by solving the following ridge regression problem:
[Equation]
Lemma 11.
By optimality conditions, we can obtain a closed form for (cid:98) Γ k as
[Equation]
By noting that ∥ h k − 1 ,j ∥ ∞ ≤ 1 for any k , it follows that
Corollary 6.
For χ = O ( log mδ βN ) , with probability at least 1 − δ , it holds that
[Equation]
Corollary 7.
For χ = O ( log mδ βN ) , and N ≥ max (cid:16) O (cid:16) mβϵ 2 log(1 /δ )
(cid:17) , O (cid:16) √ mβ 3 ϵ log( m/δ ) (cid:17)(cid:17) , with probability at least 1 − δ , it holds that (cid:13)(cid:13)(cid:13)
[Equation]
H.1.3 Estimator for feature expectation vector ρ Φ ( π k − 1 )
The goal is to estimate ρ Φ ( π k − 1 ) .
Consider the sample transitions { s ( n ) k − 1 , a ( n ) k − 1 } Nn =1 ∼ d
Nk − 1 .
Then we estimate ρ Φ ( π k − 1 ) =
Φ ⊺ d k − 1 by ρ Φ ( (cid:91)
π k − 1 ) ≜ 1 N (cid:80) Nn
=1 ϕ ( s ( n ) k − 1 , a ( n ) k − 1 ) .
In the next lemma, we provide a useful concentration result.
Lemma 12.
With probability at least 1 − δ , for all N ≥ 1 . 4loglog(2 N )+log 10 .
4 m δ βϵ, and for all i ∈ [ m ] simultaneously, it holds that
[Equation]
Proof.
Consider the martingale difference sequence Z i ( n )
=
ϕ
i ( s ( n ) k − 1 , a ( n ) k − 1 ) − ρ Φ ( π k − 1 )
( i ) with the variance process V i ( n ) =
(cid:80) nj =1 E (cid:2)
Z 2 i ( j ) |F
j − 1 (cid:3) , where F j − 1 being the filtration up to the state action pair ( s ( j ) k − 1 , a ( j ) k − 1 ) .
We have,
[Equation]
The martingale difference sequence Z i ( j ) satisfies the sub- ψ P condition of [52] (see Bennet case in their Table 3) with constant c = 2 .
Therefore, by Lemma 13 in [89] with m = ρ Φ ( π k − 1 )( i ) , with probability at least 1 − δ 2 m , for all N ≥ 1 . 4loglog(2 N )+log 10 .
4 m δ βϵsimultaneously, it holds that
[Equation]
Similarly, with probability at least 1 − δ 2 m , for all N ≥ 1 . 4loglog(2 N )+log 10 .
4 m δ βϵsimultaneously, it holds that ρ Φ ( (cid:91) π k − 1 )( i ) ≤ ρ Φ ( π k − 1 )( i ) + 2 .
26 ρ Φ ( π k − 1 )
( i ) Nϵ .
A union bound concludes the proof.
H.1.4 Estimators for (cid:98) B k w , θ
We can directly invoke Lemma 17 in [89] to get guarantees for the estimator (cid:98) B k w , θ ( i ) .
In particular, we obtain the following result.
Lemma 13.
Let
[Equation]
i
[Equation]
Then, it holds that (cid:12)(cid:12)(cid:12)(cid:98) B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵ B k w , θ ( i ) ≤ 38 ηϵβ .
Proof.
First, we notice that (cid:13)(cid:13)(cid:13) MV kθ − (cid:99) MV kθ (cid:13)(cid:13)(cid:13) ∞ ≤ ϵγ implies that (cid:98) δ k w , θ ( i ) − δ k w , θ ( i ) ≤ ϵ .
Therefore, by Lemma 17 in [89] we get (cid:12)(cid:12)(cid:12)(cid:98)
B k w , θ ( i ) − B k w , θ ( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵ B k w , θ ( i ) .
Moreover, it holds that
[Equation]
Therefore, (cid:12)(cid:12)(cid:12)(cid:98) B k w , θ ( i ) − B k w , θ
( i ) (cid:12)(cid:12)(cid:12) ≤ 38 ηϵ β , and (cid:98) B k w , θ ( i ) ≤ B k w , θ ( i ) (1 + 38 ηϵ ) ≤ 1 β (1 + 38 ηϵ ) .
Corollary 8.
Let N 1 ≥ max (cid:16) O (cid:16) γ 2 mD 2 βϵlog(2 /δ )
(cid:17) , O (cid:16) γ √ mDβ 3 ϵ log(2 m/δ ) (cid:17)(cid:17) and N 2 ≥
8 m δ βϵ.
Then, for χ = O (cid:16) log 2 mδ βN (cid:17) , and for N ≥ max ( N 1 , N 2 ) , with probability at least 1 − δ , it holds that (cid:12) (cid:12) (cid:12)(cid:98)
B k w , θ ( i ) − B k w , θ ( i ) (cid:12) (cid:12) (cid:12) ≤ 38 ηϵβ , for all i ∈ [ m ] .
Proof.
By Corollary 5, we have that with N ≥ N 1 it holds that (cid:13)(cid:13)(cid:13) MV kθ − (cid:99) MV kθ (cid:13)(cid:13)(cid:13) ∞ ≤ ϵγ , with probability 1 − δ/ 2 .
Furthermore, Lemma 12 gives that for N ≥ 1 . 4loglog(2 N )+log 20 .
8 m δ βϵ, it holds with probability 1 − δ/ 2 that (cid:12)(cid:12) ρ Φ ( (cid:91)
π k − 1 )( i ) − ρ Φ ( π k − 1 )
( i ) (cid:12)(cid:12) ≤ 2 .
26 ϵ ρ Φ ( (cid:91)
π k − 1 )( i ) , for all i ∈ [ m ] simultaneously.
Therefore, a union bound gives that for N ≥ max ( N 1 , N 2 ) , with probability 1 − δ , we have that (cid:13)(cid:13)(cid:13) MV kθ − (cid:99) MV kθ (cid:13)(cid:13)(cid:13) ∞ ≤ ϵγ , and (cid:12)(cid:12) ρ Φ ( (cid:91)
π k − 1 )( i ) − ρ Φ ( π k − 1 )
( i ) (cid:12)(cid:12) ≤ 2 .
26 ϵ ρ Φ ( (cid:91)
π k − 1 )( i ) , for all i ∈ [ m ] .
An application of Lemma 13 concludes the proof.
H.1.5 Estimators for B k w , θ ( i ) Γ k ( i, j )
We obtain
Lemma 14.
Assume that for any ( i, j ) ∈ [ m ] 2 , it holds that (cid:12)(cid:12)(cid:12)(cid:98) B k w , θ ( i ) − B k w , θ
[Table] [Table]
≤ 38 ηϵβ and
[Table]
Γ k
[Table]
, j ) − Γ
[Table]
, j )
[Table]
≤ ϵ .
Then, (cid:12)(cid:12)(cid:12)
[Table]
k w , θ ( i ) Γ k ( i, j ) − (cid:98)
[Table]
k
[Table]
, θ ( i ) (cid:98) Γ k ( i, j ) (cid:12)(cid:12)(cid:12) ≤ ϵβ (1 + (1 + ϵ )38 η ) , for all ( i, j ) ∈ [ m ] 2 .
Proof.
We have that
[Table]
[Equation]
where we used the bound ˆ Γ k
( i, j ) ≤ Γ k
( i, j ) + ϵ ≤ 1 + ϵ .
Lemma 15.
For χ = O (cid:16) log mδ βN (cid:17) , choose N ≥ max
[Equation] [Equation]
O
[Equation]
with N 1 and N 2 as defined in Corollary 8, then with probability 1 − δ , for all ( i, j ) ∈ [ m ]simultaneously:
[Equation]
Proof.
By Corollary 8, when χ = O (cid:16) log mδ βN (cid:17) , and N ≥ max ( N 1 , N 2 ) , it holds with probability at least 1 − δ that
[Equation]
i
[Equation]
Moreover, when N ≥ max (cid:16) O (cid:16) mβϵ 2 log( m/δ ) (cid:17) , O (cid:16) √ mβ 3 ϵ log( m 2 /δ ) (cid:17)(cid:17) , by Corollary 7, with probability at least 1 − δ , it holds that (cid:13)(cid:13)(cid:13) Γ k ( · , j ) − (cid:98) Γ k ( · , j ) (cid:13)(cid:13)(cid:13) 2 ≤ ϵ , for all j ∈ [ m ] simultaneously.
Finally, a union bound and Lemma 14 give that with probability at least 1 − δ , it holds that
[Equation]
H.2 Properties of Stochastic gradients
Lemma 16.
Let N ≥ max (cid:16) N 1 , N 2 , O (cid:16) m βϵ 2 log( m/δ ) (cid:17) , O (cid:16) √ m β 3 ϵ log( m 2 /δ )
(cid:17)(cid:17) with N 1 and Nwith N 1 and N 2 as defined in Corollary 8.
Then, with probability 1 − δ , the following bounds on the stochastic gradient variance hold simultaneously:
[Equation]
Furthermore, with probability at least 1 − δ , the following bounds on the stochastic gradient bias hold simultaneously:
[Equation]
Proof.
Variance for gradient wrt θ .
Recall that by definition of the stochastic gradient we have that
[Equation]
It then follows that
[Equation]
Invoking Lemma 15, we have that if N ≥ max
[Equation]
N
[Equation]
with N 1 and N 2 as defined in Corollary 8, then with probability 1 − δ ,
[Equation]
Similarly, by Corollary 8, for N ≥ max ( N 1 , N 2 ) , we have that with probability 1 − δ ,
[Equation]
Hence, a union bound gives that with probability 1 − δ ,
[Equation]
This implies that
[Equation]
Therefore, by introducing a filtration F N = σ (cid:16) { ( s ( n ) k − 1 , a ( n ) k − 1 , s ′ ( n ) k − 1 ) } Nn =1 (cid:17)
[Equation]
and noticing that (cid:98) B k w , θ and (cid:98) Γ k are F N -measurable, we get
[Equation]
At this point
[Equation]
we can simply notice that
[Equation]
Therefore
[Equation]
with probability 1 − δ
[Equation]
it holds that
[Equation]
Variance for gradient wrt w .
Similarly with Corollary
[Equation]
we obtain that if N ≥ max ( N 1 , N 2 ) , then with probability at least 1 − δ ,
[Equation]
This implies that
[Equation]
[Table]
Bias for gradient wrt θ .
By using the unbiased estimator (cid:101) ∇ θ ,j G k ( w , θ ) in Equation (25), we get
[Equation]
∇
[Equation]
By choosing χ and N as in Lemma 15 and Corollary
[Equation]
and by a union bound
[Equation]
we have that with probability 1 − δ ,
[Equation]
Using that (cid:101) ∇ θ ,j G k ( w , θ ) is an unbiased estimator of ∇ θ ,j G k ( w , θ ) , we get
[Equation]
Hence, we have that (cid:13)(cid:13)(cid:13) E
[Equation]
∇
[Equation]
j G
[Equation]
w , θ
[Equation]
|F N
[Equation]
− ∇ θ
[Equation]
j G k
[Equation]
w
[Equation]
θ
[Equation]
1 ≤ m ϵβ ( γ + 38 η (1 + γ
[Equation]
1 + ϵ ))) .
Bias bound for the gradient wrt w .
Similarly, we can notice that with probability at least 1 − δ , it holds that
[Equation]
Since we have only one non-zero element, and by the unbiasedness of (cid:101) ∇ w ,j G k ( w , θ ) , we get
[Equation]
I Proof of Theorem
We first prove a generalization of the Azuma-Hoeffding inequality (Theorem 3.14 in [75]) that holds when the martingale difference sequence is bounded with high probability but not almost surely.
Lemma 17 (Modified Azuma-Hoeffding) .
Let { Y i } ni be a martingale difference sequence adapted to F i , such that for each i , | Y i | ≤
c i with probability at least 1 − δ 2 .
Then, it holds that
[Equation]
Proof.
Define the events E i =
{ Y i ≤ c i } and the intersection E = ∩ ni =1 { E i } , and notice that P [ E c ] = P [ ∪ ni =1 { E ci } ] ≤ (cid:80) ni
=1 P [ E ci ] = nδ 2 .
We then have the following decomposition:
[Equation]
i =
Y
i ≥ ϵ (cid:35) =
P (cid:34) { n (cid:88) i =
Y
i ≥ ϵ } ∩ E (cid:35) + P (cid:34) { n (cid:88) i =
Y i ≥ ϵ } ∩ E c (cid:35) ≤ P (cid:34) { n (cid:88) i =
Y
i ≥ ϵ } ∩ E (cid:35) + P [ E c ] ≤ P (cid:34) n (cid:88) i =
Y i ≥ ϵ (cid:12)(cid:12) E (cid:35)
P [ E ] (cid:124) (cid:123)(cid:122) (cid:125) ≤+ nδ 2 ≤ exp (cid:18) − 2 ϵ(cid:80)
ni
=
c 2 i (cid:19) + nδ 2 , where in the last step we noticed that under the event E , the martingale difference sequence is bounded almost surely, therefore we can apply the standard Azuma-Hoeffding inequality.
Corollary 9.
Let { Y i } ni be a martingale difference sequence adapted to F i , such that for each i ,
| Y i | ≤ c i with probability at least 1 −
δ 2 .
Then, with probability 1 − δ 1 (with δ 1 > nδ 2 ), it holds that
[Equation]
Proof of Theorem 2.
We fix a policy evaluation step k ∈ [ K ] , i.e., we study the k -th iteration of the outer loop of Algorithm 1.
Similarly to the proof of Lemma 19 in [89]
[Equation]
the biased SGD subroutine can be seen as an inexact gradient ascent scheme with updates
[Equation]
with
[Equation] [Equation]
By
Lemma
[Equation]
and a union bound, we get that for n ( t ) ≥ max (cid:110) O (cid:16) γ 2 mD
[Equation]
log
[Equation] [Equation]
[Equation]
, with probability at least 1 − δ/ 2 , for all t = 1 , . . .
, T simultaneously, it holds that
[Equation] [Equation]
41) where we used that { ξ t } Tt =1 ∪ { γ } ⊂ (0 , 1) .
Moreover, by Hölder’s inequality, we get
[Equation]
where we used that by the triangle inequality and Proposition 3, it holds that ∥ θ kt − θ ⋆k ∥ ∞ ≤ 2 1+ | log β | 1 − γ ≜ 2 D .
We recall that D ≜ 1+log ( 1 β ) 1 − γ ≥ 1 .
Since (cid:110)
X k θ ,t ≜
[Equation]
ϵ k θ ,t , θ kt − θ ⋆k
[Equation]
(cid:111) ∞ t =1 and (cid:110)
X k w ,t ≜ (cid:10) ϵ k w ,t , w kt − w ⋆k (cid:11) (cid:111) ∞
t =1 are martingale differences, by using Corollary 9 and a simple union bound, we get that with probability at least 1 − δ/ 2 ,
[Equation]
Furthermore, note that G k is η + α -smooth with respect to the ∥·∥ ∞ -norm, and so by Lemmain [89], we can bound the ∥·∥ 1 -norm of its gradients.
In particular, we have
[Equation]
This in turn implies that
[Equation]
By smoothness and concavity of the objective G k , we can apply Lemma 9 in [89].
In particular, by
Equations (38)–(47), a union bound, and by summing over t in the bound of Lemma 9 in [89], we have the following guarantee for our inexact gradient scheme:
If
[Equation] [Equation]
, and β t ≤ 2 α + η , then with probability at least 1 − δ , it holds that
[Equation]
where
[Equation] [Equation]
We choose β t =
L √ t , for some constant L .
Then a telescoping sum gives
T (cid:88) t =
[Equation]
Moreover, (cid:80) Tt =1 β t L 1 ≤ 2 L 1 L √ T .
By combining this inequality with Equations (55) and (61), we get that
[Equation]
The optimal choice for L is L = √
+ D 2 2 √
L 1 .
In addition, by setting ξ t = (cid:113) L 1 t , we conclude that
[Equation]
Therefore, by combining Equations (56)–(59) and Equation (64)
[Equation]
and by Jensen’s inequality
[Equation]
we get that if n
[Equation]
t
[Equation]
, and β t
= O ( 1 √ t ) , then, with probability at least 1 − δ , it holds that G k
[Equation]
w
[Equation]
I.1 Proof of Corollary
Proof of Corollary 1.
We plug the upper bound for ϵ k given by Theorem 2 in the error propagation analysis of Theorem 1.
In particular, from Theorem 1, with probability at least 1 − δ 1 , it holds that
[Equation]
where we replaced we made explicit the fact the constant (wrt to K and T ) C depends on α and η (See Theorem 1 for the exact expression).
By plugging in the bound for ϵ k given by Theorem 2, and a union bound, we get that and if we use n ( t ) ≥ max (cid:16) O (cid:16) γ 2 mDtβ log Tmδ 2 (cid:17) , O (cid:16) mtβ log Tmδ 2 (cid:17)(cid:17) samples per iteration, then with probability at least 1 − δ 1 − δ 2 , it holds that
[Equation]
Setting η = α = 1 , letting C 1 ≜ C (1 , 1) and keeping only the dominant terms, we obtain
[Equation]
Then, choosing K = D ( λ ∗ || Φ
[Equation]
d 0 )
+ H ( d
[Equation]
|| d 0 ) ϵ and T = Ω (cid:16) m 4 D 4 β 4 C
ϵ 4 (cid:17) , we can ensure that d C ( (cid:98) π, π E ) ≤ ϵ + ε .
The overall sample complexity is Kn ( T )
=
Ω ( KT ) =
Ω (cid:0) ϵ − 5 (cid:1) .
Notice that the corollary improves upon the sample complexity bound of Ω (cid:0) ϵ − 8 (cid:1) derived in [89].
J Offline imitation learning version
Inspecting Equation (5), one can notice that estimating the empirical logistic Bellman evaluation objective G k or its gradients requires sampling from d k − 1 .
Hence, the algorithm needs interactions with the environment at every iteration k .
It is possible to alleviate this requirement, changing the center point for the relative entropy.
This is akin to smoothing [82] choosing a convenient center point.
In particular, we replace Equation (2) with the following update:
[Equation]
Input: Feature matrix Φ , number of iterations K , step sizes η , α , and β
Input: Expert demonstrations D n
[Equation]
,H E ◦ Initialize π 0 as uniform distribution over A , and set w 0 = 1 m◦ Compute the empirical FEV (cid:99) ρ Φ ( π E ) using expert demonstrations ◦ Sample { ( s ( n ) , a ( n ) , s ′ ( n ) ) }
Nn =1 with s ( n ) , a ( n ) sampled i.i.d.
from µ π E and s ′ ( n ) ∼
P ( ·| s ( n ) , a ( n ) ) and compute the empirical offline logistic Bellman error by
[Equation]
policy evaluation & cost update
◦ Find an approximate maximizer of the negative empirical logistic Bellman error
[Equation]
// policy improvement
Policy update:
[Equation]
Output: Policy π d
Note that we have removed the iteration index k , since the offline version does not require to iteratively collect new samples from the environment.
Changing the reference distribution from Φ ⊺ d k to Φ ⊺ µ π E gives Algorithm 3.
In this case, the logistic Bellman evaluation objective takes the form
[Equation]
The difference with the online variant is that in the first term we have the expert occupancy measure instead of the occupancy measure induced by the current policy.
We describe the corresponding empirical estimate in Algorithm 3.
Furthermore, we suppress the index k , since the offline algorithm does not require multiple iterations.
J.1 Theoretical guarantees for the offline case
With minor modifications of the error propagation analysis given in Theorem 1, one can prove the following result.
Theorem 4.
Under the same assumptions as in Theorem 1, and by choosing α =
[Equation]
2 / 3 , we obtain
[Equation]
where ϵ 1 is the error in the maximization of the logistic Bellman error, i.e. ϵ = max w ∈W , θ G ( w , θ ) −
G ( w 1 , θ 1 ) and ε is the error in estimating the expert feature expectation vector as in Lemma 6.
Proof.
Following exactly the same steps in the proof of Theorem 1 for the special case of K = 1 , we get
[Equation]
By using the bound ∥ d 1 − d ⋆ 1 ∥ 1 ≤ (cid:113) 2 αϵ 1 1 − γ , we have
[Equation]
Therefore, by choosing α as stated in the theorem we conclude the proof.
Notice that if the expert is nearly optimal, the step size η can be taken small, ensuring low bias in the gradients.
This allows to use the original empirical logistic Bellman error analysis, proposed in [14], where one can control the bias by choosing η appropriately small.
To this end, we need to relate the logistic bellman error in the feature space to the one in the state-action space.
As we will show, this introduces an additional bias of order O ( η ) .
The statement is made precise in Theorem 5.
Thanks to this result and Theorem 2 in [14], we have that ϵ ≤
[Equation] [Equation]
e )
[Equation] [Equation]
where N is the number of expert transitions in the dataset.
We have the following result.
Corollary 10.
Let C
[Equation]
=
[Equation]
and N = ˜
O ( mϵ − 6 log (1 /δ )) .
Then, with probability 1 − δ , it holds that
[Equation]
Remark 2.
We notice that the optimal choice of η is smaller as the expert is closely optimal, i.e.
D ( λ ⋆ ||
Φ ⊺ µ π E ) is small.
In this condition, we can use the empirical objective estimator proposed in[14] ensuring small bias.
This means that estimating the objective from sample is feasible in the offline setting.
It is still an open question if this is viable for the online setting improving the error propagation analysis.
Next, we present an important result showing that it is possible to replace the minimization of G , with its counterpart in the state-action space defined as
[Equation]
where we introduced δ S , A w , θ = Φ δ w , θ .
Theorem 5.
Let B ≜ 1 + 2 1+ | log β | 1 − γ .
Suppose η is chosen such that ηB ≤ 1 .
Then, it holds that
[Equation]
Proof.
From Proposition 3, we have that ∥ θ ∥ ∞ ≤ 1+ | log β | 1 − γ and ∥ V θ ∥ ∞ ≤ 1+ | log β | 1 − γ , for all θ ∈ R m .
It follows that for any ( w , θ ) ∈ W × R m , it holds that ∥ δ θ , w ∥ ∞ =
∥ w + γ MV θ − θ ∥ ∞ ≤ 1 + 2 1+ | log β | 1 − γ = B .
Hence, it holds that η ∥ δ θ , w ∥ ∞ ≤ ηB ≤ 1 .
First, we recall the assumption that the rows of Φ are probability distributions, i.e., ϕ ( s, a ) ∈ ∆ [ m ] , for all ( s, a ) .
We then have δ S , A w , θ
[Equation]
s, a
[Equation] [Equation]
i
[Equation]
Moreover, we have
G
[Equation]
We can then lower bound W as
[Equation]
where the inequality follows by Jensen’s inequality for expectations.
We will now upper bound the term W .
Thanks to the choice of η such that ηB ≤ 1 , we have that η ≤ 1 B ≤ 1 | δ w , θ ( i ) | , for all i .
Therefore, we can apply the inequality e x ≤ 1 + x + x 2 for x = − η δ SA w , θ ( i ) ≤ 1 and obtain
[Equation]
where in the third line we used Equation (72), and in the last line we used the inequality 1 − x ≤ e − x for x = δ SA w , θ ( s, a ) .
By taking expectations with respect to µ π E and logarithms on both sides, we get
[Equation]
Subtracting W yields
[Equation]
where in the third line we used the inequality log(1 + x ) ≤ x for x = ( ηB
[Equation]
E ( s,a )
[Equation]
πE (cid:20) e − η δ SA w , θ ( s,a ) (cid:21) , while in the last line we used that ηB ≤ 1 .
This concludes the proof.
After having established with Theorem 5 that G S , A can be used as biased estimate of G , we can proceed as in [14].
In particular, we maximize the empirical objective (cid:98) G (see Algorithm 3) that is a biased estimate of G S , A
([14, Theorem 2]).
Then, we compute unbiased gradients of (cid:98) G , recurring to the Donsker-Varadhan formula [18, Corollary 4.15] that implies the following result.
Theorem 6.
Given a batch of expert data { (cid:101) S n , (cid:101) A n , (cid:101) S ′ n } Nn =1 ∼ µ π E × P , the following is true:
[Equation] [Equation]
with:
[Equation]
and the minimum attained at z
[Equation]
[Table]
Hence, in the deep learning implementation we update the cost and the value networks backpropagating through S ( θ , w , z ⋆ ) .
J.2 Practical implementation
We test a practical relaxation of Algorithm 3 that uses two separate neural networks for cost and value function approximation.
We use a two layers neural network with 128 units per layer with ReLu activation for the CartPole-v1 environment.
Whereas, for Acrobot-v1 and LunarLander-v2 we used a 3 layers architecture with 64 units per layer.
K Mirror Descent versus Proximal Point
To highlight an important message of our work, in this section, we briefly discuss a mirror descent scheme with alternating updates, and we compare it to our proximal point algorithm in Figure 6.
Note that in contrast to the classical RL setting, where proximal point and mirror descent coincide because of the linear objective, in imitation learning this is not the case.
The updates for the mirror descent scheme involve alternation between updating the occupancy measure d k and the feature expectation vector λ k in one stage and the cost weights in a second stage.
That is,
[Equation] [Equation]
One can notice that the update in Equation (77) corresponds to one update of Logistic Q -Learning [14].
Therefore, it can be implemented by maximizing the negative logistic Bellman error that is now a function only of the variable θ and not of both ( θ , w ) as in PPM.
The next proposition is the counterpart of Proposition 2 for the mirror descent scheme.
Proposition 5.
For a parameter θ ∈ R m , we define the state-action logistic value function Q θ ∈
R |S||A| by Q θ ≜ Φ θ , and the k -step state logistic value function V k θ ∈ R |S| by
[Equation]
Moreover, for a fixed cost c = c w , we define the k -step Bellman error function δ k θ , w by δ k θ , w ≜ w + γ MV k θ − θ .
Then, the unique solution of the aforementioned problem is given by
[Equation]
where θ k is the maximizer of the negative k -step logistic Bellman error function
[Equation]
Proposition 5 leads to an actor critic scheme that has three separate and alternating updates: (i) policy update stage, (ii) policy evaluation update, and (iii) cost weights update.
Similar actor critic-schemes for different MDP models, and different policy evaluation objectives (e.g., minimizing the squared
Bellman error) have been also proposed in [122, 70, 105].
Contrary to these schemes, in our proximal imitation learning algorithm, the policy evaluation step involves optimization of a single objective over both cost and Q -functions.
In this way, we avoid instability or poor convergence in optimization due to nested policy evaluation and cost update steps.
In section L.5, we verify numerically that PPM outperforms Mirror Descent in simple tabular environments (see Figure 6).
[Figure]
We report the results of some environments omitted main text.
L Experimental Details L.1 Refereences for environments description
In the tabular case we used the environments ( DoubleChain [39], SingleChain [39], RiverSwim [108], WideTree [8], Two States Deterministic [9], Two States Stochastic [14] and
WindyGrid [110]).
While for the offline setting, we used the environments CartPole [12], Acrobot [42] and LunarLander [19].
The curves are averaged over 50 seeds.
For the environments Cartpole and Acrobot , we used a three layer neural network to approximate the value function.
In these cases we averaged 5 seeds.
L.2 Hyperparameters
We report the hyperparameters for the tabular online experiments in Table 1 and for the offline experiments in Table
Environment n-trajs lr w lr θ η α optimizer
TwoStateStochastic-v  0.
FoRB
TwoStateStochastic-v  0.
Adam
WideTree
[Table]
L.3 On the data sampling
In all the experiments
[Table]
we perform a relaxation of our theoretical scheme.
In particular, to increase the sample efficiency we sample state action pairs from the Markovian stream of experience.
Analyzing this setting is an open problem.
[Table] [Table]
As optimizer, we used OGD in all cases.
L.4 Offline experiments setting
We consider a training environment and a test environment with different random seeds.
We train both IQLearn and Proximal Point for 2 e 5 environment steps and we evaluate the policy runningepisodes on the evaluation environment every 1 e 3 steps.
We report the maximum evaluation result achieved at the end of training.
We average the seeds from 0 to 10 for the results shown in Figure 2.
We use two separate instances of the same architecture as function approximation for the Q -values and cost respectively.
Finally, since the algorithm operates offline it has no access to the distribution ν 0 .
In order to approximate the term ⟨ ν 0 , V ⟩ , we use the Bellman flow constraints and the fact that the expert occupancy measure is feasible, i.e. (1 − γ ) ⟨ ν 0 , V ⟩ = (cid:10) µ π E , − γ PV + BV (cid:11) where the last term can be estimated from the expert samples.
L.5 Comparison with mirror descent
We designed also a mirror descent scheme with alternating updates for imitation learning, briefly described in Appendix K.
The best hyperparameters are given in Table 3.
Furthermore, we show a comparison with our proximal point scheme in Figure 6.
It is interesting to notice that mirror descent and proximal point have been used interchangeably in the RL literature.
Indeed, in that case the objective is linear therefore the two algorithms coincide.
However, when considering the max-form objective in imitation learning the equivalence between mirror descent and proximal point does not longer hold true.
We verify numerically that PPM outperforms mirror descent in simple tabular environments (see Figure 6).
L.6 Hyperparameters for Pong (Atari)
We use a convolutional neural network to learn the Q values instead of the linear function approximation class we considered in the theoretical analysis.
We set the parameter α to 1 e − 3 and η to 8 e − 2 , we used expert samples to approximate expectation with respect to the initial distribution.
For optimizing the network we used Adam [61] with learning rate 1 e − 4 and defaults value for β 1 , β
Instead of hard constraints on the euclidean norm of the elements of W we consider a ℓ 2 penalty to the loss function.
As expert trajectories we used the dataset released by [40].
This is the only hyperparameters configuration we tried using a single seed (using the seed 0 ) on our method because of the high computation requirements of this environment.
[Figure]
L.7 Hyperparameters for MuJoCo (continuous control)
The policy network outputs a distribution over continuous action and is parametrized by independent gaussian distributions for every component of the continuous action vector.
We use a three layer neural network to estimate their means and variances.
We used as center point in the divergence
D the expert feature expectation vector.
With further modifications our method can extend also to continuous control tasks in MuJoCo [113].
The main challenge is that the policy improvement step can not be computed in closed form.
We therefore approximate it with a SAC architecture as proposed in [40].
We set α to 1 e − 3 , η to 8 e − 2 , the SAC actor learning rate to 3 e − 5 using
Adam as optimizer using default values of β 1 , β 2 ,for the critic we used again Adam with learning rate 3 e − 4 and default values for β 1 , β 2 .
The actor training of SAC is performed using a transition buffer containing expert and learner data in equal proportion.
We used samples from the expert policy to estimate expectations wrt the initial distribution.
We avoid using target networks.
We tested our algorithm on both the environment Ant and HalfCheetah using either the data provided in [40] or fresh expert data that we generated training experts with PPO [101].
The results are averaged across 5 seeds.
For Hopper , we used a larger SAC actor learning rate equal to 2 e − 4 and α = 1 e − 2 .
In addition, we notice that for this environment having a large β 1 in Adam was harmful.
Hence, we used β 1 = 0 .
For Walker , we set the actor learning to 1 e − 4 .
L.8 Acknowledging existing assets and license.
We built on the code and expert data provided in [40].
They are open sourced for academic scope according to their GitHub page https://github.com/Div99/IQ-Learn/blob/main/LICENSE.
md .
L.9 On the importance of the dataset
We observed that the performance of our imitation learning algorithm and IQ-Learn can be affected by the choice of the expert data.
In particular, in Appendix L.9, we show that IQ-Learn works better with the expert data provided in [40].
L.10 Hardware
We ran the experiments on our internal cluster.
[Figure]
M Recovered Costs
A unique algorithmic feature of the proposed methodology is that we can explicitly recover a cost along with the Q -function without requiring adversarial training.
In Figures 8 and 3, we visualize our recovered costs in several simple tabular environments ( River Swim , Single Chain , Double
Chain , and Gridworld , respectively).
Most importantly, we verify that the recovered costs induce nearly optimal policies w.r.t.
the unknown true cost function.
Compared to IQ-Learn, the we do not require knowledge or further interaction with the environment.
Therefore, the recovered cost functions show promising transfer capability to new dynamics.
We experimented with a transfer reward setting on a Gridworld (Figure 4).
We consider two different Gridworld MDP environments, say M and (cid:102) M , with opposite action effects.
This means that action Down in (cid:102)
M corresponds to action Left in M and vice versa.
Similarly, the effects of
Up and Right are swapped between (cid:102) M and M .
We denote by V π (cid:102)
M, c true (resp.
V ⋆ (cid:102) M, c true ) the value function of policy π (resp.
optimal value function) in the MDP environment (cid:102)
M with cost function c true .
Moreover, we denote by π ⋆M, c
the optimal policy in the MDP environment M under cost function c .
We notice that the recovered cost induces an optimal policy for the new dynamics while the imitating policy fails.
Albeit, cost transfer is successful in this experiment we do not expect this fact to be true in general because we do not tackle the issue of cost shaping [87].
M.1 Preliminary theoretical arguments
We have some preliminary theoretical arguments justifying the near optimality of the recovered costs/rewards.
We present briefly the reasoning.
For brevity, we consider the case W
=
B m 1 .
Then π E is optimal for the IL problem.
Moreover, for simplicity, we consider the case Φ
= I .
Otherwise, in the following derivations, we replace Q -values by parameterized Q θ .
Let ( (cid:98) w K , (cid:98) Q K ) be the output (average iterate) of P 2 IL after K outer loop iterations.
We give a sketch of proof that (cid:98) w K converges to an optimal solution to the inverse problem as K → ∞ , i.e., (cid:98) w K converges to some w A ∈ W such that π E is optimal for c w A .
To this end, we first introduce the following definition.
Definition 1.
We say that w ∈ W is ε 1 -optimal and ε 2 -feasible for the (Dual) program if-f there exists V ∈ R |S| , such that
[Equation]
In this case, V ∈ R |S| is called a certificate.
Note that the definition of ε 1 -optimality for the (Dual) program follows from the fact that the dual optimal value is ζ ⋆ = 0 .
Moreover, in the definition of ε 2 -feasibility we have relaxed the nonnegativity constraint in the dual program (Dual).
We make the following conjecture.
[Table]
S t a t e I n d e x 5311(j)
− c K 0.40.20.00.20.41357    (k) - V ⋆ c true 0.40.20.00.20.41357217.5 220.
(l) - V ⋆ c K
Conjecture: For a sufficiently large number of samples N = O (cid:16) poly (cid:0) 1 ε , log( 1 δ (cid:1) , m ) (cid:17) , with probability at least 1 − δ , the output cost weight (cid:98) w K is ε -optimal and ε -feasible for the (Dual) program, with certificate the corresponding logistic value function V (cid:98) Q K .
This is easy to show for the exact PPM updates, since ( d (cid:98) π K , d (cid:98) π K , (cid:98) w K , V (cid:98) Q K , (cid:98) Q K ) is a saddle-point of the (SPP).
The proof needs much more effort for the inexact updates used in the sampling-based algorithm.
Lemma 18.
Assume that (cid:101) w is ε 1 -optimal and ε 2 -feasible for the (Dual) program.
Then, π E is ( ε 1 + ε 2 ) -optimal for c (cid:101) w .
Proof.
There exists (cid:101) V ∈ R |S| , such that
[Equation]
Let (cid:101) π be an optimal policy for c (cid:101) w .
Then, we have that
[Equation]
By using that ( B − γ P ) T µ (cid:101) π
= (1 − γ ) ν 0 , we equivalently that
[Equation]
Therefore,
[Equation]
Thus, π E is ( ε 1 + ε 2 ) -optimal for c (cid:101) w .
Claim: As K → ∞ one may approach as closely as desired an optimal solution to the inverse problem.
Proof for the ideal PPM updates.
We recall that by Proposition 4, the set of such solutions is characterized as the set of w -optimizers to (Dual).
Let (cid:98) V K = V (cid:98) Q K .
By the conjecture, for all K , we have
[Equation] [Equation]
[Equation]
for some sequence { ε K } ∞ K =1 such that lim K →∞ ε K = 0 .
The sequence { (cid:98) w K } ∞ K =1 ⊂ W is bounded and so there exists a subsequence { (cid:98) w K l } ∞ l =1 , such that lim l →∞ (cid:98) w K l = w A , for some w A ∈ W .
Similarly, by Proposition 3 the sequence { (cid:98) V K l } ∞ l =1 is bounded and so there exists a subsequence { (cid:98) V K ln } ∞ n =1 , such that lim n →∞ (cid:98) V K
ln = V A , for some V A .
By Equations(86)–(87), we have that for all n ∈ N ,
[Equation]
Taking n → ∞ , we end up that
[Equation]
Equivalently,
[Equation]
Therefore, by Proposition 4, π E is optimal for c w A .